[talk] Strength of internet connection has significant impact on OpenBSD 'pkg-add'

[James E Keenan]

OpenBSD's pkg_add command and certain forms of pkg_info -- e.g., 
pkg_info -Q -- are sensitive to the quality of one's internet connection 
to a surprising degree.

Background

In the last week of December I noticed that my home internet service -- 
Verizon DSL -- was deteriorating severely.  Previously, I might get at 
most 3% packet loss during 'ping'.  Now I was losing 11% -- and that was 
to go up to over 20% at times.  I could still download most files, 
albeit much more slowly.  However, when I tried to download the Vagrant 
box holding an OpenBSD-6.2 VM prepared by Brian Callahan for the Jan 3 
NYCBUG meeting, the estimated time was more than 14 hours.  (Even when I 
got to LMHQ for the meeting, the download still took between 60 and 90 
minutes, so that was a very large file indeed.)

The packet loss problem got to the point where I threw in the towel on 
Verizon (after 14 years) and next week I'll have Optimum installed.

But while I would have expected that a lousy net connection would have 
impeded large downloads of files, I would not have expected that it 
would have impeded *other* network calls even more completely.

The case in point:  OpenBSD's 'pkg' command and (at least) the 'pkg_info 
-Q' syntax of that command.  Last week, following some guidance from 
Brian, I was able to use the ports system to install certain programs 
(e.g., git, perl, subversion) and to prepare two new ports for 
submission.  So I did not anticipate that when I would switch the 'pkg' 
for software installation purposes, I would have no success.

As I described on freenode #nycbug,

#####
Anyone around who can answer questions about OpenBSD pkg_add?
Based on http://ports.su/devel, I know that there exists a package 
'p5-Path-Tiny-0.014' at http://ports.su/devel/p5-Path-Tiny.
But when I call: sudo pkg_add p5-Path-Tiny, I get (after a delay):
https://ftp4.usa.openbsd.org/pub/OpenBSD/snapshots/packages/amd64/: Read 
short file.
Can't find p5-Path-Tiny
#####

This morning I got diagnostic suggestions on IRC from gman999, okan and 
bcallah.  To make a long story short, there was no problem in 
/etc/installurl or in /etc/doas.conf.  Setting $PKG_CONF (per M Lucas 
book) or not made no difference.  okan indicated that "a short read 
means that the file was not fully fetched -- as in, ftp fetch knows the 
size it should have gotten, but the actual number of bytes doesn't 
match."  To which I responded, "s it correct to infer from this that, 
when 'pkg_info -Q' or 'pkg_add' starts up, it goes out to the net and 
first tries to read a file so big that packet loss can be deadly?"  okan 
replied, "the index file is largish, so could be."

So I now write from an internet cafe where there is presumably a 
satisfactory network connection (most importantly, 0% packet loss from 
ping).  Here's what I got.

#####
$ time pkg_info -Q p5-Path-Tiny
p5-Path-Tiny-0.104 (installed)
     0m01.94s real     0m00.16s user     0m00.04s system
#####

And pkg-add subsequently took only a few seconds to install that package.

#####
$ perl -MPath::Tiny -E 'say q|hello world|'
hello world
#####

Now, notwithstanding the fact that OpenBSD can be used as a desktop OS, 
my hunch is that the vast majority of OpenBSD installations are on 
servers where the internet connection is fast and solid.  So the total 
number of humans affected by this problem is likely to be small. 
Nonetheless, the vulnerability of 'pkg_add', etc., to inferior network 
connections is, to me at least, surprising.  So perhaps an area for 
improvement.

Thank you very much.
Jim Keenan