[nycbug-talk] network strangeness (resource starvation?)

[Charles Sprickman]

Hey all,

I've pursued this on other lists for a few years, and it's getting under 
my skin more and more.  Maybe someone here can give me some pointers...

Have a host (FBSD 4.9) that does lots of dns work - both queries from 
outside and other hosts inside doing a ton of lookups.

We have run up against a few hurdles and cleared them.   First it was 
ipfilter running out of state entries.  Upped the size in "ip_state.h" in 
the ipfilter includes, and that helped.  Eventually we hit another wall, 
so we relaxed the ipfilter rules to make them work for inbound/outbound 
without generating state entries.  Since then, no problems reported by 
"ipfstat -s" that would indicate we're running out of resources there.

One of the ongoing symptoms is that ssh sessions to the box will start 
*dropping* characters when udp traffic is really high.  Even after we 
solved the problem of outgrowing the state table, the problem still 
remains.

We've bumped a number of things, nmbclusters is way up there, and netstat 
-m shows that we're not hitting a peak there.  However looking at full 
"netstat -s" stats after the box only being up for less than 12 hours 
shows this:

         8297 dropped due to no socket
         0 broadcast/multicast datagrams dropped due to no socket
         31 dropped due to full socket buffers

So that's a hint.  I can look for whatever obscure sysctl variable to set 
the listen queue deeper.  Not sure about the "no socket"...

Lately the newest wrinkle is that the box will just go unresponsive. 
Pingable, but nothing on serial console, no ssh.

So can things getting starved in udp-land cause other networking stuff to 
choke?  Any pointers where else to look?

Thanks,

Charles