[nycbug-talk] Apache and FreeBSD

[Matt Juszczak]

Hi all,

Recently did a migration to a new data center.  All is well, except some 
of the webs are spiraling out of control.  They are actually pegging at 
about 99% CPU usage, but hardly any memory usage, and not a ton of I/O.

Load average climbs to be about the total amount of apache processes (so 
about 128.00) and we get lots of lockf's:

  2105 www           1  20    0   179M 29692K lockf  2   0:06 17.09% httpd
  2139 www           1  20    0   177M 28020K lockf  2   0:02 12.06% httpd
  2134 www           1  52    0   177M 29316K select 1   0:03  9.86% httpd
  2126 www           1  60    0   177M 28300K select 1   0:02  9.67% httpd
  2150 www           1  53    0   177M 28028K select 0   0:04  9.57% httpd
  2136 www           1  50    0   180M 30836K CPU2   0   0:03  9.08% httpd
  2108 www           1  20    0   178M 28992K lockf  0   0:04  7.28% httpd
  2118 www           1  49    0   180M 30264K CPU1   1   0:03  7.28% httpd
  2121 www           1  20    0   177M 29480K lockf  3   0:04  6.79% httpd
  2113 www           1  20    0   179M 30044K lockf  1   0:05  6.49% httpd
  2112 www           1  20    0   178M 29240K lockf  2   0:03  5.86% httpd
  2143 www           1  20    0   178M 28936K lockf  3   0:03  5.57% httpd
  2123 www           1  20    0   178M 29068K lockf  0   0:03  5.18% httpd
  2131 www           1  20    0   178M 29972K lockf  1   0:02  4.49% httpd
  2130 www           1  20    0   177M 28592K lockf  0   0:02  4.05% httpd
  2114 www           1  20    0   178M 30968K lockf  3   0:03  3.96% httpd
  2127 www           1  20    0   178M 28920K lockf  2   0:02  3.66% httpd
  2119 www           1  20    0   178M 29016K lockf  1   0:02  3.56% httpd
  2132 www           1  20    0   177M 27700K lockf  0   0:01  3.56% httpd
  2128 www           1  46    0   177M 27644K CPU0   0   0:01  3.17% httpd
  2140 www           1  20    0   177M 27580K lockf  1   0:03  2.98% httpd
  2147 www           1  20    0   177M 28000K lockf  1   0:02  2.88% httpd
  2145 www           1  46    0   178M 28612K select 3   0:01  2.88% httpd
  2129 www           1  45    0   177M 29120K select 3   0:01  2.78% httpd
  2142 www           1  20    0   177M 27552K lockf  3   0:01  2.49% httpd
  2117 www           1  20    0   179M 29872K lockf  0   0:02  2.39% httpd
  2115 www           1  45    0   177M 29164K select 2   0:02  2.39% httpd
  2141 www           1  46    0   177M 27956K select 2   0:02  2.29% httpd
  2148 www           1  20    0   178M 28820K lockf  3   0:02  2.29% httpd
  2124 www           1  20    0   177M 28100K lockf  1   0:01  2.10% httpd

I did some reading online and found out that this could be do to the way 
freebsd handles binding to specific IP's, and that doing *:80 fixes this. 
We made that change, but we're still seeing tons of processes in lockf.

Running ktrace (and kdump), I'm able to see a bunch of resources not 
available (see about 3/4th of the way down below):

  67604 httpd    RET   write 259/0x103
  67604 httpd    CALL  gettimeofday(0x7fffffffea50,0)
  67604 httpd    RET   gettimeofday 0
  67604 httpd    CALL  getrusage(0,0x7fffffffe9a0)
  67604 httpd    RET   getrusage 0
  67604 httpd    CALL  getrusage(0xffffffff,0x7fffffffe9a0)
  67604 httpd    RET   getrusage 0
  67604 httpd    CALL  clock_gettime(0x4,0x7fffffffea30)
  67604 httpd    RET   clock_gettime 0
  67604 httpd    CALL  gettimeofday(0x7fffffffe840,0)
  67604 httpd    RET   gettimeofday 0
  67604 httpd    CALL  shutdown(0x22,<invalid=1>)
  67604 httpd    RET   shutdown 0
  67604 httpd    CALL  poll(0x7fffffffe850,0x1,0x7d0)
  67604 httpd    RET   poll 1
  67604 httpd    CALL  read(0x22,0x7fffffffe8a0,0x200)
  67604 httpd    GIO   fd 34 read 0 bytes
        ""
  67604 httpd    RET   read 0
  67604 httpd    CALL  close(0x22)
  67604 httpd    RET   close 0
  67604 httpd    CALL  read(0x9,0x7fffffffead7,0x1)
  67604 httpd    RET   read -1 errno 35 Resource temporarily unavailable
  67604 httpd    CALL  gettimeofday(0x7fffffffea80,0)
  67604 httpd    RET   gettimeofday 0
  67604 httpd    CALL  flock(0x1d,LOCK_EX)
  67604 httpd    RET   flock 0
  67604 httpd    CALL  kevent(0x21,0,0,0x808207478,0x3,0)
  67604 httpd    GIO   fd 33 wrote 0 bytes



Anything anyone can do to help enlighten me would be great!

-Matt