[nycbug-talk] Apache and FreeBSD
Christopher Olsen
cwolsen at ubixos.com
Wed Jul 1 17:11:58 EDT 2009
Do you have fs snapshots?
-Christopher
Ubix Technologies
T: 212-514-6270
C: 516-903-2889
32 Broadway Suite 204
New York, NY 10004
http://www.tuve.tv/mrolsen
-----Original Message-----
From: Matt Juszczak <matt at atopia.net>
Sent: Wednesday, July 01, 2009 4:05 PM
To: talk at lists.nycbug.org
Subject: [nycbug-talk] Apache and FreeBSD
Hi all,
Recently did a migration to a new data center. All is well, except some
of the webs are spiraling out of control. They are actually pegging at
about 99% CPU usage, but hardly any memory usage, and not a ton of I/O.
Load average climbs to be about the total amount of apache processes (so
about 128.00) and we get lots of lockf's:
2105 www 1 20 0 179M 29692K lockf 2 0:06 17.09% httpd
2139 www 1 20 0 177M 28020K lockf 2 0:02 12.06% httpd
2134 www 1 52 0 177M 29316K select 1 0:03 9.86% httpd
2126 www 1 60 0 177M 28300K select 1 0:02 9.67% httpd
2150 www 1 53 0 177M 28028K select 0 0:04 9.57% httpd
2136 www 1 50 0 180M 30836K CPU2 0 0:03 9.08% httpd
2108 www 1 20 0 178M 28992K lockf 0 0:04 7.28% httpd
2118 www 1 49 0 180M 30264K CPU1 1 0:03 7.28% httpd
2121 www 1 20 0 177M 29480K lockf 3 0:04 6.79% httpd
2113 www 1 20 0 179M 30044K lockf 1 0:05 6.49% httpd
2112 www 1 20 0 178M 29240K lockf 2 0:03 5.86% httpd
2143 www 1 20 0 178M 28936K lockf 3 0:03 5.57% httpd
2123 www 1 20 0 178M 29068K lockf 0 0:03 5.18% httpd
2131 www 1 20 0 178M 29972K lockf 1 0:02 4.49% httpd
2130 www 1 20 0 177M 28592K lockf 0 0:02 4.05% httpd
2114 www 1 20 0 178M 30968K lockf 3 0:03 3.96% httpd
2127 www 1 20 0 178M 28920K lockf 2 0:02 3.66% httpd
2119 www 1 20 0 178M 29016K lockf 1 0:02 3.56% httpd
2132 www 1 20 0 177M 27700K lockf 0 0:01 3.56% httpd
2128 www 1 46 0 177M 27644K CPU0 0 0:01 3.17% httpd
2140 www 1 20 0 177M 27580K lockf 1 0:03 2.98% httpd
2147 www 1 20 0 177M 28000K lockf 1 0:02 2.88% httpd
2145 www 1 46 0 178M 28612K select 3 0:01 2.88% httpd
2129 www 1 45 0 177M 29120K select 3 0:01 2.78% httpd
2142 www 1 20 0 177M 27552K lockf 3 0:01 2.49% httpd
2117 www 1 20 0 179M 29872K lockf 0 0:02 2.39% httpd
2115 www 1 45 0 177M 29164K select 2 0:02 2.39% httpd
2141 www 1 46 0 177M 27956K select 2 0:02 2.29% httpd
2148 www 1 20 0 178M 28820K lockf 3 0:02 2.29% httpd
2124 www 1 20 0 177M 28100K lockf 1 0:01 2.10% httpd
I did some reading online and found out that this could be do to the way
freebsd handles binding to specific IP's, and that doing *:80 fixes this.
We made that change, but we're still seeing tons of processes in lockf.
Running ktrace (and kdump), I'm able to see a bunch of resources not
available (see about 3/4th of the way down below):
67604 httpd RET write 259/0x103
67604 httpd CALL gettimeofday(0x7fffffffea50,0)
67604 httpd RET gettimeofday 0
67604 httpd CALL getrusage(0,0x7fffffffe9a0)
67604 httpd RET getrusage 0
67604 httpd CALL getrusage(0xffffffff,0x7fffffffe9a0)
67604 httpd RET getrusage 0
67604 httpd CALL clock_gettime(0x4,0x7fffffffea30)
67604 httpd RET clock_gettime 0
67604 httpd CALL gettimeofday(0x7fffffffe840,0)
67604 httpd RET gettimeofday 0
67604 httpd CALL shutdown(0x22,<invalid=1>)
67604 httpd RET shutdown 0
67604 httpd CALL poll(0x7fffffffe850,0x1,0x7d0)
67604 httpd RET poll 1
67604 httpd CALL read(0x22,0x7fffffffe8a0,0x200)
67604 httpd GIO fd 34 read 0 bytes
""
67604 httpd RET read 0
67604 httpd CALL close(0x22)
67604 httpd RET close 0
67604 httpd CALL read(0x9,0x7fffffffead7,0x1)
67604 httpd RET read -1 errno 35 Resource temporarily unavailable
67604 httpd CALL gettimeofday(0x7fffffffea80,0)
67604 httpd RET gettimeofday 0
67604 httpd CALL flock(0x1d,LOCK_EX)
67604 httpd RET flock 0
67604 httpd CALL kevent(0x21,0,0,0x808207478,0x3,0)
67604 httpd GIO fd 33 wrote 0 bytes
Anything anyone can do to help enlighten me would be great!
-Matt
_______________________________________________
talk mailing list
talk at lists.nycbug.org
http://lists.nycbug.org/mailman/listinfo/talk
More information about the talk
mailing list