[nycbug-talk] make -jN performance

[Bob Ippolito]

On Jan 20, 2005, at 11:59, Isaac Levy wrote:

> Hey All,
>
> On Jan 19, 2005, at 5:34 PM, Pete Wright wrote:
>
>> hmm...then it looks like I should submit a PR to the doc team 
>> then...well
>> atleast it was an interesting exercise :)
>>
>> -pete
>
> Yeah- I wanted to pipe in quicklike with this one with some 
> super-crude test results,
>
> Basically I've been compiling a lot of FreeBSD jails lately, so what 
> the hey, I've thrown in some -j flags informally to see what happens, 
> here's my results:
>
> --
> Single CPU (crufty old office 800mhz PIII):
>
> - No flags, make world takes 49 min average.
> - j2, j4, j6, j10, all take 50 min average.
>
> --
> Dual CPU (new-ish Dual Xeon 2.6mhz, under other server load):
>
> - No flags, make world takes 22 min average.
>
> - j2, j4, j6, j10, all take about 12 min average. (NICE.)
>
>
>
> --
> Lots of other stuff could affect my results here, but in a nutshell, 
> it seems that the j flag has no effect unless there's more than 1 
> processor.
>
> That stated, I'd think that in my scripted jail-build stuff, it would 
> be worth my time to throw a -j4 flag in there, as it doesn't seem to 
> hurt much, and what the hey- the systems may port to 4 cpu machines in 
> the not-too-distant future...

You should have some way to detect the number of CPUs in the build 
system.  I don't know how to do this with FreeBSD, but on Mac OS X (any 
Darwin) you would do the following:

make -j`sysctl -n hw.ncpu`

I wouldn't recommend just guessing -j4.  Most people don't have 4 way 
systems, there is no gain (as you two have proven empirically) if you 
have less than 4 cpus, and in some cases people are using distcc.  I 
don't know how common this is for FreeBSD, but it can happen often on 
Mac OS X because it's so damn easy to setup (a checkbox in Xcode 
preferences per machine).  I would recommend something equivalent to 
the above by default, but make a special note for distcc users.  If I 
have 16 cpus on my network, all running distcc, then I would want "make 
CC=distcc -j16".  I don't think there's a quick and dirty way to get 
the number of CPUs available for distcc.

-bob