[nycbug-talk] multiple make buildkernel

[Bjorn Nelson]

NYCBuggers and self,

On Dec 1, 2005, at 4:19 PM, Bjorn Nelson wrote:

> On Dec 1, 2005, at 1:49 PM, Bjorn Nelson wrote:
>
>> NYCBuggers,
>>
>> I have a quad processor box that is fairly slow cpu-wise.  I am  
>> using the make buildworld -j option (benchmarks so far are 47 min  
>> 9 sec at j11, 47 min 18 sec at j10,  47 min 5 sec at j9, still  
>> playing with this).  Buildkernel seems to take an hour and half at  
>> least.  I am wondering what would be involved to get multiple  
>> buildkernel's to run for different kernels.  I believe a make  
>> buildkernel only writes to /usr/obj/usr/src/sys/KERNELNAME.  If  
>> this is true, it should be okay for multiple buildkernel's to run,  
>> right?
>
> I just ran four make buildkernels with different kernels  
> simultaneously without error, much faster.

I just did an installkernel on an nfs client after doing this and it  
came up fine.  This worked so well, I am adding this to my script  
that locks /usr/src from allowing an install during a build.

use & abuse

====


#!/bin/sh

# Created by: Bjorn Nelson 051128
#
# Description: Build sources while locking them.

# Check for proper usage
if [ ! $1 ]; then
   echo "Usage: $0 numberofthreads"
   exit 1
fi

# Set Your Variables
THREADS=$1
DIRLOC="/usr/src"
CATLOC="/bin/cat"
RMLOC="/bin/rm"
TOUCHLOC="/usr/bin/touch"
DATELOC="/bin/date"
MAKELOC="/usr/bin/make"
LOGDIR="/var/log"
MAKEFILE_NAME="BSDmakefile"
LOCKFILE_NAME="BUILDING"
DORUN="TRUE"
KERNEL_LIST="SMP GENERIC SMPPAE CARP SMPCARP PAELOCAL"

# Start "here" document to supercede /usr/src/Makefile
$CATLOC << EOF > $DIRLOC/$MAKEFILE_NAME

.if exists(${DIRLOC}/${LOCKFILE_NAME}) && empty(BUILDOVERRIDE)
.error ${DIRLOC}/${LOCKFILE_NAME} file exists. Cancelling install.
.else
.include <Makefile>
.endif

EOF

# Lock sources
$TOUCHLOC $DIRLOC/$LOCKFILE_NAME

# Start buildworld while overriding lock
echo "buildworld Started: `$DATELOC`" >> $LOGDIR/buildworld.out
if [ $DORUN ]; then
   $MAKELOC -C $DIRLOC -j $THREADS buildworld BUILDOVERRIDE=TRUE >>  
$LOGDIR/build
world.out 2> /var/log/buildworld.err
else
echo "Make buildworld skipped"
fi
echo "buildworld Completed: `$DATELOC`" >> $LOGDIR/buildworld.out

# Start buildkernel while overriding lock
# Running multiple builds simultaneously.  This isn't supported but
# is much faster and easier to manage

# Start buildkernels in a subshell simultaneously for performance gain
for i in $KERNEL_LIST
do
(
echo "buildkernel Started: `$DATELOC`" >> $LOGDIR/buildkernel-${i}.out
if [ $DORUN ]; then
   $MAKELOC buildkernel -C $DIRLOC BUILDOVERRIDE=TRUE KERNCONF=${i}>>  
$LOGDIR/bui
ldkernel-${i}.out 2> /var/log/buildkernel-${i}.err
else
echo "Make buildkernel skipped"
sleep 10
echo "Done"
fi
echo "buildkernel Completed: `$DATELOC`" >> $LOGDIR/buildkernel-${i}.out
) &
done

# Wait for child process to finish before removing lock
wait

# Unlock sources
echo "Unlock $DIRLOC"
$RMLOC $DIRLOC/$LOCKFILE_NAME