[nycbug-talk] Lab environment

[G.Rosamond]

On May 20, 2004, at 3:53 PM, Sunny Dubey wrote:

> On Wednesday 19 May 2004 10:40 pm, ike wrote:
>
>> I think from this kind of testing, we can come up with a boatload of
>> really useful information- but I believe it would be a real waste of
>> time to run this benchmark style, even as the same hardware can 
>> perform
>> very differently with different *nixes.
>
> I think you are attempting to test too much.

Agree. . .If a useful comparison is going to be made, there needs to be 
ONE variable.

If we are testing network protocols, then let's get several boxes with 
the same BSD on each.

We can separate into multiple BSD's, but I think we'd have to make it a 
separate night.. .rebuild the boxes, do the documented or instructed 
tweaks.

Conclusions would be something like, NFS runs amazingly faster than 
SMB, etc on NetBSD 1.5.3 with an Apple iBook g4 as client.

Very simple, but we can at least draw some useful conclusions.

Maybe it's too simple. . .maybe we use three boxes. . .Net, Free, Open, 
and test multiple protocols on each box.  But then the tweaks and 
configs may change too much that it would distort the conclusions.  But 
we should decide on the client side box.


>
> I wouldn't mind seeing some raw numbers primarily because I haven't 
> seen a
> good set in a while, and because when i do see them, they usually are 
> crap
> cuz the tester forgot to do simple things like enlarge the network 
> buffer,
> etc.
>
> We all know OS-X is going to win the GUI battle.  And that the three 
> BSDs
> are going to tie each other for CLI mastery.
>
> However raw numbers aren't so bad.  First because I haven't seen 
> numbers
> comparing all 3 major BSDs together for sometime.  Secondly because 
> when
> most people do test, I've always noticed that they manage to leave out
> stupid simple shit like creating a custom kernel or increasing the 
> network
> buffer size.

But that angle has to be dealt with too. . .what tweaking and how much? 
  Every single open source related run in a test lab ends with some 
party or another complaining (usually legitimately) about custom 
configs.

Maybe the best way to deal with this is to have an email to each 
project, stating our test parameters, and request their recommended 
settings, so that no complaining can happen later on. . .

>
> Additionally we all know that the BSDs are different.  I'd be 
> interested in
> seeing how that difference plays up in simulated real world usage.  
> ie:  If
> FreeBSD's support for SMP helps it, while Obsd lack of SPM hurts it. 
> etc
> etc etc ( http://www.watson.org/~robert/freebsd/netperf/ nifty hacks)
>
>
>> Invite:
>> Sunny- since your a Mandrake user, and a very knowledgeable Linux user
>> and developer, would you be interested in participating and following
>> our testing with a Linux or a few relevant Linuxes of your choosing?
>
> thats fine.  I can use various live CDs and write a script to generate 
> lots
> of network activity.
>
>> It is outside of our scope to be able to include a Linux as anything
>> more than perhaps a client in our test network,
>
> thats fine as well
>
> Sunny Dubey

g