[nycbug-talk] shared disks for heavy I/O

[Marco Scoffier]

Hello all,

I am looking for recommendations for a shared disk solution in the 3 to 
5TB range that can support heavy reading and writing.  Standalone ? 
Dedicated server ? Fiber channel ?  Budget is around $4000.  Has someone 
looked into this recently?  How does Gig-ethernet performance compare to 
other solutions in real world situations?

I am working with large data sets (presently 100G but eventually to 
become larger). I have to store intermediate steps of the 
computation to disk.  Often to speed things up I run parallel 
computations which each read and write 1 to 10G of data at the beginning 
and the end of the computation.  I have 2 16cpu servers with SATA disks 
which each exports its' disks to the other using NFS.  Very often all 
cpus on both systems are maxed at 100%.  Lately when ramping up the 
computations NFS has been locking up (even when only reading remotely 
and writing locally).  I/O has been slow (ls takes forever to return for 
eg.).  I think that we are probably asking too much of the current setup 
where we are both running computations and exporting NFS from the same 
machines.

Thanks,

Marco