[nycbug-talk] shared disks for heavy I/O

[Pete Wright]

On 4-May-09, at 11:10 AM, marco scoffier wrote:

> Thanks a lot for the details Pete. I actually had you in might when  
> I posed the question :)
>
> <snip>
>
> > on that budget i'd say you should be able to get pretty fast  
> storage for ~5TB.  it may not >be reliable though (i.e. not  
> something like a netapp or isilon were you can suffer nfs >server  
> failures w/ no downtime)
>
> Sorry but too many double negatives in the opener... I think I  
> understood, netapp and isilon are good but more expensive ?  But I  
> think I am more interested in the system you describe below...
>

gah - that's awful typing on my part, sorry about man.  basically i  
was trying to say that storage vendors like NetApp can provide you  
with high performance, reliable storage.  But it is quite expensive,  
which would be well over the 4k budget.

> Pete Wright wrote:
>>
>> out setup was pretty simple:
>>
>> 1 dual quad-core workstation with 32GB ram
>> 1 3ware 9000 series sata raid controller (no BBU - although that'd  
>> probably help with your use case, but it'd drive up the cost).
>> 1 external sata JBOD
>> (something similar to this: http://rackmountmart.stores.yahoo.net/sa3urastch10.html)
>> a bunch of large sata drives.
>>
> Forgive me for being a bit clueless here.   I haven't done one of  
> these external disk setups before.   There are 10 cables running  
> between the workstation and the external JBOD ?  The RAID controller  
> is in the workstation or the external ?  The idea is that the  
> workstation exports NFS shares through gigabit ethernet but uses all  
> its memory and CPU for disk access ?

so in our setup what happens is you have the external drive bay with  
lets say 10 SATA drives in them.  The drives connect on a backplane  
which concentrates some (up to 4 i believe) SATA interfaces into one  
external SATA cable.  The cable(s) then connect to external ports on  
our 3ware cards.  The cards see the 10 individual drives though - so  
you can do hardware RAID on the 3ware card, or pass them through to  
your OS.  If I have time today I can google up the parts we were using  
to do this...but here's a link from 3ware that may help get ya started:

http://www.3ware.com/products/cables.asp

look under Cables for 9590SE and 3Ware Sidecar.  we were using the 19"  
SATA "Multilance" CBL-IB-05M.  Another configuration we've used is the  
3Ware sidecar (check out the Drive Cages menu on the left hand side) -  
but this limits you to 4 drives.

In our case we had one workstation that had the storage directly  
attached to it for video playback.  In your case I would recommend  
setting up a dedicated NFS server if possible.  Then you can tune your  
systems accordingly.



>>
>> The only hack we did was to format the disks in such a way that we  
>> did not use any of the inside tracks of the individual disks.  this  
>> ensured that we'd be laying down, and reading blocks in a  
>> contiguous manner on the outside of tracks of the disk.  it  
>> actually had a significant impact on the performance for us (at a  
>> slight storage penalty).
> I didn't know one had access to know where tracks are on the disk.   
> That the drive manufacturer could lay down tracks randomly  
> distributed across the disk if that helped them get the performance  
> specs they required.
>

yea this was achieved in the fdisk/parted phase of preparing the disks  
for a filesystem.  it took a little math, hard drive knowledge and  
testing to get the correct values here :)

>> a Battery Backup Unit on our RAID controller will further help with  
>> caching - and give you a little security in case of power failures  
>> etc.
>>
> Why does a BBU help with caching?  I understand that it allows a  
> write to finish from cache in the event of a power failure, but I  
> didn't know it could help with performance, or did I misunderstand.

sorry, I should have been more clear.  The cache should help with  
performance of writes, as your disk subsystem can return an file  
handle of a write when it is in the BBU cache rather than waiting for  
the bits to hit the disk itself.

>> also - don't forget about tuning your NFS client options.  use  
>> large read and write block sizes; think about using async writes if  
>> your data isn't *that* important <grin>.  and if you can use jumbo  
>> frames use them - that'll help both the client and server.
>>
> Thanks for the tips.  We could do some async writes but then would  
> need some integrity checks.  This is financial data so someone cares  
> about every number :)

oh yea - then i'd stay away from async writes then :)

-p
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nycbug.org/pipermail/talk/attachments/20090504/2ff21dbc/attachment.html>