[nycbug-talk] shared disks for heavy I/O

[marco scoffier]

Thanks a lot for the details Pete. 
I actually had you in might when I posed the question :)

<snip>

 > on that budget i'd say you should be able to get pretty fast storage 
for ~5TB.  it may not >be reliable though (i.e. not something like a 
netapp or isilon were you can suffer nfs >server failures w/ no downtime)

Sorry but too many double negatives in the opener... I think I 
understood, netapp and isilon are good but more expensive ?  But I think 
I am more interested in the system you describe below...

Pete Wright wrote:
>
> out setup was pretty simple:
>
> 1 dual quad-core workstation with 32GB ram
> 1 3ware 9000 series sata raid controller (no BBU - although that'd 
> probably help with your use case, but it'd drive up the cost).
> 1 external sata JBOD
> (something similar to this: 
> http://rackmountmart.stores.yahoo.net/sa3urastch10.html)
> a bunch of large sata drives.
>
Forgive me for being a bit clueless here.   I haven't done one of these 
external disk setups before.   There are 10 cables running between the 
workstation and the external JBOD ?  The RAID controller is in the 
workstation or the external ?  The idea is that the workstation exports 
NFS shares through gigabit ethernet but uses all its memory and CPU for 
disk access ?
> since we were a linux shop with a bunch of former SGI'ers we used the 
> XFS filesystem which has *very* good streaming I/O performance.  For 
> your workload ZFS would probably suite fine - and you'd get to use 
> FreeBSD :)
>
Of course I would like the FreeBSD with ZFS solution :)  Have to see 
what works best with the tech who will be administering...
> The only hack we did was to format the disks in such a way that we did 
> not use any of the inside tracks of the individual disks.  this 
> ensured that we'd be laying down, and reading blocks in a contiguous 
> manner on the outside of tracks of the disk.  it actually had a 
> significant impact on the performance for us (at a slight storage 
> penalty).
I didn't know one had access to know where tracks are on the disk.  That 
the drive manufacturer could lay down tracks randomly distributed across 
the disk if that helped them get the performance specs they required.

> a Battery Backup Unit on our RAID controller will further help with 
> caching - and give you a little security in case of power failures etc.
>
Why does a BBU help with caching?  I understand that it allows a write 
to finish from cache in the event of a power failure, but I didn't know 
it could help with performance, or did I misunderstand.
> also - don't forget about tuning your NFS client options.  use large 
> read and write block sizes; think about using async writes if your 
> data isn't *that* important <grin>.  and if you can use jumbo frames 
> use them - that'll help both the client and server.
>
Thanks for the tips.  We could do some async writes but then would need 
some integrity checks.  This is financial data so someone cares about 
every number :) 
> sorry for rambling post - i've been neck deep in designing some new 
> storage systems and have been kicking around alot of ideas lately :)
>
I appreciate the ramblings.  That's why I subscribe to mailing lists, 
for the slightly open ended brain picking :) 

Marco