<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 4-May-09, at 11:10 AM, marco scoffier wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Thanks a lot for the details Pete. I actually had you in might when I posed the question :)<br><br><snip><br><br>> on that budget i'd say you should be able to get pretty fast storage for ~5TB. it may not >be reliable though (i.e. not something like a netapp or isilon were you can suffer nfs >server failures w/ no downtime)<br><br>Sorry but too many double negatives in the opener... I think I understood, netapp and isilon are good but more expensive ? But I think I am more interested in the system you describe below...<br><br></div></blockquote><div><br></div><div>gah - that's awful typing on my part, sorry about man. basically i was trying to say that storage vendors like NetApp can provide you with high performance, reliable storage. But it is quite expensive, which would be well over the 4k budget.</div><br><blockquote type="cite"><div>Pete Wright wrote:<br><blockquote type="cite"><br></blockquote><blockquote type="cite">out setup was pretty simple:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">1 dual quad-core workstation with 32GB ram<br></blockquote><blockquote type="cite">1 3ware 9000 series sata raid controller (no BBU - although that'd probably help with your use case, but it'd drive up the cost).<br></blockquote><blockquote type="cite">1 external sata JBOD<br></blockquote><blockquote type="cite">(something similar to this: <a href="http://rackmountmart.stores.yahoo.net/sa3urastch10.html)">http://rackmountmart.stores.yahoo.net/sa3urastch10.html)</a><br></blockquote><blockquote type="cite">a bunch of large sata drives.<br></blockquote><blockquote type="cite"><br></blockquote>Forgive me for being a bit clueless here. I haven't done one of these external disk setups before. There are 10 cables running between the workstation and the external JBOD ? The RAID controller is in the workstation or the external ? The idea is that the workstation exports NFS shares through gigabit ethernet but uses all its memory and CPU for disk access ?<br></div></blockquote><div><br></div><div>so in our setup what happens is you have the external drive bay with lets say 10 SATA drives in them. The drives connect on a backplane which concentrates some (up to 4 i believe) SATA interfaces into one external SATA cable. The cable(s) then connect to external ports on our 3ware cards. The cards see the 10 individual drives though - so you can do hardware RAID on the 3ware card, or pass them through to your OS. If I have time today I can google up the parts we were using to do this...but here's a link from 3ware that may help get ya started:</div><div><br></div><div><a href="http://www.3ware.com/products/cables.asp">http://www.3ware.com/products/cables.asp</a></div><div><br></div><div>look under Cables for 9590SE and 3Ware Sidecar. we were using the 19" SATA "Multilance" CBL-IB-05M. Another configuration we've used is the 3Ware sidecar (check out the Drive Cages menu on the left hand side) - but this limits you to 4 drives. </div><div><br></div><div>In our case we had one workstation that had the storage directly attached to it for video playback. In your case I would recommend setting up a dedicated NFS server if possible. Then you can tune your systems accordingly.</div><div><br></div><div><br></div><br><blockquote type="cite"><div><blockquote type="cite"><font class="Apple-style-span" color="#144FAE"><font class="Apple-style-span" color="#006312"><br></font></font></blockquote><blockquote type="cite">The only hack we did was to format the disks in such a way that we did not use any of the inside tracks of the individual disks. this ensured that we'd be laying down, and reading blocks in a contiguous manner on the outside of tracks of the disk. it actually had a significant impact on the performance for us (at a slight storage penalty).<br></blockquote>I didn't know one had access to know where tracks are on the disk. That the drive manufacturer could lay down tracks randomly distributed across the disk if that helped them get the performance specs they required.<br><br></div></blockquote><div><br></div><div>yea this was achieved in the fdisk/parted phase of preparing the disks for a filesystem. it took a little math, hard drive knowledge and testing to get the correct values here :)</div><br><blockquote type="cite"><div><blockquote type="cite">a Battery Backup Unit on our RAID controller will further help with caching - and give you a little security in case of power failures etc.<br></blockquote><blockquote type="cite"><br></blockquote>Why does a BBU help with caching? I understand that it allows a write to finish from cache in the event of a power failure, but I didn't know it could help with performance, or did I misunderstand.<br></div></blockquote><div><br></div><div>sorry, I should have been more clear. The cache should help with performance of writes, as your disk subsystem can return an file handle of a write when it is in the BBU cache rather than waiting for the bits to hit the disk itself.</div><br><blockquote type="cite"><div><blockquote type="cite">also - don't forget about tuning your NFS client options. use large read and write block sizes; think about using async writes if your data isn't *that* important <grin>. and if you can use jumbo frames use them - that'll help both the client and server.<br></blockquote><blockquote type="cite"><br></blockquote>Thanks for the tips. We could do some async writes but then would need some integrity checks. This is financial data so someone cares about every number :) <br></div></blockquote><div><br></div><div>oh yea - then i'd stay away from async writes then :)</div></div><br><div>-p</div></body></html>