[nycbug-talk] Extremely large SAN/NAS

[Edward Capriolo]

On Fri, Jan 20, 2012 at 4:34 PM, Pete Wright <pete at nomadlogic.org> wrote:

> On Fri, Jan 20, 2012 at 08:11:25AM -0500, Christopher Olsen wrote:
> > Hello Everyone,
> >
> > Anyone here have any experience or thoughts on how to put together a
> large Data store?
> >
> > What I would like to accomplish would be to have something with the
> capacity in the area of 5,000 terabytes and also have the ability to take
> snapshots...
> >
> > It wouldn't necessarily need to appear as a single node but I definitely
> want to get the highest possible storage density per node. Also performance
> need not be considered as long as its within reason.
> >
>
> i think there are a couple things to keep in mind when building any
> large storage architecture.  the first is - what is your
> application/use-case?  this will help you figure out if you need a SAN
> or if a NAS will suffice.  For example - building out a huge SAN, what
> filesystem will eventually be overlayed on your LUN's, do you really
> need a 5PB SAN or can it be broken down into more managable pools...etc.
>
> Lets assume you are building a NAS infrastructure though, as I imagine
> that would be a more common use-case for a 5PB storage architecture.  My
> opinion is that if you are building out something this big you really
> would benefit working with an appliance vendor - esp. if this a tier-1
> system you are building (interestingly enough you'll find that vendors
> like Isilon and NetApp are actually based on FreeBSD).  Aside from
> hardware integration and support - appliances will also generally take
> care of HA clustering and other difficult problems.  You wouldn't want
> your 5PB datastore to have a SPOF would you :)
>
> A final thought is - check out clustered filesystems like Gluster,
> Ceph (http://ceph.newdream.net/) or something similar.  These solutions
> will allow you to leverage off the shelf hardware w/o sacrificing HA
> capabilities.  They also should scale if designed correctly from the
> begining.
>
>
> Although - like I said in the begining you really need to figure out
> your usecase when buiding something to scale like this.  Once you figure
> out how data is being accessed (block level via a SAN, at the IP layer
> via a NAS, or via an API from a clustered filesystem) that'll help you
> figure out what your system will look like at the end of the day.  Each
> one has is benefits and drawbacks.
>
> HTH,
> -pete
>
> --
> Pete Wright
> pete at nomadlogic.org
>
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> http://lists.nycbug.org/mailman/listinfo/talk
>


Tools like Lustre, Ceph, or OCFS2 seemed attractive to me at first, in that
you get the full semantics that you are used to from a filesystem, but you
really cant open a 9 TB file with vi. Would I want to run MySQL ontop of
OCFS2. Would I want to have 5 systems sharing the same OCFS2 partition with
mysql external locking? (probably not)


These solutions (Lustre, Ceph, or OCFS2) are great if I have a bank of 1000
web servers each run by a different entity and I want to give everyone
there own secured LUN in a SAN style. But for my needs I determined that
was not what I wanted. I wanted to scale a single application horizontally
in both storage an processing. I wanted to be able to store all my web logs
from a very busy site in a single system and process them with distributed
storage and distributed computation, or I wanted a key value store that
stretched out across 30 nodes. These very large storage use cases were why
I got into hadoop. http://hadoop.apache.org/. Hadoop is not a POSIX
filesystem (so you can not mount it directly) Although some vendors do
offer alternative hadoop implementations that can be mounted and used like
an NFS disk. Also why I work with Cassandra http://cassandra.apache.org .

To wrap it up knowing what you wish to to with your 5PB is important. You
may just want a large pool of SAN/NAS storage. That could be as simple as
having thousands of beefy servers with ZFS and managing who mounts what.
You could go with an Isilon if you really want one /export/bigdrive that
every server in your company can write to and read from. Or if you just
want to manage all your storage as single system you can go with an iscsi
vendor. (wow the $ value for a 5PB SAN would be sick! I would love to see
that quote), or if what you really want is a scalable "database" that is
not a filesystem you can check out some of the other tools I mentioned.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nycbug.org/pipermail/talk/attachments/20120120/ec6e36f7/attachment.html>