[talk] S3 Work-a-likes

[Edward Capriolo]

On Thu, Nov 3, 2016 at 11:09 AM, Pete Wright <pete at nomadlogic.org> wrote:

>
>
> On 11/03/2016 07:38, Mark Saad wrote:
>
>> All
>>   Last night there was a discussion about s3 work-a-likes. Here are the
>> two that I know of.
>>
>> 1. LeoFS .
>>      Written in Erlang, and runs well enough on FreeBSD, Solaris, Illumos
>> , Linux and maybe even  Windows.
>>
>> Written by rakuten and used in production on a number of things.
>> Including Project-Fifo.net, Rakuten.com
>> and a number of other systems.
>>
>> http://leo-project.net/leofs/
>>
>> 2. Minio .
>>     Written Go , and claims to be supported on many platforms .
>> Its fairly new but aims to be fully compatable.
>> https://minio.io
>>
>>
>
> Minio is pretty good - I worked with them for a while (updated their wiki
> as well for FreeBSD support, testing and ran a POC at my old shop).
>
> They are very eager to have more FreeBSD involvement, and ZFS is a really
> good fit for Minio as well.  I would classify it as a system capable of
> supporting small'ish S3 environments and i don't think it has support yet
> for distributed backing datastores.  If you were building a huge S3-like
> system you'd probably want it backed by a clustered filesystem so you can
> scale out storage and IOPS between many nodes in your cluster.
>
> Their "mc" S3 client is also pretty awesome - much better than the "aws
> s3" tools IMHO for interacting with S3.
>
> Hope this helps,
> -pete
>
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> http://lists.nycbug.org/mailman/listinfo/talk
>

When I mentioned S3-work alikes there are a few things to consider. I know
of some commercial ones. The challenge is it is not as simple as just the
object store.

For example:
using s3 you can setup notifications such that when a new object is created
a message is automatically sent to amazon SQS (simple queuing service)
using s3 you can setup notifications such that when a new object is created
a message is automatically sent to amazon kinesis (distributed high
throughput queuing service)
You can use amazon lamda to trigger code to execute on s3 events. So for
example if I am running a thumbnail service i can build that as
s3(raw)->lamba->imagamagik->s3(thumbnail)
When using elastic bean stalk you logs can automatically be written to S3
You can serve content directly from an S3 bucket as a web server
You can use elastic map reduce to query and s3 bucket using HADOOP!

On the administrative side you have amazons death-of-1,000,000 paper cuts
billing. However if you are using mogilefs lets say you likely need to
build your own reporting monitoring. For example if you are attempting to
bill-back sections of the business based on usage you need to create that
flow and process.

>From the standpoint of an software architect  you have to look at the
solution space. s3(raw)->lamba->imagamagik->s3(thumb nail) is a trivial
thing. I have done it without amazon the flow looks like this
Web/Appliation server->NAS->mesage on kafka queue <- Stream processor)
storm listening on kafka queue running code to do thumb-> write output to
other nas location.

People these days are into loose coupling and micro-services. "amazon
lambda" sounds better to most than "eds storm-thumbnail thing". Also
companies are not doing there part. I am sure there are hundreds or
thousands of people/organizations  that have written a scalable thumbnail
engine. But if you google this you get stack-overflow howtos

http://stackoverflow.com/questions/11376315/creating-a-thumbnail-from-an-uploaded-image

As much as I dislike some things about amazon they have couple build
technologies that make sense, work at scale, WORK WITH EACH OTHER
SEAMLESSLY, and are available via APIs for common development platforms.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20161103/fd8784bc/attachment.htm>