<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 3, 2016 at 11:09 AM, Pete Wright <span dir="ltr"><<a href="mailto:pete@nomadlogic.org" target="_blank">pete@nomadlogic.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
<br>
On 11/03/2016 07:38, Mark Saad wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
All<br>
Last night there was a discussion about s3 work-a-likes. Here are the two that I know of.<br>
<br>
1. LeoFS .<br>
Written in Erlang, and runs well enough on FreeBSD, Solaris, Illumos , Linux and maybe even Windows.<br>
<br>
Written by rakuten and used in production on a number of things. Including Project-Fifo.net, Rakuten.com<br>
and a number of other systems.<br>
<br>
<a href="http://leo-project.net/leofs/" rel="noreferrer" target="_blank">http://leo-project.net/leofs/</a><br>
<br>
2. Minio .<br>
Written Go , and claims to be supported on many platforms .<br>
Its fairly new but aims to be fully compatable.<br>
<a href="https://minio.io" rel="noreferrer" target="_blank">https://minio.io</a><br>
<br>
</blockquote>
<br>
Minio is pretty good - I worked with them for a while (updated their wiki as well for FreeBSD support, testing and ran a POC at my old shop).<br>
<br>
They are very eager to have more FreeBSD involvement, and ZFS is a really good fit for Minio as well. I would classify it as a system capable of supporting small'ish S3 environments and i don't think it has support yet for distributed backing datastores. If you were building a huge S3-like system you'd probably want it backed by a clustered filesystem so you can scale out storage and IOPS between many nodes in your cluster.<br>
<br>
Their "mc" S3 client is also pretty awesome - much better than the "aws s3" tools IMHO for interacting with S3.<br>
<br>
Hope this helps,<br>
-pete<br>
<br>
______________________________<wbr>_________________<br>
talk mailing list<br>
<a href="mailto:talk@lists.nycbug.org" target="_blank">talk@lists.nycbug.org</a><br>
<a href="http://lists.nycbug.org/mailman/listinfo/talk" rel="noreferrer" target="_blank">http://lists.nycbug.org/mailma<wbr>n/listinfo/talk</a><br>
</blockquote></div><br>When I mentioned S3-work alikes there are a few things to consider. I know of some commercial ones. The challenge is it is not as simple as just the object store.</div><div class="gmail_extra"><br></div><div class="gmail_extra">For example: <br>using s3 you can setup notifications such that when a new object is created a message is automatically sent to amazon SQS (simple queuing service) <br>using s3 you can setup notifications such that when a new object is created a message is automatically sent to amazon kinesis (distributed high throughput queuing service) <br>You can use amazon lamda to trigger code to execute on s3 events. So for example if I am running a thumbnail service i can build that as s3(raw)->lamba->imagamagik->s3(thumbnail)<br>When using elastic bean stalk you logs can automatically be written to S3</div><div class="gmail_extra">You can serve content directly from an S3 bucket as a web server<br>You can use elastic map reduce to query and s3 bucket using HADOOP!<br><br>On the administrative side you have amazons death-of-1,000,000 paper cuts billing. However if you are using mogilefs lets say you likely need to build your own reporting monitoring. For example if you are attempting to bill-back sections of the business based on usage you need to create that flow and process.</div><div class="gmail_extra"><br></div><div class="gmail_extra">From the standpoint of an software architect you have to look at the solution space. s3(raw)->lamba->imagamagik->s3(thumb nail) is a trivial thing. I have done it without amazon the flow looks like this Web/Appliation server->NAS->mesage on kafka queue <- Stream processor) storm listening on kafka queue running code to do thumb-> write output to other nas location.<br><br>People these days are into loose coupling and micro-services. "amazon lambda" sounds better to most than "eds storm-thumbnail thing". Also companies are not doing there part. I am sure there are hundreds or thousands of people/organizations that have written a scalable thumbnail engine. But if you google this you get stack-overflow howtos</div><div class="gmail_extra"><br></div><div class="gmail_extra"><a href="http://stackoverflow.com/questions/11376315/creating-a-thumbnail-from-an-uploaded-image">http://stackoverflow.com/questions/11376315/creating-a-thumbnail-from-an-uploaded-image</a><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">As much as I dislike some things about amazon they have couple build technologies that make sense, work at scale, WORK WITH EACH OTHER SEAMLESSLY, and are available via APIs for common development platforms.</div></div>