[nycbug-talk] mapreduce, hadoop, and UNIX

[Isaac Levy]

Awesome,

On Aug 25, 2007, at 11:34 AM, Alex Pilosov wrote:
> On Sat, 25 Aug 2007, Isaac Levy wrote:
>> Afterthought addition,
>> On Aug 25, 2007, at 10:48 AM, Isaac Levy wrote:
>>>  From kernel to userland to network, I'm dying to find similar  
>>> works,
>>> any help is much appreciated!
>>
>> E.G.:
>>
>> Distributed computing implementations:
>> - Plan 9?
>> - DragonflyBSD Clustering?
> We all are hoping today to have clusters similar to what VMS had 25  
> years
> ago - fully transparent non-shared memory clustering aka "single  
> system
> image". You don't know, and you don't care which node on the  
> cluster the
> job is running on, and jobs can be migrated to and from nodes  
> depending on
> the load.
>
> For proper clustering, you need a distributed filesystem,  
> distributed lock
> manager, and job distribution engine.
>
> On linux front, closest thing would be MOSIX, which is *almost* that.
> Unfortunately, MOSIX is first and foremost a research project, with
> restrictive licensing and fragmented community (see, openmosix).  
> Today,
> the project to have properly working clusters is openssi.org - I  
> believe
> it is based on openmosix and opengfs.

/sigh

>
> Clustering is hard, comparing to writing an OS - even Linus can do  
> that
> one.

/heh

>
>> Data implementations:
>> - Sun ZFS?
>> - AFS and the like?
>> - RH GFS and the like?
> If you are talking about proper distributed filesystems, they are  
> few and
> far between.
>
> gfs/opengfs
> oracle ocfs
> intermezzo/lustre
> pvfs
> veritas dfs
> sgi cxfs (distributed xfs)

AWESOME list, thanks again Alex.

>
> Distributed filesystems are hard, compared to writing an OS.

Dude, if it wasn't a hard problem, we'd all have been using them for  
many years now :)

Rocket-
.ike