From tekronis at gmail.com Wed Aug 1 19:18:12 2007 From: tekronis at gmail.com (H. G.) Date: Wed, 1 Aug 2007 19:18:12 -0400 Subject: [nycbug-talk] Xen & FreeBSD Message-ID: <60131f920708011618m14ab9b11rfbefad90adc42fb3@mail.gmail.com> I was just wondering, has anyone managed to get FreeBSD running under Xen? Anyone managed to get it to run as dom0, even? In my humble (and worthless) opinion, I think Xen would make an awesome companion to jails, since having the both of them means you have the option of both "lightweight" and "heavyweight" virtualization. So if anyone has managed to get this going, I'd be happy to hear about it. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tekronis at gmail.com Wed Aug 1 19:20:25 2007 From: tekronis at gmail.com (H. G.) Date: Wed, 1 Aug 2007 19:20:25 -0400 Subject: [nycbug-talk] Xen & FreeBSD In-Reply-To: <60131f920708011618m14ab9b11rfbefad90adc42fb3@mail.gmail.com> References: <60131f920708011618m14ab9b11rfbefad90adc42fb3@mail.gmail.com> Message-ID: <60131f920708011620s592f12a8s24614441a0585a5e@mail.gmail.com> On 8/1/07, H. G. wrote: > > I was just wondering, has anyone managed to get FreeBSD running under Xen? > Anyone managed to get it to run as dom0, even? > > In my humble (and worthless) opinion, I think Xen would make an awesome > companion > to jails, since having the both of them means you have the option of both > "lightweight" > and "heavyweight" virtualization. > > So if anyone has managed to get this going, I'd be happy to hear about it. > :) > Sorry about replying to my own post, but I wanted to be clear that I was referring to using a modified kernel, not using Xen + hardware virtualization (which would let you run anything, even Windows, I think). I'm interested in running a modified FreeBSD kernel expressly modified for running on the Xen hypervisor. There was work being done on that in CURRENT, I believe. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ike at lesmuug.org Wed Aug 1 19:28:18 2007 From: ike at lesmuug.org (Isaac Levy) Date: Wed, 1 Aug 2007 19:28:18 -0400 Subject: [nycbug-talk] Xen & FreeBSD In-Reply-To: <60131f920708011618m14ab9b11rfbefad90adc42fb3@mail.gmail.com> References: <60131f920708011618m14ab9b11rfbefad90adc42fb3@mail.gmail.com> Message-ID: <86BBB590-4466-469C-94C1-4F9B5F114692@lesmuug.org> Hi HG, On Aug 1, 2007, at 7:18 PM, H. G. wrote: > I was just wondering, has anyone managed to get FreeBSD running > under Xen? > Anyone managed to get it to run as dom0, even? > > In my humble (and worthless) opinion, I think Xen would make an > awesome companion > to jails, since having the both of them means you have the option > of both "lightweight" > and "heavyweight" virtualization. Well, I would actually like to modify your worthwhile opinion: Having both of them means you have a production service/server and security oriented virtualization engine (jail(8)), and a kernel oriented virtualization engine (xen). Both very powerful tools, respectively applied :) > > > So if anyone has managed to get this going, I'd be happy to hear > about it. :) Me too... :) Rocket- .ike From pete at nomadlogic.org Wed Aug 1 19:47:43 2007 From: pete at nomadlogic.org (Peter Wright) Date: Wed, 1 Aug 2007 16:47:43 -0700 (PDT) Subject: [nycbug-talk] Xen & FreeBSD In-Reply-To: <60131f920708011618m14ab9b11rfbefad90adc42fb3@mail.gmail.com> References: <60131f920708011618m14ab9b11rfbefad90adc42fb3@mail.gmail.com> Message-ID: <50449.160.33.20.11.1186012063.squirrel@webmail.nomadlogic.org> > I was just wondering, has anyone managed to get FreeBSD running under Xen? > Anyone managed to get it to run as dom0, even? > > In my humble (and worthless) opinion, I think Xen would make an awesome > companion > to jails, since having the both of them means you have the option of both > "lightweight" > and "heavyweight" virtualization. > > So if anyone has managed to get this going, I'd be happy to hear about it. > :) I would not expect any para-virt bit's to get committed to the FreeBSD kernel any time soon. There has been some work to get this going, but it was for the 5.3 branch. Even running a full-virt instance (e.x. using an intel vti chipset) does not work cleanly with freebsd. google.com/bsd is your friend here. so - I wouldn't hold my breath on a domU implementation of FreeBSD. getting a dom0 is also probably not going to happen any time soon either - although the NetBSD team has had this working for some time. Here's my two bits (i've been doing alot of work with Xen and Jails while building HPC datacenters spread globally ) - i think both methods have their place. Jailing works great in many environments where something like Xen would be overkill (core IT services come immediately to mind ). One of Xen's strengths is it's ability to set hard caps on memory and cpu usage, along with "live-migration"; although both potentially come with performance cost. so really, i think they compliment each other. I think before any of the *BSDs start tackling something as complicated as Xen i'd like to see better support for things like iSCSI (both hardware and software initiators/targets), FC and PXE. when you get into virtualizing, the ability to decouple your storage from your CPU/RAM is a very important piece of this puzzle. -p -- ~~oO00Oo~~ Peter Wright pete at nomadlogic.org www.nomadlogic.org/~pete 310.869.9459 From skreuzer at exit2shell.com Wed Aug 1 19:47:45 2007 From: skreuzer at exit2shell.com (Steven Kreuzer) Date: Wed, 1 Aug 2007 16:47:45 -0700 Subject: [nycbug-talk] Xen & FreeBSD In-Reply-To: <60131f920708011620s592f12a8s24614441a0585a5e@mail.gmail.com> References: <60131f920708011618m14ab9b11rfbefad90adc42fb3@mail.gmail.com> <60131f920708011620s592f12a8s24614441a0585a5e@mail.gmail.com> Message-ID: <20070801234745.GA26019@clamps.exit2shell.com> On Wed, Aug 01, 2007 at 07:20:25PM -0400, H. G. wrote: > On 8/1/07, H. G. wrote: > > > > I was just wondering, has anyone managed to get FreeBSD running under Xen? > > Anyone managed to get it to run as dom0, even? > > > > In my humble (and worthless) opinion, I think Xen would make an awesome > > companion > > to jails, since having the both of them means you have the option of both > > "lightweight" > > and "heavyweight" virtualization. > > > > So if anyone has managed to get this going, I'd be happy to hear about it. > > :) > > > > Sorry about replying to my own post, but I wanted to be clear that I was > referring to using > a modified kernel, not using Xen + hardware virtualization (which would let > you run anything, > even Windows, I think). I'm interested in running a modified FreeBSD kernel > expressly modified > for running on the Xen hypervisor. There was work being done on that in > CURRENT, I believe. For a while, Kip Macy was working on it, but I am not sure why he stopped. Last time I checked, a college student was working on it for Google's Summer of Code I believe the goal is to allow FreeBSD to run under DomU without SMP or PAE support in 7.0-RELEASE On a somewhat unrelated note, there was a posting today on freebsd-announce about a Jail rc.d script privilege escalation that makes it possible for root inside the jail to overwrite files on the host system outside the jail with arbitrary content. Sorry to derail slighty, but I figured it was worth mentioning Detials can be found at http://marc.info/?l=freebsd-announce&m=118600366231797&w=2 -- Steven Kreuzer http://www.exit2shell.com/~skreuzer From tekronis at gmail.com Wed Aug 1 20:17:15 2007 From: tekronis at gmail.com (H. G.) Date: Wed, 1 Aug 2007 20:17:15 -0400 Subject: [nycbug-talk] Xen & FreeBSD In-Reply-To: <60131f920708011713l11a029c7u96cff96c405d001e@mail.gmail.com> References: <60131f920708011618m14ab9b11rfbefad90adc42fb3@mail.gmail.com> <50449.160.33.20.11.1186012063.squirrel@webmail.nomadlogic.org> <60131f920708011713l11a029c7u96cff96c405d001e@mail.gmail.com> Message-ID: <60131f920708011717l65cbabd5x4d72b04d726b814f@mail.gmail.com> My deepest apologies; my response to Pete's email didn't hit the list. Message below: On 8/1/07, Peter Wright wrote: > > > > I was just wondering, has anyone managed to get FreeBSD running under > Xen? > > Anyone managed to get it to run as dom0, even? > > > > In my humble (and worthless) opinion, I think Xen would make an awesome > > companion > > to jails, since having the both of them means you have the option of > both > > "lightweight" > > and "heavyweight" virtualization. > > > > So if anyone has managed to get this going, I'd be happy to hear about > it. > > :) > > I would not expect any para-virt bit's to get committed to the FreeBSD > kernel any time soon. There has been some work to get this going, but it > was for the 5.3 branch. :-( Thats saddening. so - I wouldn't hold my breath on a domU implementation of FreeBSD. > getting a dom0 is also probably not going to happen any time soon either - > although the NetBSD team has had this working for some time. I was using NetBSD to run Xen 2 guests for a while there. Its pretty nice, and the whole reason I would rather a BSD as a dom0 instead of Linux is because, well, BSD feels much more "solid". (Apologies for my n00bish descriptive terms.) I believe Net supports Xen 3 now too, but I was really interested in running it on FreeBSD. But alas.... :-( Here's my two bits (i've been doing alot of work with Xen and Jails while > building HPC datacenters spread globally ) - i think both methods have > their place. Jailing works great in many environments where something > like Xen would be overkill (core IT services come immediately to mind ). Totally agree here. Although wouldn't it be great just to run an OpenBSD guest just for your infrastructure bits? :) (Although I do recognize that as stupid since you're just increasing layers and attack surface for no good reason. "Just 'cause I can" is as good a reason as any, no? :] ) One of Xen's strengths is it's ability to set hard caps on memory and cpu > usage, along with "live-migration"; although both potentially come with > performance cost. so really, i think they compliment each other. Is there any work to bring these features to jails? I've not been around the BSD universe for very long, but I'm 100% positive that _hordes_ of people would be absolutely thrilled at the idea of being able to jail CPU and memory utilization. I think before any of the *BSDs start tackling something as complicated as > Xen i'd like to see better support for things like iSCSI (both hardware > and software initiators/targets), FC and PXE. when you get into > virtualizing, the ability to decouple your storage from your CPU/RAM is a > very important piece of this puzzle. Agreed. On a related note, haven't mainframes and big iron been working this way already for quite some time? I have no experience with these, but the whole concept or IBM's LPARS (if I'm not mistaken) is connected to this. Its as if tech on mainframes and tech on plain-old COTS desktops and servers are converging upon each other. -------------- next part -------------- An HTML attachment was scrubbed... URL: From compustretch at gmail.com Thu Aug 2 02:22:28 2007 From: compustretch at gmail.com (=?UTF-8?B?c8m5yZDJryDKh3PHncm5b8mf?=) Date: Wed, 01 Aug 2007 23:22:28 -0700 Subject: [nycbug-talk] Xen & FreeBSD In-Reply-To: <60131f920708011620s592f12a8s24614441a0585a5e@mail.gmail.com> References: <60131f920708011618m14ab9b11rfbefad90adc42fb3@mail.gmail.com> <60131f920708011620s592f12a8s24614441a0585a5e@mail.gmail.com> Message-ID: <46B17824.2050100@gmail.com> H. G. wrote: > On 8/1/07, s??? ?s??o? wrote: > > >> Having said that, question back at you-- why is it you would want to run Free as dom0 as opposed to say as DomU on a FreeBSD Dom0 machine ? >> > [SNIP] > as much as I like me my Linux, I would personally prefer a non-Linux box > (say BSD, > or Solaris) as a dom0. Just personal preference. an/listinfo/nylug-talk > I mistyped that, should have been what the reasons are for running FreeBSD as Dom0, as opposed to running it as a DomU on a NetBSD machine. I didn't realise you were ruling out using the Intel/AMD virtualisation extensions. If you discount that approach, I think you're talking a significant wait before FreeBSD catches up. cheers, Forest Mars -- "In theory, theory and practice are exactly the same. In practice, they're completely different." ------------------------------------------------------------------ Switch to Name.Space: http://namespace.org/switch Support new domains & keep free media free! Register yours today! https://secure.name-space.com/registry -----BEGIN PGP SIGNATURE----- Version: PGPsdk version 1.7.1 (C) 1997-1999 Network Associates, Inc. and its affiliated companies. (Diffie-Helman/DSS-only version) iQA/AwUBRkjTLDbz7LySoccvEQJDcQCguZZj4M4kOVOlOX4CtbgR0rppsdovAjra 3RRXIlkdzuYI0YJz4WyvKlTn =MLhk -----END PGP SIGNATURE----- From nikolai at fetissov.org Thu Aug 2 09:45:17 2007 From: nikolai at fetissov.org (nikolai) Date: Thu, 2 Aug 2007 09:45:17 -0400 (EDT) Subject: [nycbug-talk] August 2007 meeting audio Message-ID: <41813.63.66.6.15.1186062317.squirrel@www.geekisp.com> Folks, mp3 of Mark's presentation is online at http://www.fetissov.org/public/nycbug/ -- Nikolai From skreuzer at exit2shell.com Thu Aug 2 14:24:52 2007 From: skreuzer at exit2shell.com (Steven Kreuzer) Date: Thu, 2 Aug 2007 11:24:52 -0700 Subject: [nycbug-talk] FreeBSD Jail Hosting Message-ID: <20070802182452.GA1417@clamps.exit2shell.com> Greetings- I was just curious if anyone has any recommendations on companies that provide FreeBSD jail hosting on the cheap. I would like to have a FreeBSD machine I could have root on to do some userland development on. So far all I can find is companies that sell either Linux or NetBSD virtual machines. Thanks -- Steven Kreuzer http://www.exit2shell.com/~skreuzer From marco at metm.org Thu Aug 2 14:35:14 2007 From: marco at metm.org (marco scoffier) Date: Thu, 02 Aug 2007 14:35:14 -0400 Subject: [nycbug-talk] FreeBSD Jail Hosting In-Reply-To: <20070802182452.GA1417@clamps.exit2shell.com> References: <20070802182452.GA1417@clamps.exit2shell.com> Message-ID: <46B223E2.8090009@metm.org> Steven Kreuzer wrote: > Greetings- > > I was just curious if anyone has any recommendations on companies that provide FreeBSD jail hosting on the cheap. > > I would like to have a FreeBSD machine I could have root on to do some userland development on. So far all I can find is companies that sell either Linux or NetBSD virtual machines. > > Hi Steven, I am using http://www.johncompanies.com/jc_vps.html for a project (thanks Ike). I have had only good experience with them, -- Marco > Thanks > > From ike at lesmuug.org Thu Aug 2 17:35:02 2007 From: ike at lesmuug.org (Isaac Levy) Date: Thu, 2 Aug 2007 17:35:02 -0400 Subject: [nycbug-talk] FreeBSD Jail Hosting In-Reply-To: <46B223E2.8090009@metm.org> References: <20070802182452.GA1417@clamps.exit2shell.com> <46B223E2.8090009@metm.org> Message-ID: Hi steven, All, On Aug 2, 2007, at 2:35 PM, marco scoffier wrote: > Steven Kreuzer wrote: >> Greetings- >> >> I was just curious if anyone has any recommendations on companies >> that provide FreeBSD jail hosting on the cheap. >> >> I would like to have a FreeBSD machine I could have root on to do >> some userland development on. So far all I can find is companies >> that sell either Linux or NetBSD virtual machines. >> >> > Hi Steven, > > I am using > http://www.johncompanies.com/jc_vps.html > for a project (thanks Ike). > > I have had only good experience with them, > Yep- these guys are awesome- I can't recommend anyone better. quick personal sidenotes about them JohnCompanies: 1) They were the only competetitors for my old web hosting company, iMeme- we had the same hosted jail(8) systems- back before all the big ISP's offered all the VPS packages. They were always friendly competition. 2) I met the 'John' owner last summer at DefCon, after my lecture on jailing :) Nice as can be. 3) I've got client systems with them, and simply put, it's a tight ship. Clean stock FreeBSD, as it should be. :) Rocket- .ike From ike at lesmuug.org Fri Aug 3 15:09:36 2007 From: ike at lesmuug.org (Isaac Levy) Date: Fri, 3 Aug 2007 15:09:36 -0400 Subject: [nycbug-talk] bsd beta exam this Sat. in NYC In-Reply-To: <20070730223417.A634@dru.domain.org> References: <20070730223417.A634@dru.domain.org> Message-ID: <600271E1-61EC-4293-AE1A-8F03C121AC75@lesmuug.org> Hi All, Sorry to top-post, thought it would be clearer for a repeat of this message. There are still slots available to take the BSDCert Beta Exam this Saturday! Be there or be square, RSVP to Dru. Best, .ike On Jul 30, 2007, at 10:38 PM, Dru wrote: > > Hi everyone, > > We've finally sorted the date, time and location for the BSDA beta > exam. > Details and registration info are here: > > http://ezine.daemonnews.org/200707/bsdcert_beta_exam.html > > Ike and I will be proctoring. > > The psychometrician has requested at least 30 more beta testers for > her > analysis. If you're available Saturday morning and would like to > assist > the BSD certification effort, consider taking the beta exam. If you're > reading this email, you probably are a good beta testing candidate for > this exam :-) > > Cheers, > > Dru > _______________________________________________ > % NYC*BUG talk mailing list > http://lists.nycbug.org/mailman/listinfo/talk > %Be sure to check out our Jobs and NYCBUG-announce lists > %We meet the first Wednesday of the month > From trish at bsdunix.net Fri Aug 3 17:45:12 2007 From: trish at bsdunix.net (=?utf-8?B?VHJpc2ggTHluY2g=?=) Date: Fri, 3 Aug 2007 21:45:12 +0000 Subject: [nycbug-talk] bsd beta exam this Sat. in NYC In-Reply-To: <600271E1-61EC-4293-AE1A-8F03C121AC75@lesmuug.org> References: <20070730223417.A634@dru.domain.org><600271E1-61EC-4293-AE1A-8F03C121AC75@lesmuug.org> Message-ID: <1967414891-1186177619-cardhu_decombobulator_blackberry.rim.net-368935484-@bxe117.bisx.prod.on.blackberry> Oh well - I'm on my domestic partnership honeymoon this weekend in Massachusetts - so I'm going to have to miss out on this opportunity. -Siobhan Patricia Lynch -- Trish Lynch M: 646-401-1405 H: 201-378-0434 -----Original Message----- From: Isaac Levy Date: Fri, 3 Aug 2007 15:09:36 To:NYCBug List Subject: Re: [nycbug-talk] bsd beta exam this Sat. in NYC Hi All, Sorry to top-post, thought it would be clearer for a repeat of this message. There are still slots available to take the BSDCert Beta Exam this Saturday! Be there or be square, RSVP to Dru. Best, .ike On Jul 30, 2007, at 10:38 PM, Dru wrote: > > Hi everyone, > > We've finally sorted the date, time and location for the BSDA beta > exam. > Details and registration info are here: > > http://ezine.daemonnews.org/200707/bsdcert_beta_exam.html > > Ike and I will be proctoring. > > The psychometrician has requested at least 30 more beta testers for > her > analysis. If you're available Saturday morning and would like to > assist > the BSD certification effort, consider taking the beta exam. If you're > reading this email, you probably are a good beta testing candidate for > this exam :-) > > Cheers, > > Dru > _______________________________________________ > % NYC*BUG talk mailing list > http://lists.nycbug.org/mailman/listinfo/talk > %Be sure to check out our Jobs and NYCBUG-announce lists > %We meet the first Wednesday of the month > _______________________________________________ % NYC*BUG talk mailing list http://lists.nycbug.org/mailman/listinfo/talk %Be sure to check out our Jobs and NYCBUG-announce lists %We meet the first Wednesday of the month From lists at stringsutils.com Fri Aug 3 22:40:46 2007 From: lists at stringsutils.com (Francisco Reyes) Date: Fri, 03 Aug 2007 22:40:46 -0400 Subject: [nycbug-talk] FreeBSD Jail Hosting References: <20070802182452.GA1417@clamps.exit2shell.com> <46B223E2.8090009@metm.org> Message-ID: Isaac Levy writes: > 1) They were the only competetitors for my old web hosting company, > iMeme- we had the same hosted jail(8) systems- back before all the > big ISP's offered all the VPS packages. They were always friendly > competition. http://hub.org has done FreeBSD jails for a while. Have been using them happily for many years. From ike at lesmuug.org Tue Aug 7 07:26:59 2007 From: ike at lesmuug.org (Isaac Levy) Date: Tue, 7 Aug 2007 07:26:59 -0400 Subject: [nycbug-talk] Fwd: www.freebsd.org reachability References: <1FA2A079-6707-462E-9EAA-893418CC6650@lesmuug.org> Message-ID: <128CD653-0D0A-4072-8703-E6A78EF97B07@lesmuug.org> Hi All, Sorry to top/cross-post, I thought it best to preserve the message below. Just a quick heads-up, the freebsd.org primary servers seem to be down- I just saw it come in on the hubs at freebsd.org list. CVS and CVSUP main servers seem to be running fine. -- Observations: It's a strange one, the server seems to hold the http request, (can't tell if it's making a socket request?), then it eventually times out. (An aside, I'd totally bet $10 this is an Apache 2.foobar bug, I've had a server exhibiting the EXACT same thing, in the form of untraceable hangs. The only constant for us, is that when it hangs- we're always in the process of being bot-scanned by some random host, looking for the usual crud- /phpbb /myadmin /blah... We haven't yet seen or replicated whatever URLs are coming next. Time for a new webserver, like lighttpd and tinyhttpd!) Best, .ike Begin forwarded message: > From: Isaac Levy > Date: August 7, 2007 7:11:48 AM EDT > To: Mohacsi Janos > Cc: hubs at freebsd.org > Subject: Re: www.freebsd.org reachability > > Hi Mohacsi, > > Nothing good to contribute, except I've confirmed this from 4 > locations in NYC- all different networks: > 2 datacenters > 1 WiMax office link > 1 home Cable line > > -- > $ curl -v www.freebsd.org > * About to connect() to www.freebsd.org port 80 > * Trying 69.147.83.33... Operation timed out > * couldn't connect to host > * Closing connection #0 > curl: (7) couldn't connect to host > -- > > Best, > .ike > > > > > On Aug 7, 2007, at 7:00 AM, Mohacsi Janos wrote: > >> Dear All, >> There is something wrong with www.freebsd.org. >> >> Recently www.freebsd.org cluster become unreachable. First I could >> reach via IPv6. Then my colleague could not reach it. Then I tested: >> telnet www.freebsd.org 80 >> Trying 2001:4f8:fff6::21... >> telnet: connect to address 2001:4f8:fff6::21: Host is down >> Trying 69.147.83.33... >> telnet: connect to address 69.147.83.33: Operation timed out >> telnet: Unable to connect to remote host >> >> Then after few minutes my connection via IPv6 also become unusable... >> >> >> Any hints? Announced maintenance? >> >> Best Regards, >> >> Janos Mohacsi >> Network Engineer, Research Associate, Head of Network Planning and >> Projects >> NIIF/HUNGARNET, HUNGARY >> Key 70EF9882: DEC2 C685 1ED4 C95A 145F 4300 6F64 7B00 70EF 9882 >> _______________________________________________ >> freebsd-hubs at freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hubs >> To unsubscribe, send any mail to "freebsd-hubs- >> unsubscribe at freebsd.org" >> > From skreuzer at exit2shell.com Tue Aug 7 12:45:45 2007 From: skreuzer at exit2shell.com (Steven Kreuzer) Date: Tue, 7 Aug 2007 09:45:45 -0700 Subject: [nycbug-talk] Matthew Dillon Interview Message-ID: <20070807164545.GA8376@clamps.exit2shell.com> Thought I would share this link. KernelTrap did an interview with Matthew Dillonu. In June of 2003 he forked FreeBSD 4.8 and started DragonFlyBSD, which is one of the more *ahem* controversial BSDs http://kerneltrap.org/node/14116 Enjoy -- Steven Kreuzer http://www.exit2shell.com/~skreuzer From mspitzer at gmail.com Tue Aug 7 21:05:00 2007 From: mspitzer at gmail.com (Marc Spitzer) Date: Tue, 7 Aug 2007 21:05:00 -0400 Subject: [nycbug-talk] (no subject) In-Reply-To: <8c50a3c30707152320w6b8c781amcfc6dfdd9e599c66@mail.gmail.com> References: <594368.56475.qm@web53604.mail.re2.yahoo.com> <8c50a3c30707141345o77786a98t2b46e680cef646f@mail.gmail.com> <24949F7C-FE43-4946-932C-C852C3949E00@2xlp.com> <8c50a3c30707141856y3dff5d32g9f6d815681718f26@mail.gmail.com> <7FFBCF5D-AC32-47B9-A81C-76B4E488BB27@2xlp.com> <8c50a3c30707152320w6b8c781amcfc6dfdd9e599c66@mail.gmail.com> Message-ID: <8c50a3c30708071805r1bedf2ddp6c11cd13c4b54c75@mail.gmail.com> Just to thow some more gas on the fire: http://taosecurity.blogspot.com/2007/08/black-hat-usa-2007-round-up-part-1.html marc On 7/16/07, Marc Spitzer wrote: > On 7/15/07, Jonathan Vanasco wrote: > > > > On Jul 14, 2007, at 9:56 PM, Marc Spitzer wrote: > > > > > It is part of defense in depth. Face it people screw up all the time, > > > myself included, and having having 2 ways to be "safe" is better then > > > 1. Also things like -3 TV's should be checked by unit tests before it > > > ever gets to production. I think that most problems are caused by a > > > lack of discipline not ignorance or malice. Especially when deadline > > > loom people can be pressured into doing things that may be less then > > > good. > > > > > > As long as it is a backup, and not relied upon, its fine. once you > > introduce it as something peopel rely on, it makes for bad coding. > > > > since you're also introducing something that is standardized here, > > you also start opening yourself up to new security holes-- and you > > have hackers not only looking to exploit your webapp, but mod_sec or > > whatever other standard firewall app they figure you're running and > > can look for known exploits on. > > > > those apps are great to bolster a strong defense, but as the only > > defense its irresponsible. > > > > I think I did mention unit tests. But you only test, and code for, > things you think can happen. And things that can not happen happen > all the time in computers. The question is how much paranoia is > prudent and that is something that changes from person to person and > project to project. > > I also did not say they were the only defense just that it should be > added to the existing defenses. The idea that you will not have > exploitable code in your system is foolish, web servers have bugs > after all. What you will have is code that you think is safe, good > code/app/webserver *and* properly configured, but sooner or later you > will find out you were wrong or you wont find out which could be much > worse. And yes firewalls have had exploitable code also. But the > Idea is to have a layered defense here and I have just recommended > adding a layer not lessing the other layers. > > marc > -- > Freedom is nothing but a chance to be better. > Albert Camus > -- Freedom is nothing but a chance to be better. Albert Camus From mspitzer at gmail.com Thu Aug 9 13:42:11 2007 From: mspitzer at gmail.com (Marc Spitzer) Date: Thu, 9 Aug 2007 13:42:11 -0400 Subject: [nycbug-talk] Cambridge Researcher Breaks OpenBSD Systrace Message-ID: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> Link: http://it.slashdot.org/it/07/08/09/138224.shtml An anonymous reader writes "University of Cambridge researcher Robert Watson has published a paper at the First USENIX Workshop On Offensive Technology in which he describes serious vulnerabilities in OpenBSD's Systrace, Sudo, Sysjail, the TIS GSWTK framework, and CerbNG. The technique is also effective against many commercially available anti-virus systems. His slides include sample exploit code that bypasses access control, virtualization, and intrusion detection in under 20 lines of C code consisting solely of memcpy() and fork(). Sysjail has now withdrawn their software, recommending against any use, and NetBSD has disabled Systrace by default in their upcoming release." -- Freedom is nothing but a chance to be better. Albert Camus From pete at nomadlogic.org Thu Aug 9 17:07:18 2007 From: pete at nomadlogic.org (Peter Wright) Date: Thu, 9 Aug 2007 14:07:18 -0700 (PDT) Subject: [nycbug-talk] Cambridge Researcher Breaks OpenBSD Systrace In-Reply-To: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> References: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> Message-ID: <23956.160.33.20.11.1186693638.squirrel@webmail.nomadlogic.org> > Link: http://it.slashdot.org/it/07/08/09/138224.shtml > > An anonymous reader writes "University of Cambridge researcher Robert > Watson has published a paper at the First USENIX Workshop On Offensive > Technology in which he describes serious vulnerabilities in OpenBSD's > Systrace, Sudo, Sysjail, the TIS GSWTK framework, and CerbNG. The > technique is also effective against many commercially available > anti-virus systems. His slides include sample exploit code that > bypasses access control, virtualization, and intrusion detection in > under 20 lines of C code consisting solely of memcpy() and fork(). > Sysjail has now withdrawn their software, recommending against any > use, and NetBSD has disabled Systrace by default in their upcoming > release." > i read the paper this morning - it's quite interesting read actually: http://www.watson.org/~robert/2007woot/2007usenixwoot-exploitingconcurrency.pdf http://www.watson.org/~robert/2007woot/ -p -- ~~oO00Oo~~ Peter Wright pete at nomadlogic.org www.nomadlogic.org/~pete 310.869.9459 From carton at Ivy.NET Thu Aug 9 18:55:07 2007 From: carton at Ivy.NET (Miles Nordin) Date: Thu, 09 Aug 2007 18:55:07 -0400 Subject: [nycbug-talk] Cambridge Researcher Breaks OpenBSD Systrace In-Reply-To: <23956.160.33.20.11.1186693638.squirrel@webmail.nomadlogic.org> (Peter Wright's message of "Thu, 9 Aug 2007 14:07:18 -0700 (PDT)") References: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> <23956.160.33.20.11.1186693638.squirrel@webmail.nomadlogic.org> Message-ID: I find it a bit disgusting that he understood the issues in 2002 but is only now five years later turning them into a security crisis. http://docs.freebsd.org/cgi/getmsg.cgi?fetch=493173+0+archive/2002/freebsd-hackers/20020602.freebsd-hackers and it's not like he just recently became interested in this. so, I think it'll be interesting to see if there is some particular reason he picked this moment for his paper, some reason which becomes clear over the next few months. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: From spork at bway.net Thu Aug 9 19:03:02 2007 From: spork at bway.net (Charles Sprickman) Date: Thu, 9 Aug 2007 19:03:02 -0400 (EDT) Subject: [nycbug-talk] Cambridge Researcher Breaks OpenBSD Systrace In-Reply-To: References: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> <23956.160.33.20.11.1186693638.squirrel@webmail.nomadlogic.org> Message-ID: On Thu, 9 Aug 2007, Miles Nordin wrote: > I find it a bit disgusting that he understood the issues in 2002 but > is only now five years later turning them into a security crisis. > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=493173+0+archive/2002/freebsd-hackers/20020602.freebsd-hackers > > and it's not like he just recently became interested in this. so, I > think it'll be interesting to see if there is some particular reason > he picked this moment for his paper, some reason which becomes clear > over the next few months. Change of heart? More research? He says this in the link above about systrace in regards to bringing it to FreeBSD: "So I would suggest someone port it over, and write a cool paper on what they ran into, because there are probably a lot of interesting problems. And at the end of the day, it works really well, it would be a great thing to add to our growing arsenol of security features." While the OpenBSD aspect is interesting, I think that the greatest impact is in the windows world where apparently most common resident virus scanners use similar tricks (the syscall wrapping) to do "on access" scanning. If someone finds an easy way to hack most existing windows AV software, that's a big deal. He did (does?) work for a company that produced such software I believe... Charles From nycbug at cyth.net Thu Aug 9 19:05:35 2007 From: nycbug at cyth.net (Ray Lai) Date: Thu, 9 Aug 2007 19:05:35 -0400 Subject: [nycbug-talk] Cambridge Researcher Breaks OpenBSD Systrace In-Reply-To: References: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> <23956.160.33.20.11.1186693638.squirrel@webmail.nomadlogic.org> Message-ID: <20070809230558.GT9310@cybertron.cyth.net> On Thu, Aug 09, 2007 at 06:55:07PM -0400, Miles Nordin wrote: > I find it a bit disgusting that he understood the issues in 2002 but > is only now five years later turning them into a security crisis. > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=493173+0+archive/2002/freebsd-hackers/20020602.freebsd-hackers > > and it's not like he just recently became interested in this. so, I > think it'll be interesting to see if there is some particular reason > he picked this moment for his paper, some reason which becomes clear > over the next few months. Perhaps he has only recently developed a proof-of-concept? -Ray- From alex at pilosoft.com Thu Aug 9 19:22:09 2007 From: alex at pilosoft.com (Alex Pilosov) Date: Thu, 9 Aug 2007 19:22:09 -0400 (EDT) Subject: [nycbug-talk] Cambridge Researcher Breaks OpenBSD Systrace In-Reply-To: Message-ID: On Thu, 9 Aug 2007, Miles Nordin wrote: > I find it a bit disgusting that he understood the issues in 2002 but > is only now five years later turning them into a security crisis. > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=493173+0+archive/2002/freebsd-hackers/20020602.freebsd-hackers > > and it's not like he just recently became interested in this. so, I > think it'll be interesting to see if there is some particular reason he > picked this moment for his paper, some reason which becomes clear over > the next few months. I believe it is because doing security research inherently takes more time than making up conspiracy theories. -alex From pete at nomadlogic.org Thu Aug 9 19:38:34 2007 From: pete at nomadlogic.org (Peter Wright) Date: Thu, 9 Aug 2007 16:38:34 -0700 (PDT) Subject: [nycbug-talk] Cambridge Researcher Breaks OpenBSD Systrace In-Reply-To: References: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> <23956.160.33.20.11.1186693638.squirrel@webmail.nomadlogic.org> Message-ID: <30280.160.33.20.11.1186702714.squirrel@webmail.nomadlogic.org> > I find it a bit disgusting that he understood the issues in 2002 but > is only now five years later turning them into a security crisis. > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=493173+0+archive/2002/freebsd-hackers/20020602.freebsd-hackers > > and it's not like he just recently became interested in this. so, I > think it'll be interesting to see if there is some particular reason > he picked this moment for his paper, some reason which becomes clear > over the next few months. you know who robert watson is right? so what's up - posting his specific concerns on a public mailing list, then going to Cambridge from the private sector - spending time to create a proper academic paper for a conference (which is in its inaugural year WOOT '07) is considered underhanded, or cause for suspicion of ulterior motives? got it. -p -- ~~oO00Oo~~ Peter Wright pete at nomadlogic.org www.nomadlogic.org/~pete 310.869.9459 From nycbug-list at 2xlp.com Thu Aug 9 19:59:04 2007 From: nycbug-list at 2xlp.com (Jonathan Vanasco) Date: Thu, 9 Aug 2007 19:59:04 -0400 Subject: [nycbug-talk] Cambridge Researcher Breaks OpenBSD Systrace In-Reply-To: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> References: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> Message-ID: On Aug 9, 2007, at 1:42 PM, Marc Spitzer wrote: > An anonymous reader writes "University of Cambridge researcher Robert > Watson has published a paper at the First USENIX Workshop On Offensive I'm just wondering if he contacted OpenBSD , "Systrace, Sudo, Sysjail, the TIS GSWTK framework, and CerbNG" first, and worked out a disclosure timeframe I couldn't find that information anywhere. Personally, I find that the difference between wanting to offer a security researcher a "THANK YOU!!!!" or a 'F**k You for disclosing holes in software before I had time to patch my system' From tekronis at gmail.com Thu Aug 9 21:20:30 2007 From: tekronis at gmail.com (H. G.) Date: Thu, 9 Aug 2007 21:20:30 -0400 Subject: [nycbug-talk] BSD Embedded Solutions for Commodity Home Routers Message-ID: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> Greets folks. By now, most of us have heard about projects like OpenWRT, Sveasoft and other Linux solutions for running embedded on commodity home routers, like the infamous Linksys WRT54G. I haven't heard of any equivelent BSD offerings, and the most I've found through Web searches has been "WifiBSD", which doesn't appear to have seen activity since 2005. Do there really exist no solutions in this space? If there are, anyone have practical experience w/ them? Purely curious. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pete at nomadlogic.org Fri Aug 10 00:43:29 2007 From: pete at nomadlogic.org (Peter Wright) Date: Thu, 9 Aug 2007 21:43:29 -0700 (PDT) Subject: [nycbug-talk] BSD Embedded Solutions for Commodity Home Routers In-Reply-To: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> References: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> Message-ID: <51235.76.167.183.64.1186721009.squirrel@webmail.nomadlogic.org> > Greets folks. > > By now, most of us have heard about projects like OpenWRT, Sveasoft and > other Linux > solutions for running embedded on commodity home routers, like the > infamous > Linksys > WRT54G. I haven't heard of any equivelent BSD offerings, and the most > I've > found > through Web searches has been "WifiBSD", which doesn't appear to have seen > activity > since 2005. Do there really exist no solutions in this space? If there > are, anyone have > practical experience w/ them? > these are the two that i believe are most popular: http://m0n0.ch/wall/ http://www.pfsense.org/ i'm personally more familiar with m0n0wall, this would probably be most helpful for you: http://doc.m0n0.ch/handbook/hardware-wireless.html it has captive portal etc and works great with Soekris machines. pfsense - as the name implies - uses the OpenBSD pf packet filter. -pete -- ~~oO00Oo~~ Peter Wright pete at nomadlogic.org www.nomadlogic.org/~pete 310.869.9459 From chris at chrisclymer.com Fri Aug 10 00:55:58 2007 From: chris at chrisclymer.com (Chris Clymer) Date: Fri, 10 Aug 2007 00:55:58 -0400 Subject: [nycbug-talk] BSD Embedded Solutions for Commodity Home Routers In-Reply-To: <51235.76.167.183.64.1186721009.squirrel@webmail.nomadlogic.org> References: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> <51235.76.167.183.64.1186721009.squirrel@webmail.nomadlogic.org> Message-ID: <62E251CA-85E3-4433-94DF-4340643C775A@chrisclymer.com> On Aug 10, 2007, at 12:43 AM, Peter Wright wrote: > >> Greets folks. >> >> By now, most of us have heard about projects like OpenWRT, >> Sveasoft and >> other Linux >> solutions for running embedded on commodity home routers, like the >> infamous >> Linksys >> WRT54G. I haven't heard of any equivelent BSD offerings, and the >> most >> I've >> found >> through Web searches has been "WifiBSD", which doesn't appear to >> have seen >> activity >> since 2005. Do there really exist no solutions in this space? If >> there >> are, anyone have >> practical experience w/ them? >> > > > these are the two that i believe are most popular: > http://m0n0.ch/wall/ > http://www.pfsense.org/ > > i'm personally more familiar with m0n0wall, this would probably be > most > helpful for you: > http://doc.m0n0.ch/handbook/hardware-wireless.html > > it has captive portal etc and works great with Soekris machines. > pfsense > - as the name implies - uses the OpenBSD pf packet filter. > > -pete > > > -- > ~~oO00Oo~~ > Peter Wright > pete at nomadlogic.org > www.nomadlogic.org/~pete > 310.869.9459 > _______________________________________________ > % NYC*BUG talk mailing list > http://lists.nycbug.org/mailman/listinfo/talk > %Be sure to check out our Jobs and NYCBUG-announce lists > %We meet the first Wednesday of the month I've used both, and pfsense quite a bit. I'm a huge fan. Its got oodles more features, like solid openvpn support, CARP, and the capability for a lot more expansion. And it can be fully managed through a web interface, so you can give it to your cisco or windows guys and they're comfortable. From spork at bway.net Fri Aug 10 01:45:38 2007 From: spork at bway.net (Charles Sprickman) Date: Fri, 10 Aug 2007 01:45:38 -0400 (EDT) Subject: [nycbug-talk] BSD Embedded Solutions for Commodity Home Routers In-Reply-To: <62E251CA-85E3-4433-94DF-4340643C775A@chrisclymer.com> References: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> <51235.76.167.183.64.1186721009.squirrel@webmail.nomadlogic.org> <62E251CA-85E3-4433-94DF-4340643C775A@chrisclymer.com> Message-ID: On Fri, 10 Aug 2007, Chris Clymer wrote: > On Aug 10, 2007, at 12:43 AM, Peter Wright wrote: > >> >>> Greets folks. >>> >>> By now, most of us have heard about projects like OpenWRT, >>> Sveasoft and >>> other Linux >>> solutions for running embedded on commodity home routers, like the >>> infamous >>> Linksys >>> WRT54G. I haven't heard of any equivelent BSD offerings, and the >>> most >>> I've >>> found >>> through Web searches has been "WifiBSD", which doesn't appear to >>> have seen >>> activity >>> since 2005. Do there really exist no solutions in this space? If >>> there >>> are, anyone have >>> practical experience w/ them? >>> >> >> >> these are the two that i believe are most popular: >> http://m0n0.ch/wall/ >> http://www.pfsense.org/ >> >> i'm personally more familiar with m0n0wall, this would probably be >> most >> helpful for you: >> http://doc.m0n0.ch/handbook/hardware-wireless.html >> >> it has captive portal etc and works great with Soekris machines. >> pfsense >> - as the name implies - uses the OpenBSD pf packet filter. >> >> -pete >> >> >> -- >> ~~oO00Oo~~ >> Peter Wright >> pete at nomadlogic.org >> www.nomadlogic.org/~pete >> 310.869.9459 > > I've used both, and pfsense quite a bit. I'm a huge fan. Its got > oodles more features, like solid openvpn support, CARP, and the > capability for a lot more expansion. And it can be fully managed > through a web interface, so you can give it to your cisco or windows > guys and they're comfortable. > Ditto here, I started using it at home and then put it at two client sites as well. Very nice setup. Getting an IPSEC VPN going from one client's home to office was easier than falling off a log. C From ike at lesmuug.org Fri Aug 10 09:53:02 2007 From: ike at lesmuug.org (Isaac Levy) Date: Fri, 10 Aug 2007 09:53:02 -0400 Subject: [nycbug-talk] BSD Embedded Solutions for Commodity Home Routers In-Reply-To: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> References: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> Message-ID: Hi H.G., All, Much of my answer is opinion, please take it with a grain of salt; On Aug 9, 2007, at 9:20 PM, H. G. wrote: > Greets folks. > > By now, most of us have heard about projects like OpenWRT, Sveasoft > and other Linux > solutions for running embedded on commodity home routers, like the > infamous Linksys > WRT54G. Here's one thing about your question that I believe other posters missed- (for good reason). Basically, all the "Linksys-hack" projects are aimed at hardware which has to be reverse-engineered. The WRT54G is *not* an open hardware platform, and running anything other than the Linksys- supplied hardware is not supported. So, for sanity's sake, it's fairly insane to run this kind of thing in any environment where it *needs to work*. (e.g. a small office, home office even...). These projects, although created by brilliant people, are hobbyist toys- and nothing I'd ever dream of intentionally leaving on a client's office T1. So while all the hardware may be 'cheap' and plentiful, it's not really meant to last. (I mean last in this way: people came up with all kinds of bar-code reader software for the old WiredMag 'Cue-Cat', and those things are long-gone too, right...) http://www.freebsd.org/cgi/url.cgi?ports/misc/cuecat/pkg-descr -- Philosophical Rant: These are paths I, (and it seems most *BSD people), don't care to take- reverse-engineering proprietary and/or crappy hardware usually leads to a dead end- unless it brings HUGE short-term gains or benefits. Tearing into wonkie hardware (like a WRT54G) leads to trouble down the road, for example, all wireless cards are not alike... (e.g. so when a 2.4ghz phone conflict bites, will your card/software combo cope? Or what about little tweaks that can mean a lot, like signal strength settings? Or what about altq on your nics, when bittorrent gets out of hand?) The risks in minutia becomes nauseating with the cheapo gear- you never know what you *really* have to work with. Regarding the uses- "but this is just for my home network, and I like hacking stuff". If you really like hacking network stuff, just load up a good UNIX (OpenBSD or something) on a soekris or a PC even, and get to hacking. If you want a Web-Gui and you just want the network to *run*, so you can *do and hack other things*, drop MonoWall or PFSense into your network. The Linksys hack stuff is simply a bad middle-space between the two frames of mind. You can't do *really* powerful stuff with it, (like you could using a raw UNIX with good networking tools)- and you aren't going to be running a "tinker-free" network, (the kind you'd trust to deploy for a client). > I haven't heard of any equivelent BSD offerings, and the most I've > found > through Web searches has been "WifiBSD", which doesn't appear to > have seen activity > since 2005. Do there really exist no solutions in this space? If > there are, anyone have > practical experience w/ them? > > Purely curious. However, to constructively respond to my own whining above, there's plenty of VERY inexpensive hardware which runs PfSense and MonoWALL- both of which I've successfully deployed at multiple client offices, and in my home office network- (both are extremely reliable, I might add). MonoWall has a faster user interface by far on small hardwares, but PFSense is far more advanced and flexible (it provides shell access, for example)- I use them both. On Aug 10, 2007, at 12:43 AM, Peter Wright wrote: > http://m0n0.ch/wall/ > http://www.pfsense.org/ Great hardware is here: http://www.soekris.com/ (my favorite stuff, 4801's are like Yellow Cabs to me- standard, tough, simple.) http://www.pcengines.ch/order1.php?c=4 http://m0n0.ch/wall/hardware.php Only complaint/annoyance: The wireless drivers are different for older MonoWall and PFSense- the best rule of thumb: Stick to Atheros for the PFSense boxes and you'll be VERY happy with them. Newer MonoWALL runs the Atheros cards too, but I've not used that combo- Lucent and Prism cards rock the older MonoWall releases. -- And one more thing, in case you don't want to drop a dime to get nice little hardwares... Both MonoWall and PFSense can RUN FROM THE INSTALL CD, and optionally, the config can be written to a disk (floppy drive, old USB key in your desk drawer, etc...) This is how I first started using them- I slapped some ethernet NICS into old 350mhz machines I found on the street, and started screwing around with it. That was nearly 3 or 4 years ago, (yikes time flies), and now that I use it in client offices- this has REALLY come in handy since. A lightning storm took out a bunch of equipment in an office, (even though it was all well protected), and their 2 Soekris boards got fried. I came in that night, and had them up and running for 8am, by piecing together old PC crap- (one of the routers was just a motherboard and a CDRom drive, with the NICS sticking up in the air off the board). It held up great until we could replace the soekris boards, and the office ran without work-day downtime. And aside from being called in late that night, I didn't loose my mind setting this up- it was all simple and intuitive- (I even used my backed-up config files, I mean this was almost brainless). End point, I can't imagine there are any similar 'success stories' for people using the hacked Linksys gear. If there are, I'd bet the sysadmins spent *way* more time tinkering with esoteric hardware workaround crud than they did configuring the network... /end ike .02?, sry. perhaps way too long for this topic... Rocket- .ike From mspitzer at gmail.com Fri Aug 10 12:07:19 2007 From: mspitzer at gmail.com (Marc Spitzer) Date: Fri, 10 Aug 2007 12:07:19 -0400 Subject: [nycbug-talk] Cambridge Researcher Breaks OpenBSD Systrace In-Reply-To: References: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> Message-ID: <8c50a3c30708100907hf8b261dra7954a04506ac172@mail.gmail.com> On 8/9/07, Jonathan Vanasco wrote: > > On Aug 9, 2007, at 1:42 PM, Marc Spitzer wrote: > > > An anonymous reader writes "University of Cambridge researcher Robert > > Watson has published a paper at the First USENIX Workshop On Offensive > > I'm just wondering if he contacted OpenBSD , "Systrace, Sudo, > Sysjail, the TIS GSWTK framework, and CerbNG" first, and worked out a > disclosure timeframe > >From what I read on the slides, have not done the paper yet, I do not think you *can* fix it. What he was pointing out was a massive design flaw that can not go away given the current architecture of the systems in question. IE one of the fundamental and necessary assumptions of this system(atomicy of calling function) does not exist in the real world as the kernels in question stand. The interesting thing is minix3 and dragonfly may be better suited to defending against this problem as they make much more use of message passing for moving stuff around. > I couldn't find that information anywhere. > > Personally, I find that the difference between wanting to offer a > security researcher a "THANK YOU!!!!" or a 'F**k You for disclosing > holes in software before I had time to patch my system' This is just not patchable, its a problem if you use these things. marc > > > > _______________________________________________ > % NYC*BUG talk mailing list > http://lists.nycbug.org/mailman/listinfo/talk > %Be sure to check out our Jobs and NYCBUG-announce lists > %We meet the first Wednesday of the month > -- Freedom is nothing but a chance to be better. Albert Camus From ike at lesmuug.org Fri Aug 10 12:12:55 2007 From: ike at lesmuug.org (Isaac Levy) Date: Fri, 10 Aug 2007 12:12:55 -0400 Subject: [nycbug-talk] Cambridge Researcher Breaks OpenBSD Systrace In-Reply-To: References: <8c50a3c30708091042l2803d2fexd67066c664eb1dc9@mail.gmail.com> Message-ID: <05CB96E3-97E0-48A7-99D6-6783159284AC@lesmuug.org> Wow, On Aug 9, 2007, at 7:22 PM, Alex Pilosov wrote: > On Thu, 9 Aug 2007, Miles Nordin wrote: > >> I find it a bit disgusting that he understood the issues in 2002 but >> >> some reason which becomes clear over >> the next few months. > I believe it is because doing security research inherently takes > more time > than making up conspiracy theories. > > -alex Alex is 100% right here. On Aug 9, 2007, at 7:38 PM, Peter Wright wrote: > so what's up - posting his specific concerns on a public mailing list, > then going to Cambridge from the private sector - spending time to > create > a proper academic paper for a conference (which is in its inaugural > year > WOOT '07) is considered underhanded, or cause for suspicion of > ulterior > motives? > > got it. > > -p Pete is 100% right with the clear intention, based on the order of events. -- Working at extremely deep levels with anything, (like this new work), means a person doesn't get the luxurious time involved to install every new app/tool that comes along, even those lauded by the security community- let alone test them all in any comprehensive manner. Who stands to loose from this situation? Don't we all stand to gain? On Aug 9, 2007, at 7:59 PM, Jonathan Vanasco wrote: > On Aug 9, 2007, at 1:42 PM, Marc Spitzer wrote: > >> An anonymous reader writes "University of Cambridge researcher Robert >> Watson has published a paper at the First USENIX Workshop On >> Offensive > > I'm just wondering if he contacted OpenBSD , "Systrace, Sudo, > Sysjail, the TIS GSWTK framework, and CerbNG" first, and worked out a > disclosure timeframe > > I couldn't find that information anywhere. Well, I just Google'd around for it myself, and didn't find anything- BUT, judging from the reactions from the various groups you mention, direct open-disclosure seems the best route here. I mean really, what is Kristaps Johnson (Sysjail author and generally cool person), going to do with advanced knowledge of this vulnerability? He'd have to take time out of his work/life to fully comprehend or replicate it, and then sit on his hands until everyone knows? Additionally, to keep the sanity here, Sysjail as an example has not yet ever been advertised as production software, and it's very very new. > > Personally, I find that the difference between wanting to offer a > security researcher a "THANK YOU!!!!" or a 'F**k You for disclosing > holes in software before I had time to patch my system' On Zero-Daze: This is also a fundamental problem which is not trivially resolvable. - Therefore, there is no patch in sight on the horizon. This isn't a windows or vendor sploit'. - How you propose we patch our systems without realizing "hey, systrace isn't enabled anymore!" (and thereby giving everybody in the world X days to slam systrace, possibly succeeding in the same or similar exploits). Or even just saying "systrace is broken, everyone will know why in X days" is silly. It's not like proprietary and locked down binaries- we all have the source code here. This is relatively new software, which a small fraction of very technical people have deployed, most of whom lived with running systems far before it existed. -- In context, there's no F**k you involved here, software gets cracked- period. Any *important* system must have a diverse backup plan, for every critical component, if it's is going to survive the cracks- (like what's everyone's plan for the dreadful day that OpenSSH gets hosed? [aside from running for the hills]). Another aspect of this particular issue, systrace itself is relatively new, (although the idea is not). It's complicated enough, just in it's implementation and use. It's implications on a running system are then additionally complicated, by magnitudes of increasing complexity. These kinds of massive-scale security tools take years to mature, and more years to get refined to come close to meeting their objectives. - With that stated, my longwinded point, is that anyone who's crying because they didn't have a failure and replacement plan for a critical software they use, isn't really taking the issue seriously. Depending on any singularity is a risk. To end this thought, a related news-quote snippet: Bruce Schneier's Black Hat Keynote: "Bruce reiterated his ideas of the "security consumer" who asks "is it worth it?" when deciding whether or not to wear a bullet-proof vest when walking out his front door." From a terrific Black-Hat overview by Richard Bejtlich: http://taosecurity.blogspot.com/2007/08/black-hat-usa-2007-round- up-part-2.html Bruce, and Richard, are right on these days- IMHO. -- Additionally, nobody has discussed this angle on this list: "Systrace is dead, long live Systrace" + Systrace isn't dead because of this issue(?), it just has to be re- thought from scratch in the scope of it's implementation. A successful example of this is jail(2)/jail(8) - it was a response to chroot(2) exploits, remember? HOWEVER, jail(2) was not the only answer, jailing only works because of the audits made to the rest of the operating system- a tedious and holistic approach, which has served everyone well. Jailing required nearly every system call to be audited, (and thank goodness the TrustedBSD project just so happened to be doing a whole lot of that...) Similarly, the problems with systrace are larger architectural issues, which can likely be resolved with continued (and arduous) work- and possible consequences and tradeoffs for other kernel features. That stuff has to be figured out- and that work is slow and hard. These were some lessons learned from the TrustedBSD project, a fork to explore these kinds patterns in secure development. (and Robert Watson was a huge part of that project, btw). /me spouting .04? on this one Rocket- .ike From carton at Ivy.NET Fri Aug 10 18:28:34 2007 From: carton at Ivy.NET (Miles Nordin) Date: Fri, 10 Aug 2007 18:28:34 -0400 Subject: [nycbug-talk] BSD Embedded Solutions for Commodity Home Routers In-Reply-To: (Isaac Levy's message of "Fri, 10 Aug 2007 09:53:02 -0400") References: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> Message-ID: >>>>> "il" == Isaac Levy writes: il> Basically, all the "Linksys-hack" projects are aimed at il> hardware which has to be reverse-engineered. The WRT54G is il> *not* an open hardware platform, and running anything other il> than the Linksys- supplied hardware is not supported. well, (1) maybe it's sort-of not ``open.'' But because of the GPL, you do get documentation for it in the form of source code, except when Linksys and other vendors violate the GPL. The missing piece is the Broadcom wireless driver, which does have to be reverse-engineered, thanks to Linus's rather generous (to proprietary hardware manufacturers) interpretation of the GPL w.r.t. kernel modules. The old OpenWRT just includes the original binary module. I think the new 2.6-based OpenWRT may have the reverse-engineered GPL Broadcom driver. (2) it is absolutely not true that OpenWRT runs only on Linksys. They run on a huge list of these half-closed <$100 extremely-low-wattage platforms. They also run on open Soekris-like platforms like the ones sold by magicbox.pl. (3) In spite of the fact they're completely ``open'', BSD doesn't run on magicbox.pl hardware, either. Why? BSD has no FLASH-friendly filesystem (you have to use CF, which is too expensive and power-hungry for these platforms), and it also seems to be buggier than Linux on not-i386, since all this Soekris pfsense u.s.w. stuff you will find is all i386-only. I'm no Linux zealot. I'm not trying to maintain any Linux-based firewalls any time soon. It looks like a disaster to me. But I really don't think you can call this situation anything but a missing feature for our camp. Linksys is open ``enough'' for a decent port, and in the 'L' linux-friendly version of their router they have delivered a very consistent platform over many years. And magicbox.pl is truly open as are many other similar low-cost embedded boards. but we don't have the FLASH filesystem, so we can't run on embedded devices with small NOR FLASH chips, except as a lame stateless kernel-and-FFSramdisk image. we need a CF card. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: From nycbug at cyth.net Fri Aug 10 19:08:29 2007 From: nycbug at cyth.net (Ray Lai) Date: Fri, 10 Aug 2007 19:08:29 -0400 Subject: [nycbug-talk] BSD Embedded Solutions for Commodity Home Routers In-Reply-To: References: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> Message-ID: <20070810230852.GF9310@cybertron.cyth.net> On Fri, Aug 10, 2007 at 06:28:34PM -0400, Miles Nordin wrote: > >>>>> "il" == Isaac Levy writes: > > il> Basically, all the "Linksys-hack" projects are aimed at > il> hardware which has to be reverse-engineered. The WRT54G is > il> *not* an open hardware platform, and running anything other > il> than the Linksys- supplied hardware is not supported. > > well, > > (1) maybe it's sort-of not ``open.'' But because of the GPL, you do > get documentation for it in the form of source code, except when > Linksys and other vendors violate the GPL. Source code is no replacement for documentation. How would you extend the software once it's no longer maintained by the vender? How would you know if something is a bug or a feature in the original "documentation"? > The missing piece is the Broadcom wireless driver, which does > have to be reverse-engineered, thanks to Linus's rather generous > (to proprietary hardware manufacturers) interpretation of the GPL > w.r.t. kernel modules. The old OpenWRT just includes the > original binary module. I think the new 2.6-based OpenWRT may > have the reverse-engineered GPL Broadcom driver. > > (2) it is absolutely not true that OpenWRT runs only on Linksys. > They run on a huge list of these half-closed <$100 > extremely-low-wattage platforms. They also run on open > Soekris-like platforms like the ones sold by magicbox.pl. I think the criticism is on the hardware, not OpenWRT, but correct me if I'm wrong Ike. =) > (3) In spite of the fact they're completely ``open'', BSD doesn't run > on magicbox.pl hardware, either. Why? BSD has no FLASH-friendly > filesystem (you have to use CF, which is too expensive and > power-hungry for these platforms), and it also seems to be > buggier than Linux on not-i386, since all this Soekris pfsense > u.s.w. stuff you will find is all i386-only. Define "buggier than Linux on not-i386". > I'm no Linux zealot. I'm not trying to maintain any Linux-based > firewalls any time soon. It looks like a disaster to me. But I > really don't think you can call this situation anything but a > missing feature for our camp. Linksys is open ``enough'' for a > decent port, and in the 'L' linux-friendly version of their > router they have delivered a very consistent platform over many > years. And magicbox.pl is truly open as are many other similar > low-cost embedded boards. but we don't have the FLASH > filesystem, so we can't run on embedded devices with small NOR > FLASH chips, except as a lame stateless kernel-and-FFSramdisk > image. we need a CF card. If there is any hardware documentation on magicbox.pl I cannot find it. -Ray- From lavalamp at spiritual-machines.org Fri Aug 10 21:00:30 2007 From: lavalamp at spiritual-machines.org (Brian A. Seklecki) Date: Fri, 10 Aug 2007 21:00:30 -0400 (EDT) Subject: [nycbug-talk] BSD Embedded Solutions for Commodity Home Routers In-Reply-To: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> References: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> Message-ID: <20070810205913.B1800@arbitor.digitalfreaks.org> How embedded to you need to get? ARM/MIPS / 8MB RAM / MTD-NAND Flash etc? Its tough to get that tiny, I've found, and we dont have a flash FS yet. It could and will be done though. Check my project out. ~BAS On Thu, 9 Aug 2007, H. G. wrote: > Greets folks. > > By now, most of us have heard about projects like OpenWRT, Sveasoft and > other Linux > solutions for running embedded on commodity home routers, like the infamous > Linksys > WRT54G. I haven't heard of any equivelent BSD offerings, and the most I've > found > through Web searches has been "WifiBSD", which doesn't appear to have seen > activity > since 2005. Do there really exist no solutions in this space? If there > are, anyone have > practical experience w/ them? > > Purely curious. > > Thanks. > l8* -lava (Brian A. Seklecki - Pittsburgh, PA, USA) http://www.spiritual-machines.org/ "Guilty? Yeah. But he knows it. I mean, you're guilty. You just don't know it. So who's really in jail?" ~Maynard James Keenan -------------- next part -------------- _______________________________________________ % NYC*BUG talk mailing list http://lists.nycbug.org/mailman/listinfo/talk %Be sure to check out our Jobs and NYCBUG-announce lists %We meet the first Wednesday of the month From carton at Ivy.NET Fri Aug 10 23:50:58 2007 From: carton at Ivy.NET (Miles Nordin) Date: Fri, 10 Aug 2007 23:50:58 -0400 Subject: [nycbug-talk] BSD Embedded Solutions for Commodity Home Routers In-Reply-To: <20070810230852.GF9310@cybertron.cyth.net> (Ray Lai's message of "Fri, 10 Aug 2007 19:08:29 -0400") References: <60131f920708091820u71edee94i212e50092a21cd3b@mail.gmail.com> <20070810230852.GF9310@cybertron.cyth.net> Message-ID: >>>>> "rl" == Ray Lai writes: rl> Source code is no replacement for documentation. How would rl> you extend the software once it's no longer maintained by the rl> vender? You would get documentation from the chip manufacturers for each chip on the board, and resolve your questions about how chips are connected together and how to boot by groveling through the Linux source code and writing documentation as you go. rl> I think the criticism is on the hardware, not OpenWRT, but rl> correct me if I'm wrong Ike. =) Ike stated that OpenWRT runs only on Linksys which is emphatically untrue. If you talk to Felix he will tell you there are many OpenWRT platforms, _most_ of them _more_ interesting than Linksys, including platforms like magicbox that are designed _just_ to run OpenWRT. rl> Define "buggier than Linux on not-i386". I own zero i386 machines. I've been running NetBSD/alpha since 1999 as my main shell box. I've run NetBSD on alpha, macppc, mac68k, sun3, sparc, sparc64, dreamcast, hpcmips, and pmax, and FreeBSD/sparc64. I started using BSD in the first place because NetBSD/alpha sucked so much less than Linux for alpha. so, absolutely YMMV, but I'm really not talking out of my ass when I say this. Honestly, these days there is a lot of bit-rot. OpenWRT really does work a lot less quirkily in my experience than, say, NetBSD/hpcmips or NetBSD/mac68k. now, NetBSD is still much better-factored than Linux, still has much cleaner code, still has the excellent cross-build architecture. but Linux works and works well on a wider variety of cheaper boards that are actually still manufactured and obtainable by me, rather than the obsolete boards, >$1000 ``evaluation kits,'' or what you guys actually use, which are double-price larger higher-wattage boards that are designed to emulate a PeeCee because BSD can't handle running on anything else---I'd almost rather use Linux than stoop to these boards, which to me seem like the height of vanity. It is just a PeeCee, but if your customer found out that it's just a PeeCee they would be unimpressed so you disguise it by changing the size and shape---you spend more money for a slower quirkier machine with a crappy ethernet chip. rl> If there is any hardware documentation on magicbox.pl I cannot rl> find it. can't find any on dell.com, either. This is half a serious criticism, and half kind of a silly argument. Yes, it would be good to have some documentation from Magicbox that's not in Polish, but if you were seriously doing a port I think you need to at least ask them, not just look at the web site---since Linux is the only software they support, I expect them to be friendly. They have shipped samples of the 1.0 board to Felix in September last year for porting OpenWRT (that's how I heard about them in the first place), and it is one of his favorite platforms. Anyway, you would only need a tiny amount of documentation form Magicbox themselves, if any. It is a single-chip computer, with literally a single chip plus RAM and ROM, nothing else, soldered to the board, so most of your porting documentation comes from here: https://www.amcc.com/MyAMCC/jsp/public/productDetail/relatedDocuments.jsp?productID=PPC440EP However none of this changes the fact that Linux, not BSD, owns the <$100 space because BSD doesn't have any good filesystem for NOR FLASH, or (harder still) for NAND FLASH without the block-replacement logic built into the CF card which many of the cheap Linux boards lack. it's interesting that Magicbox added a CF slot between the 1.0 and 2.0 designs. Maybe they are already planning on a BSD port. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: From george at ceetonetechnology.com Sat Aug 11 20:17:07 2007 From: george at ceetonetechnology.com (George Rosamond) Date: Sat, 11 Aug 2007 20:17:07 -0400 Subject: [nycbug-talk] ical for future NYCBUG meetings Message-ID: <46BE5183.4000804@ceetonetechnology.com> We now have a link to a downloadable or http-accessible ical file for upcoming NYCBUG meetings. http://www.nycbug.org/index.php?NAV=iCal It's in the "web apps menu". . . George From bonsaime at gmail.com Sun Aug 12 14:20:35 2007 From: bonsaime at gmail.com (Jesse Callaway) Date: Sun, 12 Aug 2007 14:20:35 -0400 Subject: [nycbug-talk] ical for future NYCBUG meetings In-Reply-To: <46BE5183.4000804@ceetonetechnology.com> References: <46BE5183.4000804@ceetonetechnology.com> Message-ID: On 8/11/07, George Rosamond wrote: > We now have a link to a downloadable or http-accessible ical file for > upcoming NYCBUG meetings. > > http://www.nycbug.org/index.php?NAV=iCal > > It's in the "web apps menu". . . > > George > This is hot shit pie. I plugged the URL into sunbird and it works beautifully. Thanks to whoever made it! Often forget all of these groups second new moon of the leap year frequencies. -jesse From george at ceetonetechnology.com Tue Aug 14 16:08:06 2007 From: george at ceetonetechnology.com (George Rosamond) Date: Tue, 14 Aug 2007 16:08:06 -0400 Subject: [nycbug-talk] [Fwd: Andy Oram and Karl Fogel: free software advocates in NYC on August 30] Message-ID: <46C20BA6.2020805@ceetonetechnology.com> Forwarded email from O'Reilly for anyone interested. . . Could you let the various user groups that you work with (BSD user group, Ruby-NYC, NYC Python User Group) know about an interesting event coming up on the evening of Thursday, August 30? About a dozen people are gathering for an informal dinner and chat. The out-of-town guests will be: Karl Fogel, well-known free software contributor and advocate, author of Producing Open Source Software (which won a JOLT award), and currently a campaigner on copyright reform Andy Oram, O'Reilly editor (responsible for many Linux and Perl books,and most recently the best-selling Beautiful Code, which Karl also contributed to), and frequent writer on technology and policy I'm putting a more formal description below. Anyone interested in attending should keep in touch with "Seth Johnson" , because the number of people who attend will determine where it's located. --------------------------------------------------------------------------- An informal gathering with Andy Oram, Editor at O'Reilly Media and Karl Fogel, free software developer and advocate ---------------------------------------- ...to discuss open content and collaboration, funding models for content, and the economics of digital networks (and numerous other things) Date: Thursday, August 30th, 2007 Time: 8:00 pm Place: TBD Discussion will be kicked off by topics such as: * Why do people contribute free content, and what can society or businesses do to increase participation and quality? * In an age where many people can't afford books or don't want to read them, how do people learn technical skills? * What characteristics distinguish the arts in digital media from twentieth-century and pre-twentieth-century media? * How can writers earn a living from content in an age of free redistribution? * How will new stages of high-bandwidth networking be funded (can advertising carry the cost?) Biography: Andy Oram is an editor at O'Reilly Media, a highly respected book publisher and technology information provider. An employee of the company since 1992, Andy specializes in free software projects and software engineering. His work for O'Reilly includes the first books ever released by a U.S. publisher on Linux, the 2001 title Peer-to-Peer, and the recent best-seller Beautiful Code. Andy is also a member of Computer Professionals for Social Responsibility and writes frequently on policy issues and trends related to the Internet and to technical innovation and its effects on society. Copyrights, trademarks, and patents, business aspects of open source, and telecom issues are among the topics covered in his articles at: http://praxagora.com/andyo/professional/article.html He is currently doing research on free, online, technical documention, along with experiments in new tools, as described at: http://praxagora.com/community_documentation/ An article he wrote about art on the Internet, titled "Characteristics of new media in the Internet age," is maintained as a wiki at: http://commons.oreilly.com/wiki/index.php/Chapters_for_Characteristics_of_new_media From spork at bway.net Tue Aug 14 20:15:23 2007 From: spork at bway.net (Charles Sprickman) Date: Tue, 14 Aug 2007 20:15:23 -0400 (EDT) Subject: [nycbug-talk] PAM gurus? Message-ID: Hi all, I'm having some issues getting telnet + pam_mysql playing well together. If I connect to the host with a modern telnet client with SRA auth, all is well. If I use a standard telnet client, the pam auth fails. What I think is happening is that with SRA auth, telnetd is doing the auth (ie: /etc/pam.d/telnetd pam config applies). But when SRA is not being used, the login tasks are passed to /bin/login. I'm trying to get the /etc/pam.d/login pam setup right, but the default config has an option that I'm not finding in the Free/NetBSD PAM handbooks/manpages. # auth auth required pam_nologin.so no_warn auth sufficient pam_self.so no_warn -->>auth include system auth required pam_mysql.so host=... I see "required, sufficient, requisite, binding, optional" in the manpage, but I'm lost on what "include" is or how it affects the other lines. If I remove it, things work. I'm worried about just what it did though... Anyone know anything about this? And do I assume "system" means direct auth via the standard passwd db? Thanks, Charles ___ Charles Sprickman NetEng/SysAdmin Bway.net - New York's Best Internet - www.bway.net spork at bway.net - 212.655.9344 From tbackman at childrensaidsociety.org Wed Aug 15 10:47:45 2007 From: tbackman at childrensaidsociety.org (Thomas Backman) Date: Wed, 15 Aug 2007 10:47:45 -0400 Subject: [nycbug-talk] Will Backman (from BSDTalk podcast) is interviewed Message-ID: <000401c7df4b$3e44dc70$7c9710ac@Berlioz> The other Backman brother, Will Backman, producer of the BSDTalk podcast, is interviewed on the LinuxReality podcast episode 74. Will talks about himself and BSD. http://www.linuxreality.com/ --------------------------------------------------------------------------- Thomas Backman Software Developer/Analyst The Children's Aid Society IT Department mailto: tbackman at childrensaidsociety.org "Help! I'm a bug." - Calvin No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.476 / Virus Database: 269.11.19/953 - Release Date: 8/14/2007 5:19 PM From compustretch at gmail.com Wed Aug 15 16:39:47 2007 From: compustretch at gmail.com (forest mars) Date: Wed, 15 Aug 2007 13:39:47 -0700 Subject: [nycbug-talk] PAM gurus? In-Reply-To: References: Message-ID: On 8/14/07, Charles Sprickman wrote: I see "required, sufficient, requisite, binding, optional" in the manpage, > but I'm lost on what "include" is or how it affects the other lines. If I > remove it, things work. I'm worried about just what it did though... > > Anyone know anything about this? And do I assume "system" means direct > auth via the standard passwd db? > > Since your message is timestamped 19 hrs ago I'm assuming you're up to speed on this; include simply tells PAM to include all lines of given type from the configuration file given as an argument to the specified control-flag. It is what it says it is, an include, so that you can *WORM* your config info. As for your 'system' module, when called as your config path/file, that would seem like an alternate syntax for 'system-auth' which is often/usually paired with 'include' to call your system's default authentication rules. hth, Forest Mars -- "In theory, theory and practice are exactly the same. In practice, they're completely different." ------------------------------------------------------------------ Switch to Name.Space: http://namespace.org/switch Support new domains & keep free media free! Register yours today! https://secure.name-space.com/registry -----BEGIN PGP SIGNATURE----- Version: PGPsdk version 1.7.1 (C) 1997-1999 Network Associates, Inc. and its affiliated companies. (Diffie-Helman/DSS-only version) iQA/AwUBRkjTLDbz7LySoccvEQJDcQCguZZj4M4kOVOlOX4CtbgR0rppsdovAjra 3RRXIlkdzuYI0YJz4WyvKlTn =MLhk -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From spork at bway.net Fri Aug 17 01:58:27 2007 From: spork at bway.net (Charles Sprickman) Date: Fri, 17 Aug 2007 01:58:27 -0400 (EDT) Subject: [nycbug-talk] PAM gurus? In-Reply-To: References: Message-ID: On Wed, 15 Aug 2007, forest mars wrote: > On 8/14/07, Charles Sprickman wrote: > > I see "required, sufficient, requisite, binding, optional" in the manpage, >> but I'm lost on what "include" is or how it affects the other lines. If I >> remove it, things work. I'm worried about just what it did though... >> >> Anyone know anything about this? And do I assume "system" means direct >> auth via the standard passwd db? >> > Since your message is timestamped 19 hrs ago I'm assuming you're up to speed > on this; include simply tells PAM to include all lines of given type from > the configuration file given as an argument to the specified control-flag. > It is what it says it is, an include, so that you can *WORM* your config > info. I must be blind, I kept looking in the "control-flag" section of the manpage for "include", but it's up at the top: Entries in per-service policy files must be of one of the two forms below: function-class control-flag module-path [arguments ...] function-class include other-service-name I don't want any *WORMS* though. > As for your 'system' module, when called as your config path/file, that > would seem like an alternate syntax for 'system-auth' which is often/usually > paired with 'include' to call your system's default authentication rules. In short it meant include the definition in /etc/pam.d/system Still looking for a good way to figure out what program calls what pam service. Some are quite obvious, others are not, and some general pam debugging info would be really helpful. I know there are flags for each service, but I'd like something for the whole enchilada; ie: "program foobuzz asks for auth from grobknob service". Thanks, Charles > hth, > > Forest Mars > -- > "In theory, theory and practice are exactly the same. > In practice, they're completely different." > ------------------------------------------------------------------ > Switch to Name.Space: http://namespace.org/switch > Support new domains & keep free media free! Register yours today! > https://secure.name-space.com/registry > > -----BEGIN PGP SIGNATURE----- > Version: PGPsdk version 1.7.1 (C) 1997-1999 Network Associates, Inc. > and its affiliated companies. (Diffie-Helman/DSS-only version) > > iQA/AwUBRkjTLDbz7LySoccvEQJDcQCguZZj4M4kOVOlOX4CtbgR0rppsdovAjra > 3RRXIlkdzuYI0YJz4WyvKlTn > =MLhk > -----END PGP SIGNATURE----- > From ike at lesmuug.org Sat Aug 18 09:51:35 2007 From: ike at lesmuug.org (Isaac Levy) Date: Sat, 18 Aug 2007 09:51:35 -0400 Subject: [nycbug-talk] show of hands regarding spamd Message-ID: Hey All, OK- so I *know* some folks had to have experienced today's repeated friday afternoon spam deluge. If you don't know what I'm talking about, then I WANT TO KNOW ONE THING- are you using spamd? I'm serious here. The last 2 fridays, me, and several clients offices I support- across 5 different mail service providers encompasing two-dozen domain names with active email accounts, (and I mean TOTALLY different MTA's); have experienced massive spam blasts- with a serious spike in spam between about 3pm and 9pm- with between hundreds (and some individuals THOUSANDS) of spam messages making it right to the inbox. The only thing they all share in common, no spamd use in any of these accounts. -- I know we all deal with spam, but I'm mostly thinking of moving towards spamd frontends for these accounts, because the providers spam fighting techniques are toppling faster and faster as time goes on. I kindof feel like it'll be a great deal of time before spamd becomes useless against spam, but I'm curious about the realities for people using it. What do folks here think? (And I'm willing to even accept a 'just quit whining ike' response). Best, .ike From jonathan at kc8onw.net Sat Aug 18 11:21:13 2007 From: jonathan at kc8onw.net (Jonathan Stewart) Date: Sat, 18 Aug 2007 11:21:13 -0400 Subject: [nycbug-talk] show of hands regarding spamd In-Reply-To: References: Message-ID: <46C70E69.7070308@kc8onw.net> Isaac Levy wrote: > Hey All, > > OK- so I *know* some folks had to have experienced today's repeated > friday afternoon spam deluge. > > If you don't know what I'm talking about, then I WANT TO KNOW ONE > THING- are you using spamd? I'm using spamd and never noticed anything but then I run a very small personal domain so chances are I never got hit. This [1] may be a sign of getting hit but if so spamd did stop it all. Jonathan [1] > Aug 10 03:56:54 server spamd[1063]: 59.93.214.18: connected (1/0) > Aug 10 03:57:08 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:08 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:09 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:10 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:11 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:12 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:13 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:13 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:14 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:15 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:16 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:17 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:17 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:18 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:19 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:20 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:20 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:21 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:22 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:23 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:23 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:24 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:25 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:26 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:26 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:27 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:28 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:28 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:29 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:30 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:31 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:32 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:32 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:33 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:34 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:35 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:36 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:36 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:37 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:38 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:39 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:40 server spamd[1063]: (GREY) 59.93.214.18: -> > Aug 10 03:57:40 server spamd[1063]: 59.93.214.18: disconnected after 46 seconds. From nycbug at cyth.net Sat Aug 18 13:40:58 2007 From: nycbug at cyth.net (Ray Lai) Date: Sat, 18 Aug 2007 13:39:58 -0401 Subject: [nycbug-talk] show of hands regarding spamd In-Reply-To: References: Message-ID: <20070818174021.GI9310@cybertron.cyth.net> On Sat, Aug 18, 2007 at 09:51:35AM -0400, Isaac Levy wrote: > Hey All, > > OK- so I *know* some folks had to have experienced today's repeated > friday afternoon spam deluge. Duh, what spam deluge? > If you don't know what I'm talking about, then I WANT TO KNOW ONE > THING- are you using spamd? Yup! -Ray- From dan at langille.org Sat Aug 18 15:41:49 2007 From: dan at langille.org (Dan Langille) Date: Sat, 18 Aug 2007 15:41:49 -0400 Subject: [nycbug-talk] show of hands regarding spamd In-Reply-To: References: Message-ID: <46C7133D.8159.10B7444F@dan.langille.org> On 18 Aug 2007 at 9:51, Isaac Levy wrote: > If you don't know what I'm talking about, then I WANT TO KNOW ONE > THING- are you using spamd? I don't know what you are talking about. I use spamd. > What do folks here think? (And I'm willing to even accept a 'just > quit whining ike' response). Stop whining. You sound like Ike. ;) -- Dan Langille - http://www.langille.org/ Available for hire: http://www.freebsddiary.org/dan_langille.php From okan at demirmen.com Sun Aug 19 11:01:39 2007 From: okan at demirmen.com (Okan Demirmen) Date: Sun, 19 Aug 2007 11:01:39 -0400 Subject: [nycbug-talk] show of hands regarding spamd In-Reply-To: References: Message-ID: <20070819150139.GZ2888@clam.khaoz.org> On Sat 2007.08.18 at 09:51 -0400, Isaac Levy wrote: > If you don't know what I'm talking about, then I WANT TO KNOW ONE > THING- are you using spamd? of course, yes (and one or two other things in the mix). stuff can, and does, get through spamd; but the amount is so small, i typically don't really care what i use as a second line of defense. From pete at nomadlogic.org Mon Aug 20 13:29:29 2007 From: pete at nomadlogic.org (Peter Wright) Date: Mon, 20 Aug 2007 10:29:29 -0700 (PDT) Subject: [nycbug-talk] Git? Message-ID: <30995.160.33.20.11.1187630969.squirrel@webmail.nomadlogic.org> hi - anyone using Git as their SCM in production environments out there? i've been using it to interface with a project that has standardized on it and I seem pretty happy with it. I'm specifically interested in seeing if anyone has been using it in a fairly large scale environment (and no - the linux kernel tree does not count as i'm pretty sure linus et. al are not on this list ;). one of my pet-peeves with SVN is the way in which data is stored on disk prevents us from using a NAS as our repository. thx! -pete Git: http://git.or.cz/ -- ~~oO00Oo~~ Peter Wright pete at nomadlogic.org www.nomadlogic.org/~pete 310.869.9459 From jonathan at kc8onw.net Mon Aug 20 14:49:53 2007 From: jonathan at kc8onw.net (Jonathan Stewart) Date: Mon, 20 Aug 2007 14:49:53 -0400 Subject: [nycbug-talk] Git? In-Reply-To: <30995.160.33.20.11.1187630969.squirrel@webmail.nomadlogic.org> References: <30995.160.33.20.11.1187630969.squirrel@webmail.nomadlogic.org> Message-ID: <46C9E251.2070004@kc8onw.net> Peter Wright wrote: > one of my pet-peeves with SVN is the way in which data is stored on > disk prevents us from using a NAS as our repository. As I remember it FSFS would work just fine on a NAS and BDB has come a long way as well. Was there some specific reason it wouldn't work? As far as git is concerned I don't know anything about it but if your looking around Mercurial is supposed to be pretty good. I don't know how well it scales though. Jonathan From pete at nomadlogic.org Mon Aug 20 15:02:08 2007 From: pete at nomadlogic.org (Peter Wright) Date: Mon, 20 Aug 2007 12:02:08 -0700 (PDT) Subject: [nycbug-talk] Git? In-Reply-To: <46C9E251.2070004@kc8onw.net> References: <30995.160.33.20.11.1187630969.squirrel@webmail.nomadlogic.org> <46C9E251.2070004@kc8onw.net> Message-ID: <42119.160.33.20.11.1187636528.squirrel@webmail.nomadlogic.org> > Peter Wright wrote: >> one of my pet-peeves with SVN is the way in which data is stored on >> disk prevents us from using a NAS as our repository. > > As I remember it FSFS would work just fine on a NAS and BDB has come a > long way as well. Was there some specific reason it wouldn't work? > quickly: http://svn.haxx.se/dev/archive-2007-03/0067.shtml note(1): we have some SVN hackers working here, and they readily admit that NAS is not suitable note(2): yes we tried, and verified that it was suitable ;) -pete -- ~~oO00Oo~~ Peter Wright pete at nomadlogic.org www.nomadlogic.org/~pete 310.869.9459 From jonathan at kc8onw.net Mon Aug 20 15:22:45 2007 From: jonathan at kc8onw.net (Jonathan Stewart) Date: Mon, 20 Aug 2007 15:22:45 -0400 Subject: [nycbug-talk] Git? In-Reply-To: <42119.160.33.20.11.1187636528.squirrel@webmail.nomadlogic.org> References: <30995.160.33.20.11.1187630969.squirrel@webmail.nomadlogic.org> <46C9E251.2070004@kc8onw.net> <42119.160.33.20.11.1187636528.squirrel@webmail.nomadlogic.org> Message-ID: <46C9EA05.7020405@kc8onw.net> Peter Wright wrote: >> Peter Wright wrote: >>> one of my pet-peeves with SVN is the way in which data is stored on >>> disk prevents us from using a NAS as our repository. >> As I remember it FSFS would work just fine on a NAS and BDB has come a >> long way as well. Was there some specific reason it wouldn't work? >> > > quickly: > http://svn.haxx.se/dev/archive-2007-03/0067.shtml > > note(1): we have some SVN hackers working here, and they readily admit > that NAS is not suitable > > note(2): yes we tried, and verified that it was suitable ;) > > -pete Okay, so it's a performance issue not a "doesn't work at all" issue. That I can see, especially after seeing the open/close numbers in that email. Jonathan From ike at lesmuug.org Tue Aug 21 10:40:05 2007 From: ike at lesmuug.org (Isaac Levy) Date: Tue, 21 Aug 2007 10:40:05 -0400 Subject: [nycbug-talk] show of hands regarding spamd In-Reply-To: <20070819150139.GZ2888@clam.khaoz.org> References: <20070819150139.GZ2888@clam.khaoz.org> Message-ID: On Aug 19, 2007, at 11:01 AM, Okan Demirmen wrote: > stuff can, and > does, get through spamd; but the amount is so small, i typically don't > really care what i use as a second line of defense. Thanks everybody! I'm devising a strategy for a spamd MX gateway now... Rocket- .ike From dan at langille.org Thu Aug 23 08:05:58 2007 From: dan at langille.org (Dan Langille) Date: Thu, 23 Aug 2007 08:05:58 -0400 Subject: [nycbug-talk] is cvsup freebsd.nycbug.org refreshing? Message-ID: <46CD3FE6.14757.28D5BAF7@dan.langille.org> Hi, I just cvup'd my NYI server from freebsd.nycbug.org. This cvsup did pickup a fix for a recent rsync vulnerability. On other systems, I have been able to get the fix from other cvsup servers. This is just a heads up in case there's a hidden problem. -- Dan Langille - http://www.langille.org/ Available for hire: http://www.freebsddiary.org/dan_langille.php From dan at langille.org Thu Aug 23 08:11:49 2007 From: dan at langille.org (Dan Langille) Date: Thu, 23 Aug 2007 08:11:49 -0400 Subject: [nycbug-talk] is cvsup freebsd.nycbug.org refreshing? In-Reply-To: <46CD3FE6.14757.28D5BAF7@dan.langille.org> References: <46CD3FE6.14757.28D5BAF7@dan.langille.org> Message-ID: <46CD4145.26679.28DB160F@dan.langille.org> On 23 Aug 2007 at 8:05, Dan Langille wrote: > Hi, > > I just cvup'd my NYI server from freebsd.nycbug.org. This cvsup did > pickup a fix for a recent rsync vulnerability. On other systems, I > have been able to get the fix from other cvsup servers. > > This is just a heads up in case there's a hidden problem. False positive. I think the problem was my local copy of the portaudit database. Beers on me... when you get to Jupiter... -- Dan Langille - http://www.langille.org/ Available for hire: http://www.freebsddiary.org/dan_langille.php From pete at nomadlogic.org Thu Aug 23 12:32:45 2007 From: pete at nomadlogic.org (Peter Wright) Date: Thu, 23 Aug 2007 09:32:45 -0700 (PDT) Subject: [nycbug-talk] is cvsup freebsd.nycbug.org refreshing? In-Reply-To: <46CD4145.26679.28DB160F@dan.langille.org> References: <46CD3FE6.14757.28D5BAF7@dan.langille.org> <46CD4145.26679.28DB160F@dan.langille.org> Message-ID: <64551.160.33.20.11.1187886765.squirrel@webmail.nomadlogic.org> > On 23 Aug 2007 at 8:05, Dan Langille wrote: > >> Hi, >> >> I just cvup'd my NYI server from freebsd.nycbug.org. This cvsup did >> pickup a fix for a recent rsync vulnerability. On other systems, I >> have been able to get the fix from other cvsup servers. >> >> This is just a heads up in case there's a hidden problem. > > False positive. > > I think the problem was my local copy of the portaudit database. > Beers on me... when you get to Jupiter... > ahh the wonders of not checking your email - problems get fixed before you realize they might exist :) thanks for the heads up though dan, always appreciated! -pete -- ~~oO00Oo~~ Peter Wright pete at nomadlogic.org www.nomadlogic.org/~pete 310.869.9459 From ike at lesmuug.org Sat Aug 25 10:48:21 2007 From: ike at lesmuug.org (Isaac Levy) Date: Sat, 25 Aug 2007 10:48:21 -0400 Subject: [nycbug-talk] mapreduce, hadoop, and UNIX Message-ID: Hey All, This *may* be a thread which is aimed at the wrong list, but I thought it's appropriate. Feel free to yell at me to take the thread elsewhere. -- Pretext: I've been working on a personal project lately which has landed me with some home-scale monster number-crunching tasks, as well as some quickly scaling massive storage requirements. (Fun fun fun, I'm trying to scan and OCR my personal book collection, and I'm getting scared out of my mind now that I'm making some headway :) Anyhow, I've been looking really closely at Google's MapReduce system/ algorithm spec, which seems to be at the heart of how they make their massive clusters work. This seems to be the current hot topic in macho computing. http://en.wikipedia.org/wiki/MapReduce Fun for UNIX folks, Rob Pike made an awk-like utility/language called 'Sawzall' which uses Google's internal MapReduce API- I think it's pretty interesting. http://labs.google.com/papers/sawzall.html With that, I've also found that Yahoo is putting massive support into an implementation of the MapReduce idea, Open Source as a part of the Apache Project: http://lucene.apache.org/hadoop/ There's other implementations cropping up all over, it seems. Like I said, it's the current buzz... With all that, many of us have noticed that Google is good at scaling patterns in computing- to bastardize their whole tech. operation, it's their big trick. When I say scaling patterns, I mean: applying classical computing paradigms and methodology at wild scales. (E.G. with the Google Filesystem, it's simply a filesystem where the disk blocks exist as network resources, etc... You see what I mean with scale?) -- Question: Anyhow, I'm looking for more patterns in this MapReduce stuff, because I'm simply not one to dive headfirst into 'buzz-tech'. With that, aside from the map, and reduce, functions found in many programming languages, http://en.wikipedia.org/wiki/Map_%28higher-order_function%29 http://en.wikipedia.org/wiki/Fold_%28higher-order_function%29 can anyone shed some light on similar prior works in distributed computing and RPC systems which are 'old classics' in UNIX? These distributed computing problems simply can't be new. To be really straight, what I'm getting at, is why is this more or less useful than intelligently piping commands through ssh? What about older UNIX rpc mechanisms? Aren't there patterns in even kernel source code which match this work, or are even computationally more sophisticated and advanced? From kernel to userland to network, I'm dying to find similar works, any help is much appreciated! Rocket- .ike --- p.s. If anyone is interested in book-scanning stuff, Google happens to currently host the Open Source 'tessarect' project, a very nice OCR software- a very clean command-line application for OCR processing. (I was actually inspired to start all this stuff based on how much fun I had screwing around with tessarect). Apache licence, /me shrugs. http://code.google.com/p/tesseract-ocr/ From ike at lesmuug.org Sat Aug 25 10:59:38 2007 From: ike at lesmuug.org (Isaac Levy) Date: Sat, 25 Aug 2007 10:59:38 -0400 Subject: [nycbug-talk] mapreduce, hadoop, and UNIX In-Reply-To: References: Message-ID: <0480763E-2ADB-417D-A632-BCA94310EA41@lesmuug.org> Afterthought addition, On Aug 25, 2007, at 10:48 AM, Isaac Levy wrote: > From kernel to userland to network, I'm dying to find similar works, > any help is much appreciated! E.G.: Distributed computing implementations: - Plan 9? - DragonflyBSD Clustering? Data implementations: - Sun ZFS? - AFS and the like? - RH GFS and the like? -- But what I'm *really* looking for are patterns in various kernel or userland implementation patterns... Rocket- .ike From alex at pilosoft.com Sat Aug 25 11:22:20 2007 From: alex at pilosoft.com (Alex Pilosov) Date: Sat, 25 Aug 2007 11:22:20 -0400 (EDT) Subject: [nycbug-talk] mapreduce, hadoop, and UNIX In-Reply-To: Message-ID: On Sat, 25 Aug 2007, Isaac Levy wrote: > can anyone shed some light on similar prior works in distributed > computing and RPC systems which are 'old classics' in UNIX? These > distributed computing problems simply can't be new. > > To be really straight, what I'm getting at, is why is this more or less > useful than intelligently piping commands through ssh? What about older > UNIX rpc mechanisms? Aren't there patterns in even kernel source code > which match this work, or are even computationally more sophisticated > and advanced? mapreduce is most of all, an API. Unix is contrary to idea of APIs (everything is a stream of bytes). mapreduce isn't really rocket science by any means, see below. > From kernel to userland to network, I'm dying to find similar works, > any help is much appreciated! Similar things to look at: PVM and MPI - these are APIs for non-shared memory, message passing, distributed computation. They are an order of magnitude more involved than mapreduce - they are much more generic. mapreduce can be easily implemented using PVM but not vice versa. mapreduce is optimal for 'embarassingly parallel' jobs - ones that are very easy to paralellize. There hasn't been much research into that - its been a solved problem 40 years ago. -alex From alex at pilosoft.com Sat Aug 25 11:34:20 2007 From: alex at pilosoft.com (Alex Pilosov) Date: Sat, 25 Aug 2007 11:34:20 -0400 (EDT) Subject: [nycbug-talk] mapreduce, hadoop, and UNIX In-Reply-To: <0480763E-2ADB-417D-A632-BCA94310EA41@lesmuug.org> Message-ID: On Sat, 25 Aug 2007, Isaac Levy wrote: > Afterthought addition, > > On Aug 25, 2007, at 10:48 AM, Isaac Levy wrote: > > > From kernel to userland to network, I'm dying to find similar works, > > any help is much appreciated! > > E.G.: > > Distributed computing implementations: > - Plan 9? > - DragonflyBSD Clustering? We all are hoping today to have clusters similar to what VMS had 25 years ago - fully transparent non-shared memory clustering aka "single system image". You don't know, and you don't care which node on the cluster the job is running on, and jobs can be migrated to and from nodes depending on the load. For proper clustering, you need a distributed filesystem, distributed lock manager, and job distribution engine. On linux front, closest thing would be MOSIX, which is *almost* that. Unfortunately, MOSIX is first and foremost a research project, with restrictive licensing and fragmented community (see, openmosix). Today, the project to have properly working clusters is openssi.org - I believe it is based on openmosix and opengfs. Clustering is hard, comparing to writing an OS - even Linus can do that one. > Data implementations: > - Sun ZFS? > - AFS and the like? > - RH GFS and the like? If you are talking about proper distributed filesystems, they are few and far between. gfs/opengfs oracle ocfs intermezzo/lustre pvfs veritas dfs sgi cxfs (distributed xfs) Distributed filesystems are hard, compared to writing an OS. -alex From ike at lesmuug.org Sat Aug 25 11:47:11 2007 From: ike at lesmuug.org (Isaac Levy) Date: Sat, 25 Aug 2007 11:47:11 -0400 Subject: [nycbug-talk] mapreduce, hadoop, and UNIX In-Reply-To: References: Message-ID: <6AF26939-39E9-4395-9850-DE9DE70181A9@lesmuug.org> On Aug 25, 2007, at 11:22 AM, Alex Pilosov wrote: > On Sat, 25 Aug 2007, Isaac Levy wrote: > >> can anyone shed some light on similar prior works in distributed >> computing and RPC systems which are 'old classics' in UNIX? These >> distributed computing problems simply can't be new. >> >> To be really straight, what I'm getting at, is why is this more or >> less >> useful than intelligently piping commands through ssh? What about >> older >> UNIX rpc mechanisms? Aren't there patterns in even kernel source >> code >> which match this work, or are even computationally more sophisticated >> and advanced? > mapreduce is most of all, an API. Unix is contrary to idea of APIs > (everything is a stream of bytes). Damn good observation. Guess that's why Pike wrote the 'Sawzall' utility on top of it :) > > mapreduce isn't really rocket science by any means, see below. > >> From kernel to userland to network, I'm dying to find similar works, >> any help is much appreciated! > Similar things to look at: PVM and MPI - AWESOME, exactly what I was wanting to grok- Thanks Alex! -- Links for this thread, for the record: PVM (created 1989, currently actively maintained): http://www.csm.ornl.gov/pvm/ http://en.wikipedia.org/wiki/Parallel_Virtual_Machine MPI (created 1990s, man implementations in various contexts/languages): http://en.wikipedia.org/wiki/Message_Passing_Interface http://www.mpi-forum.org/ > these are APIs for non-shared > memory, message passing, distributed computation. They are an order of > magnitude more involved than mapreduce - they are much more generic. > mapreduce can be easily implemented using PVM but not vice versa. > > mapreduce is optimal for 'embarassingly parallel' jobs - ones that are > very easy to paralellize. There hasn't been much research into that > - its > been a solved problem 40 years ago. Not surprised. :) However powerful the simple idea of MapReduce is, there seems to be far too much hype over it all IMHO- and lots of confusion about applying it in discussions online, (when all you have is a hammer, everything is a nail...) Looking at it in historical context is very useful here. Rocket- and thanks Alex! .ike From ike at lesmuug.org Sat Aug 25 11:50:59 2007 From: ike at lesmuug.org (Isaac Levy) Date: Sat, 25 Aug 2007 11:50:59 -0400 Subject: [nycbug-talk] mapreduce, hadoop, and UNIX In-Reply-To: References: Message-ID: Awesome, On Aug 25, 2007, at 11:34 AM, Alex Pilosov wrote: > On Sat, 25 Aug 2007, Isaac Levy wrote: >> Afterthought addition, >> On Aug 25, 2007, at 10:48 AM, Isaac Levy wrote: >>> From kernel to userland to network, I'm dying to find similar >>> works, >>> any help is much appreciated! >> >> E.G.: >> >> Distributed computing implementations: >> - Plan 9? >> - DragonflyBSD Clustering? > We all are hoping today to have clusters similar to what VMS had 25 > years > ago - fully transparent non-shared memory clustering aka "single > system > image". You don't know, and you don't care which node on the > cluster the > job is running on, and jobs can be migrated to and from nodes > depending on > the load. > > For proper clustering, you need a distributed filesystem, > distributed lock > manager, and job distribution engine. > > On linux front, closest thing would be MOSIX, which is *almost* that. > Unfortunately, MOSIX is first and foremost a research project, with > restrictive licensing and fragmented community (see, openmosix). > Today, > the project to have properly working clusters is openssi.org - I > believe > it is based on openmosix and opengfs. /sigh > > Clustering is hard, comparing to writing an OS - even Linus can do > that > one. /heh > >> Data implementations: >> - Sun ZFS? >> - AFS and the like? >> - RH GFS and the like? > If you are talking about proper distributed filesystems, they are > few and > far between. > > gfs/opengfs > oracle ocfs > intermezzo/lustre > pvfs > veritas dfs > sgi cxfs (distributed xfs) AWESOME list, thanks again Alex. > > Distributed filesystems are hard, compared to writing an OS. Dude, if it wasn't a hard problem, we'd all have been using them for many years now :) Rocket- .ike From tekronis at gmail.com Sat Aug 25 11:56:36 2007 From: tekronis at gmail.com (H. G.) Date: Sat, 25 Aug 2007 11:56:36 -0400 Subject: [nycbug-talk] mapreduce, hadoop, and UNIX In-Reply-To: <60131f920708250855u54618b82g3017ef345219aa11@mail.gmail.com> References: <60131f920708250855u54618b82g3017ef345219aa11@mail.gmail.com> Message-ID: <60131f920708250856j7499cb65u631edb6d8eff8c9e@mail.gmail.com> On 8/25/07, Isaac Levy wrote: > > Hey All, > > This *may* be a thread which is aimed at the wrong list, but I > thought it's appropriate. Feel free to yell at me to take the thread > elsewhere. > > -- > Pretext: > I've been working on a personal project lately which has landed me > with some home-scale monster number-crunching tasks, as well as some > quickly scaling massive storage requirements. > (Fun fun fun, I'm trying to scan and OCR my personal book collection, > and I'm getting scared out of my mind now that I'm making some > headway :) > > Anyhow, I've been looking really closely at Google's MapReduce system/ > algorithm spec, which seems to be at the heart of how they make their > massive clusters work. This seems to be the current hot topic in > macho computing. > http://en.wikipedia.org/wiki/MapReduce > > Fun for UNIX folks, Rob Pike made an awk-like utility/language called > 'Sawzall' which uses Google's internal MapReduce API- I think it's > pretty interesting. > http://labs.google.com/papers/sawzall.html > > With that, I've also found that Yahoo is putting massive support into > an implementation of the MapReduce idea, Open Source as a part of the > Apache Project: > http://lucene.apache.org/hadoop/ > > There's other implementations cropping up all over, it seems. Like I > said, it's the current buzz... > > With all that, many of us have noticed that Google is good at scaling > patterns in computing- to bastardize their whole tech. operation, > it's their big trick. When I say scaling patterns, I mean: applying > classical computing paradigms and methodology at wild scales. (E.G. > with the Google Filesystem, it's simply a filesystem where the disk > blocks exist as network resources, etc... You see what I mean with > scale?) > > > -- > Question: > > Anyhow, I'm looking for more patterns in this MapReduce stuff, > because I'm simply not one to dive headfirst into 'buzz-tech'. With > that, aside from the map, and reduce, functions found in many > programming languages, > http://en.wikipedia.org/wiki/Map_%28higher-order_function%29 > http://en.wikipedia.org/wiki/Fold_%28higher-order_function%29 > > can anyone shed some light on similar prior works in distributed > computing and RPC systems which are 'old classics' in UNIX? These > distributed computing problems simply can't be new. > > To be really straight, what I'm getting at, is why is this more or > less useful than intelligently piping commands through ssh? What > about older UNIX rpc mechanisms? Aren't there patterns in even > kernel source code which match this work, or are even computationally > more sophisticated and advanced? > > From kernel to userland to network, I'm dying to find similar works, > any help is much appreciated! > > Rocket- > .ike > > > --- > p.s. > If anyone is interested in book-scanning stuff, Google happens to > currently host the Open Source 'tessarect' project, a very nice OCR > software- a very clean command-line application for OCR processing. > (I was actually inspired to start all this stuff based on how much > fun I had screwing around with tessarect). Apache licence, /me shrugs. > http://code.google.com/p/tesseract-ocr/ > > > _______________________________________________ > % NYC*BUG talk mailing list > http://lists.nycbug.org/mailman/listinfo/talk > %Be sure to check out our Jobs and NYCBUG-announce lists > %We meet the first Wednesday of the month > You might want to look into Starfish, which is a MapReduce implementation for Ruby. ( http://rufy.com/starfish/doc/ ) Should be greatly simpler than dealing with Hadoop (but thats just my personal opinion). -------------- next part -------------- An HTML attachment was scrubbed... URL: From carton at Ivy.NET Sat Aug 25 12:23:39 2007 From: carton at Ivy.NET (Miles Nordin) Date: Sat, 25 Aug 2007 12:23:39 -0400 Subject: [nycbug-talk] mapreduce, hadoop, and UNIX In-Reply-To: (Isaac Levy's message of "Sat, 25 Aug 2007 10:48:21 -0400") References: <0480763E-2ADB-417D-A632-BCA94310EA41@lesmuug.org> Message-ID: >>>>> "il" == Isaac Levy writes: il> I mean: applying classical computing paradigms and methodology il> at wild scales. (E.G. with the Google Filesystem, it's il> simply a filesystem where the disk blocks exist as network il> resources, etc... I think their ``big trick'' is working on jobs until they are finished and not teetering piles of dung like OpenMOSIX. The actual stuff they do sounds like it's not complicated or complete enough to become part of Unix---it's just some local webapp or cheesy daemon. I suspect half of their mapreduce tool is management infrastructure. Like, there is definitely a webapp the presenter showed us that draws a bar graph showing how far along each node has gotten in the current job. IIRC if a node lags for too long, its job (and bar) gets reassigned to another node. And he said often bars don't rise fast enough because, when a node's disk goes bad, first the node becomes slow for a day or two before the disk dies completely. And this management focus pushes its way down to the API---jobs _must_ be composed of restartable chunks with no side-effects that can be assigned to a node, then scrapped and reassigned. For example, a mapreduce job cannot be ``deliver an email to the 200,000 recipients on this mailing list,'' because if a node dies after sending 1000 messages successfully, mapreduce will restart that job and deliver duplicates of those 1000 messages. so it is rather an unimpressive tool. The virtue of it is how well they've hammered the quirks out of it, its ability to work on unreliable hardware (prerequisite for scaling to thousands of cheap nodes), the simplicity of the interface for new developers/sysadmins, and support for multiple languages (Python u.s.w.). il> - Sun ZFS? there's no distributed aspect to ZFS, only an analagous hype aspect. (honestly it's pretty good though. this tool I actually _am_ using for once, not just ranting about.) il> - AFS and the like? il> - RH GFS and the like? It's unfair to compare these to the google GFS because according to the presentation I saw, google's requires using very large blocks. It's a bit of a stretch to call Google's a ``file system''. It's more of a ``megabyte chunk fetcher/combiner/replicator.'' If you wanted to store the result of a crawl in one giant file, that's possible, but for most other tasks you will have to re-implement a filesystem inside your application to store the tiny files you need inside the one giant file you are allowed to performantly use. I think Sun's QFS may be GFS-ish, but with small blocks allowed, and on a SAN instead of on top of a bunch of cheesy daemons listening on sockets. There are variants of this idea from several of the old proprietary Unix vendors: one node holds the filesystem's metadata, and all the other nodes connect to it over Ethernet. but metadata only. Data, they hold on a Fibre-channel SAN, and all nodes connect to it over FC-SW. so, it is like NFS, but if you will use giant files like GFS does, and open them on only one node at a time, most of the traffic passes directly from the client to the disk, without passing through any file server's CPU. They had disgusting license terms where you pay per gigabyte and stuff. Another thing to point out about RedHat GFS, is that in a Mosix cluster a device special file points to a device _on a specific cluster node_. If I open /dev/hda on NFS, I just get whatever is /dev/hda on the NFS client. But on OpenMOSIX/GFS, the device methods invoked on the GFS client, the open/read/write/ioctl, get wrapped in TCP and sent to whichever node owns that device filename. That's needed for Mosix, but Google doesn't do anything nearly that ambitious. il> distributed computing and RPC systems i heard erlang is interesting, but haven't tried any of these---just a bibliography. so it might be silly, or broken. I think it's kind of two different tasks, if you want to write a distributed scientific-computing program from scratch? or you want to manage a scripty-type job built from rather large existing programs (which I suspect is what you want). One last thing. when my friend tried OpenMOSIX, he was really excited for about a month. Then he slowly realized that the overall system was completely unreliable---processes quietly, randomly dying, and other such stuff. That's the worst anti-recommendation I can think of. If he'd said ``I tried it---it didn't work,'' then I might try it again. but, ``I tried it. It wasted a month or two of my time before I found serious show-stopping problems about which the authors and evangelists were completely dishonest.'' so personally I want to see it in action publicly before I spend any time on it, and in action doing some job where you cannot afford to have processes randomly die. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: From ike at lesmuug.org Sat Aug 25 16:42:37 2007 From: ike at lesmuug.org (Isaac Levy) Date: Sat, 25 Aug 2007 16:42:37 -0400 Subject: [nycbug-talk] mapreduce, hadoop, and UNIX In-Reply-To: References: <0480763E-2ADB-417D-A632-BCA94310EA41@lesmuug.org> Message-ID: <7156C4C8-1769-4BF4-9224-44CFD565DF19@lesmuug.org> Wow, On Aug 25, 2007, at 12:23 PM, Miles Nordin wrote: >>>>>> "il" == Isaac Levy writes: > > il> I mean: applying classical computing paradigms and methodology > il> at wild scales. (E.G. with the Google Filesystem, it's > il> simply a filesystem where the disk blocks exist as network > il> resources, etc... > > I think their ``big trick'' is working on jobs until they are finished > and not teetering piles of dung like OpenMOSIX. The actual stuff they > do sounds like it's not complicated or complete enough to become part > of Unix---it's just some local webapp or cheesy daemon. Sure, all a small problem scaled up to massive size. Some of the results are no doubt impressive, but yes- nothing lasting enough to become a part of Unix. > > I suspect half of their mapreduce tool is management infrastructure. > Like, there is definitely a webapp the presenter showed us that draws > a bar graph showing how far along each node has gotten in the current > job. IIRC if a node lags for too long, its job (and bar) gets > reassigned to another node. And he said often bars don't rise fast > enough because, when a node's disk goes bad, first the node becomes > slow for a day or two before the disk dies completely. > > And this management focus pushes its way down to the API---jobs _must_ > be composed of restartable chunks with no side-effects that can be > assigned to a node, then scrapped and reassigned. For example, a > mapreduce job cannot be ``deliver an email to the 200,000 recipients > on this mailing list,'' because if a node dies after sending 1000 > messages successfully, mapreduce will restart that job and deliver > duplicates of those 1000 messages. > > so it is rather an unimpressive tool. The virtue of it is how well > they've hammered the quirks out of it, its ability to work on > unreliable hardware (prerequisite for scaling to thousands of cheap > nodes), the simplicity of the interface for new developers/sysadmins, > and support for multiple languages (Python u.s.w.). Interesting and spot-on take on things, you a lot in perspective here, in the context of the generated hype. > > il> - Sun ZFS? > > there's no distributed aspect to ZFS, only an analagous hype aspect. > (honestly it's pretty good though. this tool I actually _am_ using > for once, not just ranting about.) > > il> - AFS and the like? > il> - RH GFS and the like? > > It's unfair to compare these to the google GFS because according to > the presentation I saw, google's requires using very large blocks. > It's a bit of a stretch to call Google's a ``file system''. It's more > of a ``megabyte chunk fetcher/combiner/replicator.'' If you wanted to > store the result of a crawl in one giant file, that's possible, but > for most other tasks you will have to re-implement a filesystem inside > your application to store the tiny files you need inside the one giant > file you are allowed to performantly use. > > I think Sun's QFS may be GFS-ish, but with small blocks allowed, and > on a SAN instead of on top of a bunch of cheesy daemons listening on > sockets. There are variants of this idea from several of the old > proprietary Unix vendors: one node holds the filesystem's metadata, > and all the other nodes connect to it over Ethernet. but metadata > only. Data, they hold on a Fibre-channel SAN, and all nodes connect > to it over FC-SW. so, it is like NFS, but if you will use giant files > like GFS does, and open them on only one node at a time, most of the > traffic passes directly from the client to the disk, without passing > through any file server's CPU. They had disgusting license terms > where you pay per gigabyte and stuff. > > Another thing to point out about RedHat GFS, is that in a Mosix > cluster a device special file points to a device _on a specific > cluster node_. If I open /dev/hda on NFS, I just get whatever is > /dev/hda on the NFS client. But on OpenMOSIX/GFS, the device methods > invoked on the GFS client, the open/read/write/ioctl, get wrapped in > TCP and sent to whichever node owns that device filename. That's > needed for Mosix, but Google doesn't do anything nearly that > ambitious. On Aug 25, 2007, at 11:34 AM, Alex Pilosov wrote: >> Distributed filesystems are hard, compared to writing an OS. >> >> -alex :) > > > il> distributed computing and RPC systems > > i heard erlang is interesting, but haven't tried any of these---just a > bibliography. so it might be silly, or broken. > > I think it's kind of two different tasks, if you want to write a > distributed scientific-computing program from scratch? or you want to > manage a scripty-type job built from rather large existing programs > (which I suspect is what you want). That's exactly what I want- but right now, as I'm just barely getting going with this personal book-scanning project (which I suspect will take years to feel 'complete'), I'm willing to explore any path that others have had successes with. Right now, that looks like tying things together with existing programs- looks like I have plenty of choices, and room for growth with each! > > One last thing. when my friend tried OpenMOSIX, he was really excited > for about a month. Then he slowly realized that the overall system > was completely unreliable---processes quietly, randomly dying, and > other such stuff. That's the worst anti-recommendation I can think > of. If he'd said ``I tried it---it didn't work,'' then I might try it > again. but, ``I tried it. It wasted a month or two of my time before > I found serious show-stopping problems about which the authors and > evangelists were completely dishonest.'' so personally I want to see > it in action publicly before I spend any time on it, and in action > doing some job where you cannot afford to have processes randomly die. Miles, thanks for this lengthy reply, all these practical experiences are worth gold to me right now. Rocket- .ike From ike at lesmuug.org Sat Aug 25 16:47:13 2007 From: ike at lesmuug.org (Isaac Levy) Date: Sat, 25 Aug 2007 16:47:13 -0400 Subject: [nycbug-talk] mapreduce, hadoop, and UNIX In-Reply-To: <60131f920708251154u6e18e663id13806a8735c824@mail.gmail.com> References: <60131f920708250855u54618b82g3017ef345219aa11@mail.gmail.com> <60131f920708251154u6e18e663id13806a8735c824@mail.gmail.com> Message-ID: <4C4515D4-6F05-47B0-969E-397F30554857@lesmuug.org> On Aug 25, 2007, at 2:54 PM, H. G. wrote: >> On Aug 25, 2007, at 11:55 AM, H. G. wrote: >> > You might want to look into Starfish, which is a MapReduce >> > implementation for Ruby. >> > ( http://rufy.com/starfish/doc/ ) > >> Thanks! >> As a long-time happy python user, >> I'm constantly impressed by how fast the Ruby community tackles >> various projects. > >> MapReduce strategy, in the end, may be the >> simplest thing for me to maintain long-term. Not sure yet >> though... :) > In that case, you may want to look at PYRO. It doesn't do MapReduce, > but using it you can emulate that functionality in Python. > Basically an > option to be aware of if you're looking to do distributed computing > with > Python. ( http://pyro.sourceforge.net/ ) Wow! Ok, that's yet another tool which may be perfect for my task at hand. I seem to have it in my head that PYRO has been around a long time, totally worth a shot here... :) I'm exited, now I've got a ton of stuff to try! Rocket- .ike From ike at lesmuug.org Tue Aug 28 11:56:45 2007 From: ike at lesmuug.org (Isaac Levy) Date: Tue, 28 Aug 2007 11:56:45 -0400 Subject: [nycbug-talk] Tax for Hardware Donations? Message-ID: <8D8A6B5E-F4EC-4DB9-B372-34416D80B099@lesmuug.org> Hey All, I have a client with an old Apple G4 PPC Xserve, which they'd like to donate to some charity- but they want to do the tax-receipt voucher. Does the FreeBSD foundation, or any others, have a mechanism in place for this? Perhaps I should hit the PPC over there list to get someone exited about this? Thanks! Rocket- .ike From skreuzer at exit2shell.com Tue Aug 28 12:15:39 2007 From: skreuzer at exit2shell.com (Steven Kreuzer) Date: Tue, 28 Aug 2007 09:15:39 -0700 Subject: [nycbug-talk] Tax for Hardware Donations? In-Reply-To: <8D8A6B5E-F4EC-4DB9-B372-34416D80B099@lesmuug.org> References: <8D8A6B5E-F4EC-4DB9-B372-34416D80B099@lesmuug.org> Message-ID: <20070828161539.GA13907@clamps.exit2shell.com> On Tue, Aug 28, 2007 at 11:56:45AM -0400, Isaac Levy wrote: > Hey All, > > I have a client with an old Apple G4 PPC Xserve, which they'd like to > donate to some charity- but they want to do the tax-receipt voucher. > > Does the FreeBSD foundation, or any others, have a mechanism in place > for this? Perhaps I should hit the PPC over there list to get > someone exited about this? > > Thanks! > > Rocket- > .ike Donations to the FreeBSD Foundation are tax deductible. Check out: http://www.freebsdfoundation.org/donate/ for more informaiton. Cut and Paste of the lawyer sounding part from that page: Under US tax law, contributions to the Foundation are normally tax-deductible. Contributors are urged to seek professional tax advice to ensure that their particular contributions meet the requirements for deductibility. -- Steven Kreuzer http://www.exit2shell.com/~skreuzer From mikel.king at techally.com Tue Aug 28 13:25:38 2007 From: mikel.king at techally.com (Mikel King) Date: Tue, 28 Aug 2007 13:25:38 -0400 Subject: [nycbug-talk] Curious about everyones thoughts on NDRs.... Message-ID: <710B3B17-EB5C-484A-971B-1909776E780A@techally.com> Below you will find a URL to the article discussing the plight of the Non-Delivery Reports (NDR). Since I happen to know there are a number of other internet providers on this list I am curious if this will be or has been implemented. What sort of impact has happened if any as a result? The End of the NDR... http://www.eweek.com/article2/0,1895,2175935,00.asp Thanks in advance.... m! From carton at Ivy.NET Tue Aug 28 14:30:49 2007 From: carton at Ivy.NET (Miles Nordin) Date: Tue, 28 Aug 2007 14:30:49 -0400 Subject: [nycbug-talk] Curious about everyones thoughts on NDRs.... In-Reply-To: <710B3B17-EB5C-484A-971B-1909776E780A@techally.com> (Mikel King's message of "Tue, 28 Aug 2007 13:25:38 -0400") References: <710B3B17-EB5C-484A-971B-1909776E780A@techally.com> Message-ID: >>>>> "mk" == Mikel King writes: mk> http://www.eweek.com/article2/0,1895,2175935,00.asp -----8<----- It has become common for spammers to forge originating e-mail addresses and to then send large spam runs against different servers. When this happens, DynDNS sometimes receives these messages, which cannot be delivered, or worse, get bounced back to the original forged sender, who now gets the spam in his or her inbox (aka, spam blow back)... We simply feel that this is not the right thing to do. -----8<----- IMHO, they're completely right that what they're doing is wrong, and they need to stop doing it. Postfix calls it ``backscatter spam'' and includes some advice on how the victims of MTA's like DynDNS's former configuration can stop the spam _after_ it's been backscattered: http://cvsweb.netbsd.org/bsdweb.cgi/src/gnu/dist/postfix/README_FILES/BACKSCATTER_README?rev=1.1.1.4&content-type=text/x-cvsweb-markup The right way is to stop the backscattering. However there is absolutely no reason to stop generating bounces entirely! What they need to do is change their spam-checking so that it rejects spam instead of bouncing it: http://www.dontbouncespam.org/#BVR (apparently Qmail's bloody-minded absolutist disregard for the ``rough consensus and working code'' model is causing a sizeable chunk of the backscatter problem. It has to be patched to not backscatter. Can you even distribute pre-patched binaries with that man's weird licenses?) I guess what's going on in the article is, these mail ``forwarding'' services are doomed or at least less good than running your own MTA, because to stop backscatter, any MTA exposed to spammers needs to have a local list of all the valid users in the domains for which it is MX. In my opinion, you should do all your spam checks, both list-of-recipient checks and even lengthy checks like spamassassin, while the remote MTA is still connected, and send a 5xx error if you think the mail is spam. This stops backscatter, but preserves the old-Internet rule that mail should either be delivered or bounced. However, I admit that's not what my own site does right now. AFAIK with current implementations, bouncing with 5xx means you can't have ``spam folders'' because the spam isn't accepted. Someone could write one that bends the rules a little bit, and tells the sending MTA 5xx, but still secretly delivers a copy of the message to a local spam folder. Personally I don't like spam traps for things that ``might'' be spam. As a legitimate sender, I'd much rather have my mail bounced with 5xx than stuffed into a spam trap. The consensus on the Interweb seems to be, you should either reject or silently discard spam. meaning, it's okay to break the rule of the old Internet: now mail can be delivered, bounced, or silently discarded. I hate this, because it allows shifty people to say ``oh I didn't get your email. It must have gotten stuck in my spam filter.'' I'd hate to think the way I run my MTA is letting people give flakey-brained AOLexcuses like that. And it is really unnecessary. You can have the best of the new and the old world if you will run your spamassassin milter or amavis or whatever before your MTA sends its '250 queued as ...' message. But every web page I read says definitely do the silent-discarding instead of contributing to backscatter. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: From spork at bway.net Tue Aug 28 17:10:28 2007 From: spork at bway.net (Charles Sprickman) Date: Tue, 28 Aug 2007 17:10:28 -0400 (EDT) Subject: [nycbug-talk] terminals, telnet, blast from past Message-ID: All, I need a serious refresher on terminal types, general issues with telnet users (not ssh, telnet) and all other sorts of fancy shell account stuff that I haven't had to think about since 1998 or so. Things like dealing with people coming in with the windows telnet client and trying to run "pine", people with hacked up .profile or .cshrc files that complain that they get "^?^?^?" when they type a backspcae, etc. This stuff has all left my head. Google probably has something, but I'm not muttering the right words in its ear, er, search box. Somewhere the armadillo book is hiding in a closet here, I don't recall if that has a thorough discussion of how to break/unbreak things with rickety old telnet clients. Thanks, Charles From scottro at nyc.rr.com Tue Aug 28 17:38:25 2007 From: scottro at nyc.rr.com (Scott Robbins) Date: Tue, 28 Aug 2007 17:38:25 -0400 Subject: [nycbug-talk] terminals, telnet, blast from past In-Reply-To: References: Message-ID: <20070828213825.GB77864@mail.scottro.net> On Tue, Aug 28, 2007 at 05:10:28PM -0400, Charles Sprickman wrote: > All, > > I need a serious refresher on terminal types, general issues with telnet > users (not ssh, telnet) and all other sorts of fancy shell account stuff > that I haven't had to think about since 1998 or so. Things like dealing > with people coming in with the windows telnet client and trying to run > "pine", people with hacked up .profile or .cshrc files that complain that > they get "^?^?^?" when they type a backspcae, etc. > A couple of minor things that I ran into which might help. They might have to use the delete, rather than the backspace key. There are various set tty options--things like stty erase ^H or ^? in the .profile or .cshrc. Sometimes these work and sometimes they don't. Set their profile for TERM=vt100. These couple of things have helped me with a few cases of this, for example either the Windows or Mac terminal going to an AIX machine. man stty might have some ideas for you too. Isn't it aggravating to forget those things that you knew so well? -- Scott (of the senior moments) Robbins PGP keyID EB3467D6 ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 ) gpg --keyserver pgp.mit.edu --recv-keys EB3467D6 Anya: What a day. Gimme a beer. Bartender: (deadpan) ID. (Anya glares at him.) Bartender: (deadpan) ID. Anya: I'm eleven hundred and twenty years old! Just gimme a frickin' beer! Bartender: (deadpan) ID. Anya: (sigh) Gimme a Coke. From carton at Ivy.NET Tue Aug 28 18:46:37 2007 From: carton at Ivy.NET (Miles Nordin) Date: Tue, 28 Aug 2007 18:46:37 -0400 Subject: [nycbug-talk] terminals, telnet, blast from past In-Reply-To: (Charles Sprickman's message of "Tue, 28 Aug 2007 17:10:28 -0400 (EDT)") References: Message-ID: >>>>> "cs" == Charles Sprickman writes: cs> dealing with people coming in with the windows telnet client cs> and trying to run "pine", TERM=ansi is best for Windows telnet, but really it is pretty hopeless. I think it would be better to tell them to run a Java ssh client if they refuse to install anything. http://web.Ivy.NET/~carton/telnet/ If backspace doesn't work in cooked mode (try 'cat > /dev/null' and see if it works there. 'bash' will work with either backspace, even if you're misconfigured.), then you need to run 'stty erase ^?' or 'stty erase ^H'. You can type carat ? or carat space on the stty command line. In general fixing with stty isn't okay. You absolutely need to arrange for the backspace key on your terminal to send ^?, because ^H is already bound to the Help key in emacs, so it is not okay to reuse it as a backspace. Then, you won't be able to use help in emacs, unless you manually reconfigure emacs which will make it unlike emacs on other Unixes and seriously piss people off. ^? is right, and ^H is simply wrong. Even things like the FreeBSD console get this wrong. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: From nycbug at cyth.net Tue Aug 28 22:54:24 2007 From: nycbug at cyth.net (Ray Lai) Date: Tue, 28 Aug 2007 22:54:24 -0400 Subject: [nycbug-talk] Linux driver violates BSD license Message-ID: <20070829025447.GG30796@cybertron.cyth.net> I first read this on undeadly: http://undeadly.org/cgi?action=article&sid=20070829001634 Here is a snippet of the relicensing diff: diff --git a/drivers/net/wireless/ath5k.h b/drivers/net/wireless/ath5k.h index 0c6f3f5..c76b97b 100644 --- a/drivers/net/wireless/ath5k.h +++ b/drivers/net/wireless/ath5k.h @@ -2,17 +2,7 @@ * Copyright (c) 2004-2007 Reyk Floeter * Copyright (c) 2006-2007 Nick Kossifidis * - * Permission to use, copy, modify, and distribute this software for any - * purpose with or without fee is hereby granted, provided that the above - * copyright notice and this permission notice appear in all copies. - * - * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES - * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF - * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR - * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES - * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN - * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF - * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + * This file is released under GPLv2 */ From nycbug at cyth.net Tue Aug 28 22:58:59 2007 From: nycbug at cyth.net (Ray Lai) Date: Tue, 28 Aug 2007 22:57:59 -0401 Subject: [nycbug-talk] terminals, telnet, blast from past In-Reply-To: References: Message-ID: <20070829025822.GH30796@cybertron.cyth.net> On Tue, Aug 28, 2007 at 06:46:37PM -0400, Miles Nordin wrote: > In general fixing with stty isn't okay. You absolutely need to > arrange for the backspace key on your terminal to send ^?, because ^H > is already bound to the Help key in emacs, so it is not okay to reuse > it as a backspace. Then, you won't be able to use help in emacs, > unless you manually reconfigure emacs which will make it unlike emacs > on other Unixes and seriously piss people off. ^? is right, and ^H > is simply wrong. Even things like the FreeBSD console get this wrong. Just because ^H is bound to the Help key in emacs doesn't mean it is wrong. -Ray- From carton at Ivy.NET Wed Aug 29 00:22:49 2007 From: carton at Ivy.NET (Miles Nordin) Date: Wed, 29 Aug 2007 00:22:49 -0400 Subject: [nycbug-talk] Linux driver violates BSD license In-Reply-To: Ray Lai's message of "Tue, 28 Aug 2007 22:54:24 -0400" References: <20070829025447.GG30796@cybertron.cyth.net> <20070829025822.GH30796@cybertron.cyth.net> Message-ID: >>>>> "rl" == Ray Lai writes: rl> http://undeadly.org/cgi?action=article&sid=20070829001634 rl> Here is a snippet of the relicensing diff: yeah but just ten lines down on the same web page it says ``may also be distributed under the GPLv2.'' So, they are right. They _can_ just strip the BSD license right out of it. Good for Reyk in inviting them to. If they were obligated to keep both licenses forever, it wouldn't be ``dual'' licensed. It would be licensed under a third license that isn't internally consistent. Anyway, is that for real? madwifi will use Reyk's HAL? That's fantastic news. Too bad for Felix (the OpenWRT guy), though, and anyone else who signed Atheros's NDA, who will probably be forbidden from working on the new madwifi for a few years. (I'm just speculating again, though. sorry, I should stop.) rl> Just because ^H is bound to the Help key in emacs doesn't mean rl> it is wrong. This, I do not understand. That not everyone uses emacs, nor should be forced to use emacs, I do completely understand. That some competent people with opinions of merit hate emacs, I also understand. That it's ``not wrong'' for a Unix terminal to work improperly with emacs, I cannot accept. This is a very old Unix program that is absolutely expected to be installed and working on every decent Unix shell. Even a Unix sysadmin who hates emacs understands his absolute obligation to install it on a shell he offers to others, though he may leave it off a shell meant exclusively for his own use. If the delivered terminal doesn't work with emacs, then emacs users will have to fix it---hence, it's broken. It is normal for a decent Unix shell to be broken in a variety of ways that you have to fix yourself (or ask some other user how to fix). But to say it's not broken is completely ridiculous. It _is_ broken, and I _have_ fixed it myself, numerous times, because I use emacs, and it *does not work* until I *fix* the *broken* backspace key period (.) and anyway, if you want to be pedantic, I have two actual vt220's, and when you press that key they send ^?. You don't get to choose---they just send ^? no matter what. so don't come telling me your vt220 emulator that sends ^H isn't broken because the vt220 emulator in Procomm Plus for DOS sent ^H, too, and yours is like that one, and some devices designed to work with Procomm Plus for DOS expect ^H so it is just a matter of preference. I have a vt220. Mine isn't an emulator. It's a vt220. It's designed to work with Unix, and Unix is designed to work with it. It sends ^?. The emacs issue is overwhelmingly the most important one, but even if it weren't for that, anyone who sends or expects ^H---I don't care if they're Microsoft or SGI or FreeBSD or Procomm Plus---is _wrong_. They were wrong in 1983 when my vt220 was manufactured, wrong yesterday, and will be wrong tomorrow, until they fix it, or they ship it broken and I fix it for them. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: From mspitzer at gmail.com Tue Aug 28 23:30:42 2007 From: mspitzer at gmail.com (Marc Spitzer) Date: Tue, 28 Aug 2007 23:30:42 -0400 Subject: [nycbug-talk] terminals, telnet, blast from past In-Reply-To: <20070829025822.GH30796@cybertron.cyth.net> References: <20070829025822.GH30796@cybertron.cyth.net> Message-ID: <8c50a3c30708282030x8c3e5bakdb07f7d77b350d9a@mail.gmail.com> On 8/28/07, Ray Lai wrote: > On Tue, Aug 28, 2007 at 06:46:37PM -0400, Miles Nordin wrote: > > In general fixing with stty isn't okay. You absolutely need to > > arrange for the backspace key on your terminal to send ^?, because ^H > > is already bound to the Help key in emacs, so it is not okay to reuse > > it as a backspace. Then, you won't be able to use help in emacs, > > unless you manually reconfigure emacs which will make it unlike emacs > > on other Unixes and seriously piss people off. ^? is right, and ^H > > is simply wrong. Even things like the FreeBSD console get this wrong. > > Just because ^H is bound to the Help key in emacs doesn't mean it > is wrong. > > -Ray- Burn the heretic!!!!!! marc -- Freedom is nothing but a chance to be better. Albert Camus From nycbug at cyth.net Wed Aug 29 00:50:03 2007 From: nycbug at cyth.net (Ray Lai) Date: Wed, 29 Aug 2007 00:50:03 -0400 Subject: [nycbug-talk] Linux driver violates BSD license Message-ID: <20070829045026.GI30796@cybertron.cyth.net> Separating the two threads... On Wed, Aug 29, 2007 at 12:22:49AM -0400, Miles Nordin wrote: > >>>>> "rl" == Ray Lai writes: > > rl> http://undeadly.org/cgi?action=article&sid=20070829001634 > > rl> Here is a snippet of the relicensing diff: > > yeah but just ten lines down on the same web page it says ``may also > be distributed under the GPLv2.'' So, they are right. They _can_ > just strip the BSD license right out of it. Good for Reyk in inviting > them to. If they were obligated to keep both licenses forever, it > wouldn't be ``dual'' licensed. It would be licensed under a third > license that isn't internally consistent. This is explained by Theo in the comments: > > It was under a dual BSD/GPL license, so this is allowed, right? > > No. Some parts of the Atheros driver were authored by Sam Leffler, > and are actually free software. He placed those bits under a 4-term > BSD license, plus dual licensed it under the GPL. Still, that does > not give anyone except Sam Leffler the right to change that text, > on those files. > > The other files in the driver, written by Reyk, are the replacement > for the HAL. This basically is the hidden register access code which > Sam (basically employeed by Atheros) refused to release. This code > was placed by Reyk under an ISC license, something our project > prefers to use since it is so simple that even a grade 5 student > cannot misunderstand what it says. It translates to "You can do > anything, but not delete the text". > > Only Reyk could change that copyright notice, since he is the > author. From nycbug at cyth.net Wed Aug 29 00:56:46 2007 From: nycbug at cyth.net (Ray Lai) Date: Wed, 29 Aug 2007 00:55:46 -0401 Subject: [nycbug-talk] terminals, telnet, blast from past In-Reply-To: References: <20070829025447.GG30796@cybertron.cyth.net> <20070829025822.GH30796@cybertron.cyth.net> Message-ID: <20070829045609.GJ30796@cybertron.cyth.net> On Wed, Aug 29, 2007 at 12:22:49AM -0400, Miles Nordin wrote: > >>>>> "rl" == Ray Lai writes: > rl> Just because ^H is bound to the Help key in emacs doesn't mean > rl> it is wrong. > > This, I do not understand. > > That not everyone uses emacs, nor should be forced to use emacs, I do > completely understand. That some competent people with opinions of > merit hate emacs, I also understand. That it's ``not wrong'' for a > Unix terminal to work improperly with emacs, I cannot accept. This is > a very old Unix program that is absolutely expected to be installed > and working on every decent Unix shell. Even a Unix sysadmin who > hates emacs understands his absolute obligation to install it on a > shell he offers to others, though he may leave it off a shell meant > exclusively for his own use. > > If the delivered terminal doesn't work with emacs, then emacs users > will have to fix it---hence, it's broken. > > It is normal for a decent Unix shell to be broken in a variety of ways > that you have to fix yourself (or ask some other user how to fix). > But to say it's not broken is completely ridiculous. It _is_ broken, > and I _have_ fixed it myself, numerous times, because I use emacs, and > it *does not work* until I *fix* the *broken* backspace key period (.) > > and anyway, if you want to be pedantic, I have two actual vt220's, and > when you press that key they send ^?. You don't get to choose---they > just send ^? no matter what. so don't come telling me your vt220 > emulator that sends ^H isn't broken because the vt220 emulator in > Procomm Plus for DOS sent ^H, too, and yours is like that one, and > some devices designed to work with Procomm Plus for DOS expect ^H so > it is just a matter of preference. I have a vt220. Mine isn't an > emulator. It's a vt220. It's designed to work with Unix, and Unix is > designed to work with it. It sends ^?. The emacs issue is > overwhelmingly the most important one, but even if it weren't for > that, anyone who sends or expects ^H---I don't care if they're > Microsoft or SGI or FreeBSD or Procomm Plus---is _wrong_. They were > wrong in 1983 when my vt220 was manufactured, wrong yesterday, and > will be wrong tomorrow, until they fix it, or they ship it broken and > I fix it for them. I was saying that emacs is not the standard for terminal emulation. The reasons you stated above, however, are valid. -Ray- From spork at bway.net Wed Aug 29 04:05:38 2007 From: spork at bway.net (Charles Sprickman) Date: Wed, 29 Aug 2007 04:05:38 -0400 (EDT) Subject: [nycbug-talk] terminals, telnet, blast from past In-Reply-To: References: Message-ID: On Tue, 28 Aug 2007, Miles Nordin wrote: >>>>>> "cs" == Charles Sprickman writes: > > cs> dealing with people coming in with the windows telnet client > cs> and trying to run "pine", > > TERM=ansi is best for Windows telnet, but really it is pretty > hopeless. I think it would be better to tell them to run a Java ssh > client if they refuse to install anything. This is one of the areas where it gets weird. Using the same telnet client, you can telnet in to the old box (OpenBSD 2.mumble, really...) and echoing "$TERM" gives ansi. Same thing on the new box (FreeBSD 6.2). On the old box the user can type "pine" and it works. On the new box, pine complains that the terminal type "ansi" lacks required capabilities or some such and exits. There's no compile-time options in pine to alter terminal support. This is where I entered the rabbit hole... > http://web.Ivy.NET/~carton/telnet/ > > If backspace doesn't work in cooked mode (try 'cat > /dev/null' and > see if it works there. 'bash' will work with either backspace, even > if you're misconfigured.), then you need to run 'stty erase ^?' or > 'stty erase ^H'. You can type carat ? or carat space on the stty > command line. Well, I think I probably have a solution to the non-pine problems. I used to take loving care of a shell server at the last ISP I worked at. We had actual shell users that demanded all sorts of odd software and they actually used it. There were all sorts of login scripts for each shell that did a really good job of "fixing up" weird telnet/ssh clients that had problems in their default configs or were just broken. This stuff was originally started and maintained by some of the best unix admins I'd ever met. They were the type of guys that would not hesitate to "whip something up in C" to make their jobs easier - they were admins and they were also quite decent coders. So my plan is to snarf what I contributed to from that shell server, which is still running, and see if that will bring back some memories about this stuff I really wanted to forget. Of course being the paranoid that I am, I had a "*" in the password field of the master.passwd file so that I could only come in with an ssh key. At some point they seem to have turned off rsa/dsa key auth. So now I have to ask the new owners nicely if they can reset my password... :) Thanks, C > In general fixing with stty isn't okay. You absolutely need to > arrange for the backspace key on your terminal to send ^?, because ^H > is already bound to the Help key in emacs, so it is not okay to reuse > it as a backspace. Then, you won't be able to use help in emacs, > unless you manually reconfigure emacs which will make it unlike emacs > on other Unixes and seriously piss people off. ^? is right, and ^H > is simply wrong. Even things like the FreeBSD console get this wrong. > From dlavigne6 at sympatico.ca Wed Aug 29 08:26:04 2007 From: dlavigne6 at sympatico.ca (Dru) Date: Wed, 29 Aug 2007 08:26:04 -0400 (EDT) Subject: [nycbug-talk] Linux driver violates BSD license In-Reply-To: <20070829045026.GI30796@cybertron.cyth.net> References: <20070829045026.GI30796@cybertron.cyth.net> Message-ID: <20070829081859.L633@dru.domain.org> On Wed, 29 Aug 2007, Ray Lai wrote: >>> It was under a dual BSD/GPL license, so this is allowed, right? >> >> No. Some parts of the Atheros driver were authored by Sam Leffler, >> and are actually free software. He placed those bits under a 4-term >> BSD license, plus dual licensed it under the GPL. Still, that does >> not give anyone except Sam Leffler the right to change that text, >> on those files. >> >> The other files in the driver, written by Reyk, are the replacement >> for the HAL. This basically is the hidden register access code which >> Sam (basically employeed by Atheros) refused to release. This code >> was placed by Reyk under an ISC license, something our project >> prefers to use since it is so simple that even a grade 5 student >> cannot misunderstand what it says. It translates to "You can do >> anything, but not delete the text". >> >> Only Reyk could change that copyright notice, since he is the >> author. Looks like they're considering rewording, but still not showing the original copyright holder: http://lkml.org/lkml/2007/8/29/69 Dru From schmonz at schmonz.com Wed Aug 29 13:36:00 2007 From: schmonz at schmonz.com (Amitai Schlair) Date: Wed, 29 Aug 2007 13:36:00 -0400 Subject: [nycbug-talk] Curious about everyones thoughts on NDRs.... In-Reply-To: References: <710B3B17-EB5C-484A-971B-1909776E780A@techally.com> Message-ID: <46D5AE80.8080906@schmonz.com> Miles Nordin wrote: > (apparently Qmail's bloody-minded absolutist disregard for the ``rough > consensus and working code'' model is causing a sizeable chunk of the > backscatter problem. It has to be patched to not backscatter. Can > you even distribute pre-patched binaries with that man's weird > licenses?) This stuff is commonly misunderstood. qmail has no license. DJB's thoughts on licenses: There are restrictions on redistribution: In short, no, you can't distribute patched binaries. But even if you could, it'd be difficult to choose one SMTP-recipient-verification patch that'd work for everyone. Most of the options are described here: Sysadmins managing real-world mail installations are using packaging systems anyway (or really, really ought to be). For qmail, pkgsrc provides the badrcptto, qregex, and realrcptto patches as PKG_OPTIONS. In my scenario, realrcptto keeps a whole lot of crap out of my system, and a badrcptto clone blocks much of the rest. > In my opinion, you should do all your spam checks, both > list-of-recipient checks and even lengthy checks like spamassassin, > while the remote MTA is still connected, and send a 5xx error if you > think the mail is spam. I send 5xx if the mail is "really really spammy" (above a certain SpamAssassin score), otherwise it goes into the queue for local delivery. My users and I still get messages that score somewhere between "not spam" and "really really spammy", but not too many, and almost all of it goes into Spam folders which are small enough to easily eyeball and empty. Note that it's impossible to prevent all backscatter, because it's impossible to know for sure whether an arbitrary local delivery is supposed to succeed, because local delivery instructions are allowed to be complex and unverifiable. But it's very possible to get backscatter under control, and that's certainly a highly worthwhile goal for mail administrators to pursue. - Amitai From af.dingo at gmail.com Fri Aug 31 09:49:50 2007 From: af.dingo at gmail.com (Jeff Quast) Date: Fri, 31 Aug 2007 09:49:50 -0400 Subject: [nycbug-talk] terminals, telnet, blast from past In-Reply-To: References: Message-ID: Charles Sprickman, > Well, I think I probably have a solution to the non-pine problems. I used > to take loving care of a shell server at the last ISP I worked at. We had > actual shell users that demanded all sorts of odd software and they > actually used it. There were all sorts of login scripts for each shell > that did a really good job of "fixing up" weird telnet/ssh clients that > had problems in their default configs or were just broken. This stuff was > originally started and maintained by some of the best unix admins I'd ever > met. They were the type of guys that would not hesitate to "whip > something up in C" to make their jobs easier - they were admins and they > were also quite decent coders. Really miss those days. I wish everybody had an internet experience like that. I met a blind guy on a mail server through talk once, interrupted the sysadmin playing nethack, would finger @sysadmin's machines and talk them, play nethack in split-screen ytalk sessions, all kinds of neat interactive unixy stuff, trade files via /tmp, etc. You can't get an internet experience like that anymore. It was a nice transition coming from the BBS era, a bit more personal. Internet can be kinda bum lonely sometimes if you grew up dialing local BBS's. I've got this in a .kshrc alias vt220='export TERM=vt220; tset -I -Q' alias vt102='export TERM=vt102; tset -I -Q' alias wsvt25='export TERM=wsvt25; tset -I -Q' alias pcvt25='export TERM=pcvt25; tset -I -Q' alias xterm256='export TERM=xterm-256color; tset -I -Q' alias emacsmode='set +o vi; set -o emacs' alias vimode='set +o emacs; set -o vi' I have a bookshelf on termcap/termlib/curses programming. Terminals always interested me. I've been working on a modern BBS system in python, and I tried hard to support and understand the windows telnet client capabilities, and recently have given it up. It's a pathetic excuse of for a telnet client. I've accepted I'll just have to ask my userbase to download putty if they want reliable cursor control and color. Which isn't a lot to ask. You can create a session for your users configured the way it needs to be, and export a .reg key from regedit of that session. I've been making .reg files to help idiots setup tunnels, etc. You'd be amazed how many people are familiar with putty, I'm beginning to see it in the corporate world used by complete morons without much difficulty. From nycbug at maltin.org Fri Aug 31 12:31:24 2007 From: nycbug at maltin.org (Judd Maltin) Date: Fri, 31 Aug 2007 12:31:24 -0400 Subject: [nycbug-talk] drives swapped around - bad superblock Message-ID: <46D8425C.2070601@maltin.org> Hi Folks, I come, hat in hand, trying to fix a messed up FreeBSD system. I cut my Unix teeth on FreeBSD 2.2.5. I see a few things have changed since then. Now more /dev/wd0. :) I'm running: FreeBSD 6.1-RELEASE-p16 FreeBSD 6.1-RELEASE-p16 #0: Mon Apr 30 10:44:59 PDT 2007 My disks started to die, or get messed up or something, so my hosting facility put in a new drive, put their typical root partition image on it, and now I have this: # mount /dev/ad7s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/ad4s1a on /mnt/ad4s1a (ufs, local, soft-updates) /dev/ad6s1c on /mnt/ad6s1c (ufs, local, soft-updates) That's all fine and dandy, but dmesg shows me I have more goodies: ...snip.. ad4: 476940MB at ata2-master SATA150 ad5: 715404MB at ata2-slave SATA150 ad6: 715404MB at ata3-master SATA150 ad7: 476940MB at ata3-slave SATA150 And so I checked up on them with bsdlabel: # bsdlabel ad4 bsdlabel: /dev/ad4: no valid label found # bsdlabel ad4s1 # /dev/ad4s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 155877293 4194304 4.2BSD 0 0 0 b: 4194304 0 swap c: 160071597 0 unused 0 0 # "raw" part, don't edit # bsdlabel ad5s1 # /dev/ad5s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 1465144002 0 unused 2048 16384 # "raw" part, don't edit # bsdlabel ad6s1 # /dev/ad6s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 1465144002 0 unused 2048 16384 # "raw" part, don't edit # bsdlabel ad7s1 # /dev/ad7s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 972565697 0 4.2BSD 2048 16384 28512 b: 4194304 972565697 swap c: 976760001 0 unused 0 0 # "raw" part, don't edit I find it very interesting that I can mount /dev/ad6s1c just fine, that /dev/ad5s1c also mounts just fine.. but not when I reboot. I get bad superblocks. So, I've been a Linux guy for some time, so I try to fsck -t 4.2bsd /dev/ad5s1 (guessing at the type from the above): # fsck -t 4.2bsd /dev/ad5s1c ** /dev/ad5s1c BAD SUPER BLOCK: VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST ALTERNATE /dev/ad5s1c: NOT LABELED AS A BSD FILE SYSTEM (unused) That partition holds all of our backups for the past 6 months. I can't lose it. Oh, and it's in California. Uh-oh. This is where I stop. I need a FreeBSD superhero at this point. I need to perform a little surgery here on this disk, slice and partition, and I don't want to mess up. Help? From alex at pilosoft.com Fri Aug 31 12:46:49 2007 From: alex at pilosoft.com (Alex Pilosov) Date: Fri, 31 Aug 2007 12:46:49 -0400 (EDT) Subject: [nycbug-talk] drives swapped around - bad superblock In-Reply-To: <46D8425C.2070601@maltin.org> Message-ID: On Fri, 31 Aug 2007, Judd Maltin wrote: > I find it very interesting that I can mount /dev/ad6s1c just fine, that > /dev/ad5s1c also mounts just fine.. but not when I reboot. I get bad > superblocks. what's the exact error? > So, I've been a Linux guy for some time, so I try to fsck -t 4.2bsd > /dev/ad5s1 (guessing at the type from the above): > > # fsck -t 4.2bsd /dev/ad5s1c > ** /dev/ad5s1c > BAD SUPER BLOCK: VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST > ALTERNATE > /dev/ad5s1c: NOT LABELED AS A BSD FILE SYSTEM (unused) why are you specifying the -t option? what happens if you don't? > That partition holds all of our backups for the past 6 months. I can't > lose it. Oh, and it's in California. If you can mount, copy data away, trash it. Safer than messing around when you aren't sure what you are doing. > Uh-oh. This is where I stop. I need a FreeBSD superhero at this point. > I need to perform a little surgery here on this disk, slice and > partition, and I don't want to mess up.