From matt at atopia.net Sun May 2 05:08:46 2010 From: matt at atopia.net (Matt Juszczak) Date: Sun, 2 May 2010 05:08:46 -0400 (EDT) Subject: [nycbug-talk] Rebuilding dependencies after manual port removal? Message-ID: Hi all, Tonight I had to manually remove php on one of my boxes, which another package I had installed was dependent on. I forcefully removed it, rebuilt it, and reinstalled. Now, all is working well, but dependencies are messed up (pkg_info shows the package isn't dependent on the other package even though it is). I know an easy solution to this would be `portupgrade -fa`, to forcefully rebuild and reinstall all ports, but that would take forever and seems unwarranted. Does anyone have any other ideas on how I can fix the dependency lists inside /var/db/pkg? Thanks, Matt From george at ceetonetechnology.com Sun May 2 09:37:54 2010 From: george at ceetonetechnology.com (George Rosamond) Date: Sun, 02 May 2010 09:37:54 -0400 Subject: [nycbug-talk] Rebuilding dependencies after manual port removal? In-Reply-To: References: Message-ID: <4BDD8032.5090808@ceetonetechnology.com> On 05/02/10 05:08, Matt Juszczak wrote: > Hi all, > > Tonight I had to manually remove php on one of my boxes, which another > package I had installed was dependent on. I forcefully removed it, > rebuilt it, and reinstalled. Now, all is working well, but dependencies > are messed up (pkg_info shows the package isn't dependent on the other > package even though it is). > > I know an easy solution to this would be `portupgrade -fa`, to > forcefully rebuild and reinstall all ports, but that would take forever > and seems unwarranted. Does anyone have any other ideas on how I can fix > the dependency lists inside /var/db/pkg? Which BSD are we talking about here? Just rebuild the relevant port and what depends on it then. portupgrade -fr Or just look at pkgdb (1). .. -L, -F. . . And look at /usr/ports/Tools/scripts if you're using FreeBSD. g From matt at atopia.net Sun May 2 12:47:55 2010 From: matt at atopia.net (Matt Juszczak) Date: Sun, 2 May 2010 12:47:55 -0400 (EDT) Subject: [nycbug-talk] Rebuilding dependencies after manual port removal? In-Reply-To: References: <4BDD8032.5090808@ceetonetechnology.com> Message-ID: >> Or just look at pkgdb (1). .. -L, -F. . . This didn't really do what I wanted it to do. To test, I did: oak# pkg_info -R php5-zip-5.3.2 Information for php5-zip-5.3.2: and got "phpmyadmin". I make deinstalled that package, reinstalled it, and then ran pkgdb -L... it cycled through depedencies marking all as "OK", but phpmyadmin still isn't listed as a package that depends on php5-zip (where it was before). Is this really a big deal? Some have mentioned to me that these dependencies will be fixed next time I upgrade a port anyway. But I like having the "packages dependent on this port" being correct as then it's way easier to figure out things you can uninstall. -Matt From matt at atopia.net Tue May 4 13:44:16 2010 From: matt at atopia.net (Matt Juszczak) Date: Tue, 4 May 2010 17:44:16 +0000 Subject: [nycbug-talk] OT: Caching Message-ID: <1497061023-1272995048-cardhu_decombobulator_blackberry.rim.net-234762937-@bda209.bisx.prod.on.blackberry> Hi all, I'm wondering how many of you have experience with caching on an http level? I would like to get more familiar with it. For high traffic sites, it seems a combination of ha proxy for load balancing, nginx for serving content, varnish for caching content, and esi somewhere in the midst of all that is a popular way to deploy. Other tend to replace nginx with apache. What setups have people on this list used, and what seems to be the most flexible? Matt From riegersteve at gmail.com Tue May 4 14:29:33 2010 From: riegersteve at gmail.com (riegersteve at gmail.com) Date: Tue, 4 May 2010 18:29:33 +0000 Subject: [nycbug-talk] OT: Caching Message-ID: <1186130676-1272997761-cardhu_decombobulator_blackberry.rim.net-1419880112-@bda130.bisx.prod.on.blackberry> I use tomcat for dynamic content and lighttp for static/cached ------Original Message------ From: Matt Juszczak Sender: talk-bounces at lists.nycbug.org To: talk at lists.nycbug.org ReplyTo: matt at atopia.net Subject: [nycbug-talk] OT: Caching Sent: May 4, 2010 10:44 Hi all, I'm wondering how many of you have experience with caching on an http level? I would like to get more familiar with it. For high traffic sites, it seems a combination of ha proxy for load balancing, nginx for serving content, varnish for caching content, and esi somewhere in the midst of all that is a popular way to deploy. Other tend to replace nginx with apache. What setups have people on this list used, and what seems to be the most flexible? Matt _______________________________________________ talk mailing list talk at lists.nycbug.org http://lists.nycbug.org/mailman/listinfo/talk Sent via BlackBerry from T-Mobile From max at neuropunks.org Tue May 4 19:45:20 2010 From: max at neuropunks.org (Max Gribov) Date: Tue, 04 May 2010 19:45:20 -0400 Subject: [nycbug-talk] summercon Message-ID: <4BE0B190.6020109@neuropunks.org> looks like one of the oldest security cons is coming to nyc.. summercon.org From bonsaime at gmail.com Tue May 4 23:54:47 2010 From: bonsaime at gmail.com (Jesse Callaway) Date: Tue, 4 May 2010 23:54:47 -0400 Subject: [nycbug-talk] summercon In-Reply-To: <4BE0B190.6020109@neuropunks.org> References: <4BE0B190.6020109@neuropunks.org> Message-ID: wow... the schedule looks... tight Didn't Nycbug have some mixer there at The Delancey once? I kinda remember a lot of us there on the deck. 4 years ago? -jesse On Tue, May 4, 2010 at 7:45 PM, Max Gribov wrote: > looks like one of the oldest security cons is coming to nyc.. > summercon.org > _______________________________________________ > talk mailing list > talk at lists.nycbug.org > http://lists.nycbug.org/mailman/listinfo/talk > -- -jesse From george at ceetonetechnology.com Wed May 5 00:02:14 2010 From: george at ceetonetechnology.com (George Rosamond) Date: Wed, 05 May 2010 00:02:14 -0400 Subject: [nycbug-talk] summercon In-Reply-To: References: <4BE0B190.6020109@neuropunks.org> Message-ID: <4BE0EDC6.5010302@ceetonetechnology.com> On 05/04/10 23:54, Jesse Callaway wrote: > wow... the schedule looks... tight > > Didn't Nycbug have some mixer there at The Delancey once? I kinda > remember a lot of us there on the deck. 4 years ago? Yup. . . a long while back. . . Nice rooftop place. g From pete at nomadlogic.org Wed May 5 12:44:04 2010 From: pete at nomadlogic.org (Pete Wright) Date: Wed, 5 May 2010 16:44:04 +0000 Subject: [nycbug-talk] OT: Caching In-Reply-To: <1497061023-1272995048-cardhu_decombobulator_blackberry.rim.net-234762937-@bda209.bisx.prod.on.blackberry> References: <1497061023-1272995048-cardhu_decombobulator_blackberry.rim.net-234762937-@bda209.bisx.prod.on.blackberry> Message-ID: <20100505164401.GA97424@pv.nomadlogic.org> On Tue, May 04, 2010 at 05:44:16PM +0000, Matt Juszczak wrote: > Hi all, > > I'm wondering how many of you have experience with caching on an http level? I would like to get more familiar with it. > > For high traffic sites, it seems a combination of ha proxy for load balancing, nginx for serving content, varnish for caching content, and esi somewhere in the midst of all that is a popular way to deploy. Other tend to replace nginx with apache. What setups have people on this list used, and what seems to be the most flexible? > Hi Matt, One architecture I've deployed is to use some sort of server load balancer (be it a dedicated appliance like an F5 BigIP, some linux boxen running ipvs or even DNS round robbin depending on the nature of your traffic). I've been working lately with varnish quite a bit and am quite impressed. it also supports basic load balancing capabilities as well as caching static content for GET requests. http://varnish-cache.org/ oh, and it was written by PHK :) aside from that - behind the SLB and caching layer its more of finding the server that can suit your needs right? i'm a fan of lighttpd for static content or simple cgi applications for example. HTH! -pete From siraaj at khandkar.net Thu May 6 00:06:41 2010 From: siraaj at khandkar.net (Siraaj Khandkar) Date: Thu, 6 May 2010 00:06:41 -0400 Subject: [nycbug-talk] Networking on Airbus Message-ID: <15361CB6-E160-4273-863D-19F64F0E26C2@khandkar.net> Here are some links about what I was talking about today: http://en.wikipedia.org/wiki/Avionics_Full-Duplex_Switched_Ethernet http://www.four-d.de/english/pdf/cids_e.pdf http://www.scribd.com/doc/29351778/Airbus-Embedded-Network http://www.icao.int/anb/panels/acp/WG/n/swgn1-2/sgn1%20wp209%205-19-04.doc So in theory it seems that avionics net should be physically separated from cab operations that are linked to passenger accessible net, but I guess the realities of the implementation still remain to be explored... Air traffic controlled (owned?) over TCP/IP? As Captain Jack Aubrey said: "What a fascinating modern age we live in!" -- Siraaj Khandkar From siraaj at khandkar.net Thu May 6 14:55:30 2010 From: siraaj at khandkar.net (Siraaj Khandkar) Date: Thu, 6 May 2010 14:55:30 -0400 Subject: [nycbug-talk] Scapy documentation Message-ID: <009B7AC2-EDF5-4C68-8251-427429CECB08@khandkar.net> So Kevin got pretty curious yesterday.... and it seems that Scapy's doc is actually pretty good (minus the hidden features Kevin mentioned): http://www.secdev.org/projects/scapy/doc/usage.html The graphs it makes are pretty damn cool too! It even does 3D! I'm definitely convinced... -- Siraaj Khandkar From george at ceetonetechnology.com Thu May 6 20:40:35 2010 From: george at ceetonetechnology.com (George Rosamond) Date: Thu, 06 May 2010 20:40:35 -0400 Subject: [nycbug-talk] Scapy documentation In-Reply-To: <009B7AC2-EDF5-4C68-8251-427429CECB08@khandkar.net> References: <009B7AC2-EDF5-4C68-8251-427429CECB08@khandkar.net> Message-ID: <4BE36183.3070204@ceetonetechnology.com> On 05/06/10 14:55, Siraaj Khandkar wrote: > So Kevin got pretty curious yesterday.... and it seems that Scapy's doc is actually pretty good (minus the hidden features Kevin mentioned): > > http://www.secdev.org/projects/scapy/doc/usage.html > > The graphs it makes are pretty damn cool too! It even does 3D! I'm definitely convinced... > > -- Siraaj Khandkar That's great Siraaj. One thing that Kevin and I discussed after the meeting was putting together some kind of informal workshop. Is that something others would be interested in? Show up with a laptop, and play away. Kevin is cc'd. .. but I'll get him on the list now. g From nikolai at fetissov.org Thu May 6 23:12:49 2010 From: nikolai at fetissov.org (Nikolai Fetissov) Date: Thu, 6 May 2010 23:12:49 -0400 Subject: [nycbug-talk] May 2010 meeting audio Message-ID: Folks, Audio of Kevin Figueroa presentation on Scapy is online at: http://www.fetissov.org/public/nycbug/nycbug-05-05-10.mp3 Cheers, -- Nikolai From siraaj at khandkar.net Fri May 7 16:26:01 2010 From: siraaj at khandkar.net (Siraaj Khandkar) Date: Fri, 7 May 2010 16:26:01 -0400 Subject: [nycbug-talk] Scapy documentation In-Reply-To: <4BE36183.3070204@ceetonetechnology.com> References: <009B7AC2-EDF5-4C68-8251-427429CECB08@khandkar.net> <4BE36183.3070204@ceetonetechnology.com> Message-ID: <07CAE089-BA7B-4B8D-915A-38F6964F5236@khandkar.net> On 6 May 2010, at 20:40, George Rosamond wrote: > On 05/06/10 14:55, Siraaj Khandkar wrote: >> So Kevin got pretty curious yesterday.... and it seems that Scapy's doc is actually pretty good (minus the hidden features Kevin mentioned): >> >> http://www.secdev.org/projects/scapy/doc/usage.html >> >> The graphs it makes are pretty damn cool too! It even does 3D! I'm definitely convinced... >> >> -- Siraaj Khandkar > > That's great Siraaj. > > One thing that Kevin and I discussed after the meeting was putting together some kind of informal workshop. > > Is that something others would be interested in? > > Show up with a laptop, and play away. > > Kevin is cc'd. .. but I'll get him on the list now. Sounds fun - count me in. May be we can do a little research on each others' mail/web servers? :-) -- Siraaj Khandkar From spork at bway.net Tue May 11 23:55:24 2010 From: spork at bway.net (Charles Sprickman) Date: Tue, 11 May 2010 23:55:24 -0400 (EDT) Subject: [nycbug-talk] hadoop - sharing a server? Message-ID: Hi all, I just recently went back and listened to the hadoop presentation from a few months ago. The timing was great, as I've been tasked with setting up a basic hadoop environment for pulling some stats out of a ton of mail logs. We'll likely be using HBase, but will be looking at Pig as well. I have a 3-node test setup running on FreeBSD 8.0 in VMWare. I was pleasantly surprised that Java was not a real pain to get going. In short, this all looks good, and it looks like it would be easy enough to copy one of these nodes to a jail, archive that jail, and then deploy a bunch of these things all over the place. So my question... What we're looking to do with Hadoop does not yet justify going out and buying a half dozen or so servers. I'd like to jail it on a bunch of our existing servers. The nature of the load on these things is that they have widely varying workloads with many lulls during the day. The nature of the jobs we want to run on the hadoop cluster is that basically we can wait as long as it takes for now. So is anyone running hadoop nodes on servers not dedicated to this task? Does it respond to being niced down? Are there some resource utiliztion knobs I've missed in all the quicky howto's I've read? Thanks, Charles From mcevoy.pat at gmail.com Wed May 12 01:18:03 2010 From: mcevoy.pat at gmail.com (Pat McEvoy) Date: Wed, 12 May 2010 01:18:03 -0400 Subject: [nycbug-talk] hadoop - sharing a server? In-Reply-To: References: Message-ID: On Tue, May 11, 2010 at 11:55 PM, Charles Sprickman wrote: > Hi all, > > I just recently went back and listened to the hadoop presentation from a few > months ago. ?The timing was great, as I've been tasked with setting up a > basic hadoop environment for pulling some stats out of a ton of mail logs. > ?We'll likely be using HBase, but will be looking at Pig as well. > > I have a 3-node test setup running on FreeBSD 8.0 in VMWare. ?I was > pleasantly surprised that Java was not a real pain to get going. ?In short, > this all looks good, and it looks like it would be easy enough to copy one > of these nodes to a jail, archive that jail, and then deploy a bunch of > these things all over the place. > > So my question... ?What we're looking to do with Hadoop does not yet justify > going out and buying a half dozen or so servers. ?I'd like to jail it on a > bunch of our existing servers. ?The nature of the load on these things is > that they have widely varying workloads with many lulls during the day. ?The > nature of the jobs we want to run on the hadoop cluster is that basically we > can wait as long as it takes for now. ?So is anyone running hadoop nodes on > servers not dedicated to this task? ?Does it respond to being niced down? > ?Are there some resource utiliztion knobs I've missed in all the quicky > howto's I've read? > > Thanks, > > Charles > > _______________________________________________ > talk mailing list > talk at lists.nycbug.org > http://lists.nycbug.org/mailman/listinfo/talk > You can also watch the Hadoop presentation here: http://blip.tv/file/3276543 P From brian.gupta at gmail.com Thu May 13 03:35:14 2010 From: brian.gupta at gmail.com (Brian Gupta) Date: Thu, 13 May 2010 03:35:14 -0400 Subject: [nycbug-talk] 5/19 @ 6:30PM Wietse Venema the author of Postfix on Postfix: past, present and future Message-ID: Figured this would be of general enough interest to sysadmins of any ilk.. - Brian Gupta New York City user groups calendar: http://nyc.brandorr.com/ ---------- Forwarded message ---------- From: NYLUG Announcements Date: Tue, May 4, 2010 at 9:30 AM Subject: [nylug-announce] NYLUG Presents: 5/19 @ 6:30PM Wietse Venema on Postfix: past, present and future To: NYLUG Announcements Wednesday, May 19, 2010 6:30 PM - 8:00 PM IBM 590 Madison Ave, 12th Floor corner of 57th Street *** RSVP Closes at 4:30 PM the day of the meeting (sharp!) *** Please RSVP for EVERY meeting at this time. Register at http://rsvp.nylug.org/ Check in with photo ID at the lobby for badge. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Wietse Venema ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? - on - ? ? ? ? ? ? ? ? ? ? ?Postfix: past, present and future In the 12 years since its initial release by IBM, the open source Postfix mail system has become part of the email infrastructure. ?The system has proven itself on personal systems and on ISP infrastructures with 10s of millions of mailboxes. ?After Postfix reached completion by 2006, the focus of development has moved from building new functionality towards making the system more extensible and more survivable in the face of changing threats and requirements. ?In this presentation Wietse will review lessons learned, current developments, and some speculation about the future. More Information: ?* Postfix Homepage ? ?http://www.postfix.org/ ?* Homepage of Wietse Venema ? ?http://www.porcupine.org/ About Wietse Venema: Wietse Venema is known for his software such as the TCP Wrapper and the POSTFIX mail system. ?He co-authored the SATAN network scanner and the Coroner's Toolkit (TCT) for forensic analysis, as well as a book on Forensic Discovery. ?Wietse received awards from the Free Software Foundation, the System Administrator's Guild (SAGE), the Netherlands UNIX User Group (NLUUG), as well as a Sendmail innovation award. ?He served a two-year term as chair of the international Forum of Incident Response and Security Teams (FIRST). ?Wietse currently is a research staff member at the IBM T. ?J. ?Watson research center. ?After completing his Ph.D. ?in physics he changed career to computer science and never looked back. ?In his spare time he enjoys hiking and cycling with his wife, Annita. Meeting Location: ?Please note that this meeting will be held at IBM, 590 Madison Ave, ?12th floor, corner of 57th Street, and not at Google. ?This is ?the building with the IBM logo on the front of the building. Map: ?http://nylug.org/mapofibm Swag (Give Away): ?During/after the meeting... unusually terrific swag may be given ?away. Stammtisch: ?After the meeting ... You may wish to join up with other NYLUGgers ?for drinks and pub food. ?This month we'll be over at TGI Friday's ?(677 Lexington and 56th Street, Second floor, Northeast corner), ?but we are also evaluating other options for the future and welcome ?your suggestions. ?http://nylug.org/tgifridays Time: 6:00pm Duration: 2 hours Location: NY Public Library Hudson Park Branch, 66 Leroy St., NY NY 10014 Topics: ?This week, we're going to continue looking at Smalltalk via Squeak. ?There may also be some discussion related to the C++ online workshop ?taking place on nylug-talk, and general discussion about Python. ?We'll be working through example code. ?Bring something to discuss! There's a blackboard, chalk, and Internet ?access. ?Notebook computers are helpful but not required. ?All levels of Python experience from totally new to experienced welcome! Resources: ?* ?Squeak Smalltalk environment ? ? http://www.squeak.org/ ?* ?NYLUG-Talk list ? ? http://nylug.org/listinfo/nylug-talk/ Map & Directions: ?http://nylug.org/hackcalendar ?We meet in the basement. Enter the library and head to the back. If the ?door is closed when you arrive you can ask the manager of the library for ?the keys to the room if you're comfortable opening up the basement, or ?you can wait for some of the others to arrive. Description: ?We will continue meeting on a bi-weekly basis at the Hudson Library at ?66 Leroy St New York, NY 10014. ?It is helpful, but not necessary to have a notebook computer. ?The WiFi at the library works now. Mailing List: ?We have a mailing list! ?Join it here: ?http://nylug.org/mailman/listinfo/hack ?or send mail to: hack-request at nylug.org ?with a Subject: subscribe ?There is also an RSS feed for the workshop mailing list at: ?http://nylug.org/mlist/hack.rss IRC Channel: ?On Freenode, in #nylug-python . ?Stop by #nylug also. Older pyworkshop: Time: 6:00pm Duration: 2 hours Location: NY Public Library Hudson Park Branch, 66 Leroy St., NY NY 10014 Topics: ?General discussion about Python, and working through example code. ?Bring something to discuss! There's a blackboard, chalk, and Internet ?access. ?Notebook computers are helpful but not required. ?All levels of Python experience from totally new to experienced welcome! Description: ?We will continue meeting on a bi-weekly basis at the Hudson Library at ?66 Leroy St New York, NY 10014. ?It is helpful, but not necessary to have a notebook computer. ?The WiFi at the library works now. Map & Directions: ?http://nylug.org/pythoncalendar ?We meet in the basement. Enter the library and head to the back. If the ?door is closed when you arrive you can ask the manager of the library for ?the keys to the room if you're comfortable opening up the basement, or ?you can wait for some of the others to arrive. Mailing List: ?We have a mailing list! ?Join it here: ?http://nylug.org/mailman/listinfo/nylug-workshop ?or send mail to: nylug-workshop-request at nylug.org ?with a Subject: subscribe ?There is also an RSS feed for the workshop mailing list at: ?http://nylug.org/mlist/nylug-python.rss IRC Channel: ?On Freenode, in #nylug-python . ?Stop by #nylug also. ?Next meetings are May 11, and May 25. ?See the calendar at: http://nylug.org/pythoncalendar Please see our home page at http://www.nylug.org for the HTMLized version of this announcement, our archives, and a lot of other good stuff. ______________________________________________________________________ Hire expert Linux talent by posting jobs here :: http://jobs.nylug.org nylug-announce mailing list nylug-announce at nylug.org http://nylug.org/mailman/listinfo/nylug-announce From mark.saad at ymail.com Thu May 13 08:07:09 2010 From: mark.saad at ymail.com (Mark Saad) Date: Thu, 13 May 2010 05:07:09 -0700 (PDT) Subject: [nycbug-talk] Adobe Openmarkets . Message-ID: <81908.16935.qm@web113520.mail.gq1.yahoo.com> Hello Talk Adobe released a nice open letter on how "open" they are. http://www.adobe.com/choice/openmarkets.html I say we strike while the irons hot and ask , where is the native support for BSD ? If they "believe open markets" lets see an open source version of the flash player / plugin that works with something other then linux windows and mac. -- Mark Saad mark.saad at ymail.com From george at ceetonetechnology.com Thu May 13 09:56:15 2010 From: george at ceetonetechnology.com (George Rosamond) Date: Thu, 13 May 2010 09:56:15 -0400 Subject: [nycbug-talk] Adobe Openmarkets . In-Reply-To: <81908.16935.qm@web113520.mail.gq1.yahoo.com> References: <81908.16935.qm@web113520.mail.gq1.yahoo.com> Message-ID: <4BEC04FF.2050904@ceetonetechnology.com> On 05/13/10 08:07, Mark Saad wrote: > Hello Talk > Adobe released a nice open letter on how "open" they are. > > http://www.adobe.com/choice/openmarkets.html > > > I say we strike while the irons hot and ask , where is the native support for BSD ? > If they "believe open markets" lets see an open source version of the flash player / plugin > that works with something other then linux windows and mac. Yes. . there's that petition that's been out for years about FreeBSD Flash support. http://www.petitiononline.com/flash4me/petition.html But the more general point is that software (and hardware) firms support standards, much less open source, in an odd way. Many do put some resources into open source development, and sometimes not even for their own self-interest. And it's often determined by their control (or lack of) in a particular market. Who wants the playing field open when they are dominating? Yet those firms also are frequent contributors to open source in one way or another for a variety of reasons. Sometimes just for public image purposes. Holding a minority market share in an arena usually propels firms to push for open standards and open source. Surprise, surprise. IMHO, that's the context of the Jobs' comments and this Adobe reply. AFAIK, Adobe is still a huge infrastructure user of BSDs in its internal operations. g From chsnyder at gmail.com Thu May 13 10:08:24 2010 From: chsnyder at gmail.com (Chris Snyder) Date: Thu, 13 May 2010 10:08:24 -0400 Subject: [nycbug-talk] Adobe Openmarkets . In-Reply-To: <81908.16935.qm@web113520.mail.gq1.yahoo.com> References: <81908.16935.qm@web113520.mail.gq1.yahoo.com> Message-ID: On Thu, May 13, 2010 at 8:07 AM, Mark Saad wrote: > I say we strike while the irons hot and ask , where is the native support for BSD ? > If they "believe open markets" lets see an open source version of the flash player / plugin > that works with something other then linux ?windows and mac. So BSD desktops can be open to all the nasty exploits in Flash, too? Maybe you'd like to install Acrobat Reader along with that to complete the nightmare? I think we should be grateful that Macromedia and Adobe developers have only concentrated on one platform at a time. Think of all the money you save by not buying Photoshop! From fire at firecrow.com Thu May 13 10:48:29 2010 From: fire at firecrow.com (fire crow) Date: Thu, 13 May 2010 10:48:29 -0400 Subject: [nycbug-talk] Adobe Openmarkets . In-Reply-To: References: <81908.16935.qm@web113520.mail.gq1.yahoo.com> Message-ID: Source is not the only kind of open, " We publish the specifications for Flash ? meaning anyone can make their own Flash player." - http://www.adobe.com/choice/openmarkets.html thats absolutely true, when JSConf took place in DC last month, I met Tobias Schnieder who has build a flash decompiler that runs in the browser in javascript and replaces the flash object with it's SVG equivilent http://github.com/tobeytailor/gordon he said the Flash spec was complete and very useful. "We believe that consumers should be able to freely access their favorite content and applications, regardless of what computer they have, what browser they like, or what device suits their needs." - http://www.adobe.com/choice/openmarkets.html that's absolutely true This post from adobe is admirable in comparison to jobs remarks... which were... entertaining http://www.apple.com/hotnews/thoughts-on-flash/ and the facts behind it here from os news http://www.osnews.com/story/23224/Jobs_on_Flash_Hypocrisy_So_Thick_You_Could_Cut_it_with_a_Knife I'm not a big fan of flash, but working with the Flex SDK in my command line linux environment was much easier than working with other major company tools, such as XCode, Eclipse, or Visual Studio. Also adobes adoption of the never released ECMA Script 4 for it's ActionScript 3, did alot to move forward the spec for ECMA Script 5, which could have significant security improvements for javasript mostly with the introduction of immutable data types. so while Adobe does not contribute to the open source community, they have effected it in some positive ways. But it is a good thing ECMA Script 4 was never an official spec :) ~fire http://firecrow.com : 917.306.9451 On Thu, May 13, 2010 at 10:08 AM, Chris Snyder wrote: > On Thu, May 13, 2010 at 8:07 AM, Mark Saad wrote: > >> I say we strike while the irons hot and ask , where is the native support for BSD ? >> If they "believe open markets" lets see an open source version of the flash player / plugin >> that works with something other then linux ?windows and mac. > > So BSD desktops can be open to all the nasty exploits in Flash, too? > Maybe you'd like to install Acrobat Reader along with that to complete > the nightmare? > > I think we should be grateful that Macromedia and Adobe developers > have only concentrated on one platform at a time. Think of all the > money you save by not buying Photoshop! > _______________________________________________ > talk mailing list > talk at lists.nycbug.org > http://lists.nycbug.org/mailman/listinfo/talk > -- ~fire http://firecrow.com : 917.306.9451 From mark.saad at ymail.com Thu May 13 10:23:43 2010 From: mark.saad at ymail.com (Mark Saad) Date: Thu, 13 May 2010 07:23:43 -0700 (PDT) Subject: [nycbug-talk] Adobe Openmarkets . In-Reply-To: References: <81908.16935.qm@web113520.mail.gq1.yahoo.com> Message-ID: <978088.68897.qm@web113510.mail.gq1.yahoo.com> > I say we strike while the irons hot and ask , where is the native support for > BSD ? > If they "believe open markets" lets see an open source version of > the flash player / plugin > that works with something other then linux > windows and mac. So BSD desktops can be open to all the nasty exploits > in Flash, too? Why use the internet if its no fun, exploits make thins interesting. I also enjoy using things like pandora.com which uses to work gnash . Maybe you'd like to install Acrobat Reader along with that to > complete the nightmare? I use xpdf , or foxit pdf , both are good example of a adobe format reader that is not made by adobe ; that work better then the original. I would love if adobe published more info about flash so we could have gnash or swfdeck up to speed with the mainline version from adobe. > I think we should be grateful that Macromedia > and Adobe developers have only concentrated on one platform at a time. Think > of all the money you save by not buying Photoshop! I will only use Photoshop when they make aalib output a standard format. -- Mark Saad mark.saad at ymail.com From mark.saad at ymail.com Thu May 13 10:27:27 2010 From: mark.saad at ymail.com (Mark Saad) Date: Thu, 13 May 2010 07:27:27 -0700 (PDT) Subject: [nycbug-talk] hadoop - sharing a server? In-Reply-To: References: Message-ID: <110947.21109.qm@web113504.mail.gq1.yahoo.com> Pat Ed the guy who gave the hadoop talk, uses Hadoop on VServer on Debian . -- Mark Saad mark.saad at ymail.com ----- Original Message ---- > From: Pat McEvoy > To: Charles Sprickman > Cc: talk at lists.nycbug.org > Sent: Wed, May 12, 2010 1:18:03 AM > Subject: Re: [nycbug-talk] hadoop - sharing a server? > > On Tue, May 11, 2010 at 11:55 PM, Charles Sprickman < > ymailto="mailto:spork at bway.net" > href="mailto:spork at bway.net">spork at bway.net> wrote: > Hi > all, > > I just recently went back and listened to the hadoop > presentation from a few > months ago. The timing was great, as I've been > tasked with setting up a > basic hadoop environment for pulling some stats > out of a ton of mail logs. > We'll likely be using HBase, but will be > looking at Pig as well. > > I have a 3-node test setup running on > FreeBSD 8.0 in VMWare. I was > pleasantly surprised that Java was not a > real pain to get going. In short, > this all looks good, and it looks > like it would be easy enough to copy one > of these nodes to a jail, > archive that jail, and then deploy a bunch of > these things all over the > place. > > So my question... What we're looking to do with Hadoop > does not yet justify > going out and buying a half dozen or so servers. > I'd like to jail it on a > bunch of our existing servers. The nature of > the load on these things is > that they have widely varying workloads with > many lulls during the day. The > nature of the jobs we want to run on the > hadoop cluster is that basically we > can wait as long as it takes for > now. So is anyone running hadoop nodes on > servers not dedicated to this > task? Does it respond to being niced down? > Are there some resource > utiliztion knobs I've missed in all the quicky > howto's I've > read? > > Thanks, > > Charles > > > _______________________________________________ > talk mailing > list > > href="mailto:talk at lists.nycbug.org">talk at lists.nycbug.org > > http://lists.nycbug.org/mailman/listinfo/talk > You can also watch > the Hadoop presentation > here: http://blip.tv/file/3276543 P _______________________________________________ talk > mailing list > href="mailto:talk at lists.nycbug.org">talk at lists.nycbug.org > href="http://lists.nycbug.org/mailman/listinfo/talk" target=_blank > >http://lists.nycbug.org/mailman/listinfo/talk From mcevoy.pat at gmail.com Thu May 13 11:21:05 2010 From: mcevoy.pat at gmail.com (Pat McEvoy) Date: Thu, 13 May 2010 11:21:05 -0400 Subject: [nycbug-talk] hadoop - sharing a server? In-Reply-To: <110947.21109.qm@web113504.mail.gq1.yahoo.com> References: <110947.21109.qm@web113504.mail.gq1.yahoo.com> Message-ID: On Thu, May 13, 2010 at 10:27 AM, Mark Saad wrote: > Pat > ?Ed the guy who gave the hadoop talk, uses Hadoop on VServer ?on ?Debian . > > ?-- > Mark Saad > mark.saad at ymail.com Hey Mark, Cheers for the info. I put together that video to practice knitting slides and audio together in order to make a video. It took ages but I am happy with the outcome. The video happened because the raw materials for it were easily at hand at the time. Future content will be much more BSD-centered, I promise. P > > > > ----- Original Message ---- >> From: Pat McEvoy >> To: Charles Sprickman >> Cc: talk at lists.nycbug.org >> Sent: Wed, May 12, 2010 1:18:03 AM >> Subject: Re: [nycbug-talk] hadoop - sharing a server? >> >> On Tue, May 11, 2010 at 11:55 PM, Charles Sprickman < >> ymailto="mailto:spork at bway.net" >> href="mailto:spork at bway.net">spork at bway.net> wrote: >> Hi >> all, >> >> I just recently went back and listened to the hadoop >> presentation from a few >> months ago. ?The timing was great, as I've been >> tasked with setting up a >> basic hadoop environment for pulling some stats >> out of a ton of mail logs. >> ?We'll likely be using HBase, but will be >> looking at Pig as well. >> >> I have a 3-node test setup running on >> FreeBSD 8.0 in VMWare. ?I was >> pleasantly surprised that Java was not a >> real pain to get going. ?In short, >> this all looks good, and it looks >> like it would be easy enough to copy one >> of these nodes to a jail, >> archive that jail, and then deploy a bunch of >> these things all over the >> place. >> >> So my question... ?What we're looking to do with Hadoop >> does not yet justify >> going out and buying a half dozen or so servers. >> ?I'd like to jail it on a >> bunch of our existing servers. ?The nature of >> the load on these things is >> that they have widely varying workloads with >> many lulls during the day. ?The >> nature of the jobs we want to run on the >> hadoop cluster is that basically we >> can wait as long as it takes for >> now. ?So is anyone running hadoop nodes on >> servers not dedicated to this >> task? ?Does it respond to being niced down? >> ?Are there some resource >> utiliztion knobs I've missed in all the quicky >> howto's I've >> read? >> >> Thanks, >> >> Charles >> >> >> _______________________________________________ >> talk mailing >> list >> >> href="mailto:talk at lists.nycbug.org">talk at lists.nycbug.org >> >> http://lists.nycbug.org/mailman/listinfo/talk >> > > You can also watch >> the Hadoop presentation >> here: > http://blip.tv/file/3276543 > P > _______________________________________________ > talk >> mailing list > >> href="mailto:talk at lists.nycbug.org">talk at lists.nycbug.org > >> href="http://lists.nycbug.org/mailman/listinfo/talk" target=_blank >> >http://lists.nycbug.org/mailman/listinfo/talk > _______________________________________________ > talk mailing list > talk at lists.nycbug.org > http://lists.nycbug.org/mailman/listinfo/talk > From matt at atopia.net Thu May 13 20:11:16 2010 From: matt at atopia.net (Matt Juszczak) Date: Thu, 13 May 2010 20:11:16 -0400 (EDT) Subject: [nycbug-talk] Cacti Alternatives? Message-ID: Hi all, I'd like to set something up like cacti, but I'd really like to automate the creation of the graphs and templates with either puppet (which talks to LDAP for nodes) or LDAP directly. Just like powerDNS is a great alternative for bind that allows an LDAP backend, I'm looking for an alternative to cacti (or a way to make cacti do what I want) that would allow me to use puppet, or LDAP directly, to add graphs to cacti and manage them. Does anyone have any ideas? What have others done? It seems like one could always write a process that would read data from LDAP and execute the command line scripts for cacti (or have puppet execute the command line scripts for cacti based on node data in LDAP). Thanks! -Matt From lance at demonware.net Thu May 13 20:25:16 2010 From: lance at demonware.net (Lance Laursen) Date: Thu, 13 May 2010 17:25:16 -0700 Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: References: Message-ID: On Thu, May 13, 2010 at 5:11 PM, Matt Juszczak wrote: > Hi all, > > I'd like to set something up like cacti, but I'd really like to automate > the creation of the graphs and templates with either puppet (which talks to > LDAP for nodes) or LDAP directly. Just like powerDNS is a great alternative > for bind that allows an LDAP backend, I'm looking for an alternative to > cacti (or a way to make cacti do what I want) that would allow me to use > puppet, or LDAP directly, to add graphs to cacti and manage them. > > Does anyone have any ideas? What have others done? It seems like one > could always write a process that would read data from LDAP and execute the > command line scripts for cacti (or have puppet execute the command line > scripts for cacti based on node data in LDAP). > > Thanks! > > -Matt > Since Cacti does not scale, and stores all of its template/device/whatever information in a database, I would take a serious look at Graphite ( http://graphite.wikidot.com/). "generating" new graphs is as simple as modifying a URL, and it scales to hundreds of servers per instance (thousands if your polling and storage is efficient, depending on how many people will be looking at graphs at once), something that is an awful thing to manage in Cacti. Unfortunately you do not get the polling engine of Cacti, but if you are generating graphing information by other means then that won't be a problem. If you do indeed need the polling engine, I'm afraid I don't know of any good alternatives. Unfortunately the "run-once" nature of creating graph URLs (or generating templates & adding to graph trees in cacti) falls outside of puppet's purpose, but i'm confident any work done against graphite will be cleaner than using cacti's cli tools. -- Lance Laursen Demonware Systems Engineer -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at atopia.net Thu May 13 20:28:21 2010 From: matt at atopia.net (Matt Juszczak) Date: Thu, 13 May 2010 20:28:21 -0400 (EDT) Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: References: Message-ID: > Unfortunately the "run-once" nature of creating graph URLs (or generating > templates & adding to graph trees in cacti) falls outside of puppet's > purpose, but i'm confident any work done against graphite will be cleaner > than using cacti's cli tools. I disagree that it falls outside puppet's purpose. I know many people that use puppet for first time configuration. That's why there's "Unless =>". Unfortunately, that means that everytime puppet runs on a host, it'll check to see if it's in cacti (once I get the scripts written properly). That could be good, or bad. -Matt From pete at nomadlogic.org Sat May 15 13:38:25 2010 From: pete at nomadlogic.org (Pete Wright) Date: Sat, 15 May 2010 17:38:25 +0000 Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: References: Message-ID: <20100515173821.GA36472@pv.nomadlogic.org> On Thu, May 13, 2010 at 08:11:16PM -0400, Matt Juszczak wrote: > Hi all, > > I'd like to set something up like cacti, but I'd really like to automate > the creation of the graphs and templates with either puppet (which talks > to LDAP for nodes) or LDAP directly. Just like powerDNS is a great > alternative for bind that allows an LDAP backend, I'm looking for an > alternative to cacti (or a way to make cacti do what I want) that would > allow me to use puppet, or LDAP directly, to add graphs to cacti and > manage them. > > Does anyone have any ideas? What have others done? It seems like one > could always write a process that would read data from LDAP and execute > the command line scripts for cacti (or have puppet execute the command > line scripts for cacti based on node data in LDAP). > hi matt - it sounds like having puppet execute the scripts (and having puppet pull the pertinent info from ldap as well) may get you where you want to go. my only concern would be that you would become beholden to puppet to execute your snmp and other queries. perhaps you could have puppet own your host templates for cacti, and setup some sort of code or trigger that would have puppet rebuild the host templates when a new host is added. this approach may make things easier to debug on the cacti side, and you would still be able to leverage the compiled cacti poller. HTH -pete -- Pete Wright pete at nomadlogic.org From pete at nomadlogic.org Sat May 15 14:02:05 2010 From: pete at nomadlogic.org (Pete Wright) Date: Sat, 15 May 2010 18:02:05 +0000 Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: <1630943673-1273946125-cardhu_decombobulator_blackberry.rim.net-1555467632-@bda244.bisx.prod.on.blackberry> References: <20100515173821.GA36472@pv.nomadlogic.org> <1630943673-1273946125-cardhu_decombobulator_blackberry.rim.net-1555467632-@bda244.bisx.prod.on.blackberry> Message-ID: <20100515180201.GB36472@pv.nomadlogic.org> On Sat, May 15, 2010 at 05:55:38PM +0000, Mark Saad wrote: > > Master ... Master ... Master of puppets pulling their striiiiings . Sorry metalica on the brain, there is a cacti alternative written using pythons django framework , it did not use rrd tool but svg based graphs . it looked nice but the name is lost on me now . > that may be graphite*. its a pretty nifty package, although the way that data gets ingested into it is kinda funky. it does not support snmp natively - instead you stream in formatted data into a listener daemon. but hey, who doesn't love running netcat on your production network** :) pretty fun for the DIY people - as long as you don't mind having to reinvent the wheel by writing scripts to collect data/parse data etc... -p *http://graphite.wikidot.com/ **http://graphite.wikidot.com/getting-your-data-into-graphite > Mark Saad | mark.saad at ymail.com > > -----Original Message----- > From: pete at nomadlogic.org (Pete Wright) > Date: Sat, 15 May 2010 17:38:25 > To: Matt Juszczak > Cc: > Subject: Re: [nycbug-talk] Cacti Alternatives? > > On Thu, May 13, 2010 at 08:11:16PM -0400, Matt Juszczak wrote: > > Hi all, > > > > I'd like to set something up like cacti, but I'd really like to automate > > the creation of the graphs and templates with either puppet (which talks > > to LDAP for nodes) or LDAP directly. Just like powerDNS is a great > > alternative for bind that allows an LDAP backend, I'm looking for an > > alternative to cacti (or a way to make cacti do what I want) that would > > allow me to use puppet, or LDAP directly, to add graphs to cacti and > > manage them. > > > > Does anyone have any ideas? What have others done? It seems like one > > could always write a process that would read data from LDAP and execute > > the command line scripts for cacti (or have puppet execute the command > > line scripts for cacti based on node data in LDAP). > > > > hi matt - it sounds like having puppet execute the scripts (and having > puppet pull the pertinent info from ldap as well) may get you where you > want to go. my only concern would be that you would become beholden to > puppet to execute your snmp and other queries. > > perhaps you could have puppet own your host templates for cacti, and > setup some sort of code or trigger that would have puppet rebuild the > host templates when a new host is added. this approach may make things > easier to debug on the cacti side, and you would still be able to > leverage the compiled cacti poller. > > HTH > -pete > > -- > Pete Wright > pete at nomadlogic.org > > _______________________________________________ > talk mailing list > talk at lists.nycbug.org > http://lists.nycbug.org/mailman/listinfo/talk -- -- Pete Wright pete at nomadlogic.org From mark.saad at ymail.com Sat May 15 13:55:38 2010 From: mark.saad at ymail.com (Mark Saad) Date: Sat, 15 May 2010 17:55:38 +0000 Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: <20100515173821.GA36472@pv.nomadlogic.org> References: <20100515173821.GA36472@pv.nomadlogic.org> Message-ID: <1630943673-1273946125-cardhu_decombobulator_blackberry.rim.net-1555467632-@bda244.bisx.prod.on.blackberry> Master ... Master ... Master of puppets pulling their striiiiings . Sorry metalica on the brain, there is a cacti alternative written using pythons django framework , it did not use rrd tool but svg based graphs . it looked nice but the name is lost on me now . Mark Saad | mark.saad at ymail.com -----Original Message----- From: pete at nomadlogic.org (Pete Wright) Date: Sat, 15 May 2010 17:38:25 To: Matt Juszczak Cc: Subject: Re: [nycbug-talk] Cacti Alternatives? On Thu, May 13, 2010 at 08:11:16PM -0400, Matt Juszczak wrote: > Hi all, > > I'd like to set something up like cacti, but I'd really like to automate > the creation of the graphs and templates with either puppet (which talks > to LDAP for nodes) or LDAP directly. Just like powerDNS is a great > alternative for bind that allows an LDAP backend, I'm looking for an > alternative to cacti (or a way to make cacti do what I want) that would > allow me to use puppet, or LDAP directly, to add graphs to cacti and > manage them. > > Does anyone have any ideas? What have others done? It seems like one > could always write a process that would read data from LDAP and execute > the command line scripts for cacti (or have puppet execute the command > line scripts for cacti based on node data in LDAP). > hi matt - it sounds like having puppet execute the scripts (and having puppet pull the pertinent info from ldap as well) may get you where you want to go. my only concern would be that you would become beholden to puppet to execute your snmp and other queries. perhaps you could have puppet own your host templates for cacti, and setup some sort of code or trigger that would have puppet rebuild the host templates when a new host is added. this approach may make things easier to debug on the cacti side, and you would still be able to leverage the compiled cacti poller. HTH -pete -- Pete Wright pete at nomadlogic.org _______________________________________________ talk mailing list talk at lists.nycbug.org http://lists.nycbug.org/mailman/listinfo/talk From scottro at nyc.rr.com Sat May 15 16:09:57 2010 From: scottro at nyc.rr.com (Scott Robbins) Date: Sat, 15 May 2010 16:09:57 -0400 Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: <1630943673-1273946125-cardhu_decombobulator_blackberry.rim.net-1555467632-@bda244.bisx.prod.on.blackberry> References: <20100515173821.GA36472@pv.nomadlogic.org> <1630943673-1273946125-cardhu_decombobulator_blackberry.rim.net-1555467632-@bda244.bisx.prod.on.blackberry> Message-ID: <20100515200957.GB98590@mail.scottro.net> On Sat, May 15, 2010 at 05:55:38PM +0000, Mark Saad wrote: > > Master ... Master ... Master of puppets pulling their striiiiings . Sorry metalica on the brain, there is a cacti alternative written using pythons django framework , it did not use rrd tool but svg based graphs . it looked nice but the name is lost on me now . > > Mark Saad | mark.saad at ymail.com Has munin been mentioned? (I haven't used it, but one of our developers just got into it, and has been showing me a bunch of graphs that it makes.) I have to add that I have NO knowledge of it, just that it seems, at superficial glance, to be somewhat similar, however, I have no idea if it really is or not. Just throwing that one out there--there may be great reasons to not use it, and I haven't followed the thread, so I may have missed it being discussed and discarded. -- Scott Robbins PGP keyID EB3467D6 ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 ) gpg --keyserver pgp.mit.edu --recv-keys EB3467D6 Xander: Isn't that what they called The Slayer? Willow: Buffy, ohh scary. Xander: Someone has to talk to her people. That name is striking fear in nobody's hearts. From max at neuropunks.org Sat May 15 17:17:53 2010 From: max at neuropunks.org (Max Gribov) Date: Sat, 15 May 2010 17:17:53 -0400 Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: <20100515200957.GB98590@mail.scottro.net> References: <20100515173821.GA36472@pv.nomadlogic.org> <1630943673-1273946125-cardhu_decombobulator_blackberry.rim.net-1555467632-@bda244.bisx.prod.on.blackberry> <20100515200957.GB98590@mail.scottro.net> Message-ID: <4BEF0F81.7010207@neuropunks.org> On 05/15/2010 04:09 PM, Scott Robbins wrote: > On Sat, May 15, 2010 at 05:55:38PM +0000, Mark Saad wrote: > >> Master ... Master ... Master of puppets pulling their striiiiings . Sorry metalica on the brain, there is a cacti alternative written using pythons django framework , it did not use rrd tool but svg based graphs . it looked nice but the name is lost on me now . >> >> Mark Saad | mark.saad at ymail.com >> > Has munin been mentioned? (I haven't used it, but one of our developers > just got into it, and has been showing me a bunch of graphs that it > makes.) > > munin is cool, it has collectors on the monitored hosts which send data to centralized logging and graphing server. very active community, many plugins, and new plugins are very easy to write. it monitors alot of stats out of the box. also i think ganglia is very good, although i have never tried customizing it. > I have to add that I have NO knowledge of it, just that it seems, at > superficial glance, to be somewhat similar, however, I have no idea if > it really is or not. > > Just throwing that one out there--there may be great reasons to not use > it, and I haven't followed the thread, so I may have missed it being > discussed and discarded. > > From bonsaime at gmail.com Mon May 17 15:09:57 2010 From: bonsaime at gmail.com (Jesse Callaway) Date: Mon, 17 May 2010 15:09:57 -0400 Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: References: Message-ID: On Thu, May 13, 2010 at 8:11 PM, Matt Juszczak wrote: > Hi all, > > I'd like to set something up like cacti, but I'd really like to automate the > creation of the graphs and templates with either puppet (which talks to LDAP > for nodes) or LDAP directly. ?Just like powerDNS is a great alternative for > bind that allows an LDAP backend, I'm looking for an alternative to cacti > (or a way to make cacti do what I want) that would allow me to use puppet, > or LDAP directly, to add graphs to cacti and manage them. > > Does anyone have any ideas? ?What have others done? ?It seems like one could > always write a process that would read data from LDAP and execute the > command line scripts for cacti (or have puppet execute the command line > scripts for cacti based on node data in LDAP). > > Thanks! > > -Matt > _______________________________________________ > talk mailing list > talk at lists.nycbug.org > http://lists.nycbug.org/mailman/listinfo/talk > Try giving a shout out to Ed Capriolo, who did a talk on Hadoop for nycbug a few months back. He has done a lot of scripting for nagios and cacti to get his automated hadoop deployments "phoning home" for monitoring. He has some scripts up on his website which you can hack on too... Just not sure where : ) -- -jesse From edlinuxguru at gmail.com Mon May 17 17:02:23 2010 From: edlinuxguru at gmail.com (Edward Capriolo) Date: Mon, 17 May 2010 17:02:23 -0400 Subject: [nycbug-talk] hadoop - sharing a server? Message-ID: You can jail anything of course. The issue with jailing hadoop is that it is very IO heavy because data is constantly being spilled to disk. Even if your jail can limit memory or processor ticks the real problem is jails do not protect your disk. Now if you system is only being used for background batch processing that is fine. However, if you are trying to run a "real time" ish mysql instance and hadoop on the same they may not play together well if they fight for the disk. Same is true with any jail/vm solution, but hadoop batching likes to saturate things with load. -------------- next part -------------- An HTML attachment was scrubbed... URL: From edlinuxguru at gmail.com Mon May 17 17:32:55 2010 From: edlinuxguru at gmail.com (Edward Capriolo) Date: Mon, 17 May 2010 17:32:55 -0400 Subject: [nycbug-talk] Cacti Alternatives? Message-ID: Cacti has a few automation features: http://docs.cacti.net/plugin:autom8 http://docs.cacti.net/plugin:discovery Also cacti has some php cli scripts that run from the command line, I believe those same scripts will run remotely as well. If you want a sample of some CLI scripting I can provide. Of course, neither autom8 nor discovery fit my needs perfectly. I had to integrate with internal things. In my case the cli opened all of the internals I needed like adding hosts, adding graphs, placing hosts on the tree, etc. Regards, Edward P.S. Jesse, thanks for the plug -------------- next part -------------- An HTML attachment was scrubbed... URL: From spork at bway.net Mon May 17 22:37:17 2010 From: spork at bway.net (Charles Sprickman) Date: Mon, 17 May 2010 22:37:17 -0400 (EDT) Subject: [nycbug-talk] hadoop - sharing a server? In-Reply-To: References: Message-ID: On Mon, 17 May 2010, Edward Capriolo wrote: > You can jail anything of course. The issue with jailing hadoop is that it is > very IO heavy because data is constantly being spilled to disk. Even if your > jail can limit memory or processor ticks the real problem is jails do not > protect your disk. Now if you system is only being used for background batch > processing that is fine. However, if you are trying to run a "real time" ish > mysql instance and hadoop on the same they may not play together well if > they fight for the disk. Same is true with any jail/vm solution, but hadoop > batching likes to saturate things with load. Thanks for the excellent feedback... Right now I just need to get something up for various reasons: -Evaluate Hadoop/HBase/Pig running on multiple hosts -Get myself up to speed on Hadoop and to some extent, Java from a sysadmin perspective -Get the folks that will be using this an environment to evaluate it and see if this is the proper set of tools to do the type of data analysis they want to do -Shake out any BSD-specific issues If this all goes well, we'd likely just bring up a few cheap servers as a standalone cluster. Until then, the idea of jailing it on servers that have very sporadic usage patterns and don't have to really do stuff in "real time" seems like it might be a good compromise. I'll be throwing this onto a few boxes in the next few days, so I'll report back with any interesting issues. I'm going to do two things to try and keep hadoop from being a total pig - it's jail will be on it's own zfs partition with a quota to prevent it from chewing up too much space, and when I put together an rc.d script for it, I'll nice down hadoop. For the future, there's some disk scheduling stuff coming into 8.1: http://wiki.freebsd.org/Releng/8.1TODO http://info.iet.unipi.it/~luigi/papers/20090508-geom_sched-slides.pdf Charles From edlinuxguru at gmail.com Tue May 18 00:27:05 2010 From: edlinuxguru at gmail.com (Edward Capriolo) Date: Tue, 18 May 2010 00:27:05 -0400 Subject: [nycbug-talk] hadoop - sharing a server? In-Reply-To: References: Message-ID: On Mon, May 17, 2010 at 10:37 PM, Charles Sprickman wrote: > On Mon, 17 May 2010, Edward Capriolo wrote: > > You can jail anything of course. The issue with jailing hadoop is that it >> is >> very IO heavy because data is constantly being spilled to disk. Even if >> your >> jail can limit memory or processor ticks the real problem is jails do not >> protect your disk. Now if you system is only being used for background >> batch >> processing that is fine. However, if you are trying to run a "real time" >> ish >> mysql instance and hadoop on the same they may not play together well if >> they fight for the disk. Same is true with any jail/vm solution, but >> hadoop >> batching likes to saturate things with load. >> > > Thanks for the excellent feedback... Right now I just need to get > something up for various reasons: > > -Evaluate Hadoop/HBase/Pig running on multiple hosts > -Get myself up to speed on Hadoop and to some extent, Java from a sysadmin > perspective > -Get the folks that will be using this an environment to evaluate it and > see if this is the proper set of tools to do the type of data analysis they > want to do > -Shake out any BSD-specific issues > > If this all goes well, we'd likely just bring up a few cheap servers as a > standalone cluster. > > Until then, the idea of jailing it on servers that have very sporadic usage > patterns and don't have to really do stuff in "real time" seems like it > might be a good compromise. I'll be throwing this onto a few boxes in the > next few days, so I'll report back with any interesting issues. > > I'm going to do two things to try and keep hadoop from being a total pig - > it's jail will be on it's own zfs partition with a quota to prevent it from > chewing up too much space, and when I put together an rc.d script for it, > I'll nice down hadoop. > > For the future, there's some disk scheduling stuff coming into 8.1: > > http://wiki.freebsd.org/Releng/8.1TODO > http://info.iet.unipi.it/~luigi/papers/20090508-geom_sched-slides.pdf > > Charles > Shameless plug here: I am going to do another hadoop talk. http://www.meetup.com/Hadoop-NYC/calendar/13512732/ It is going to be very low level (no powerpoint slides!). I hope some of you guys can make it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at atopia.net Tue May 18 12:18:44 2010 From: matt at atopia.net (Matt Juszczak) Date: Tue, 18 May 2010 12:18:44 -0400 (EDT) Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: References: Message-ID: > http://docs.cacti.net/plugin:autom8 Took a quick look at this. It looks great! Right now, I just have a script that runs each time a new host is added to LDAP. Can autom8 be configured to look at an LDAP directory and create graphs for hosts once added? Or is it better to just do that integration on my own? > Also cacti has some php cli scripts that run from the command line, I > believe those same scripts will run remotely as well.? If you want a > sample of some CLI scripting I can provide. This is what I'm using now. The script works fine, but I don't have it looking at LDAP yet. What did you end up doing for nagios? I have puppet configuring nagios right now, but that only seems to work for hosts configured by puppet (which isn't all of my hosts). -Matt From edlinuxguru at gmail.com Tue May 18 12:25:48 2010 From: edlinuxguru at gmail.com (Edward Capriolo) Date: Tue, 18 May 2010 12:25:48 -0400 Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: References: Message-ID: On Tue, May 18, 2010 at 12:18 PM, Matt Juszczak wrote: > http://docs.cacti.net/plugin:autom8 >> > > Took a quick look at this. It looks great! Right now, I just have a > script that runs each time a new host is added to LDAP. Can autom8 be > configured to look at an LDAP directory and create graphs for hosts once > added? Or is it better to just do that integration on my own? > > > Also cacti has some php cli scripts that run from the command line, I >> believe those same scripts will run remotely as well. If you want a sample >> of some CLI scripting I can provide. >> > > This is what I'm using now. The script works fine, but I don't have it > looking at LDAP yet. > > What did you end up doing for nagios? I have puppet configuring nagios > right now, but that only seems to work for hosts configured by puppet (which > isn't all of my hosts). > > -Matt Matt, Normally you would run autom8 and discovery in conjunction. discovery finds nodes and automate adds graphs to them. If you ldap code discovers graphs you do not need discovery and autom8 takes care of the rest. >> What did you end up doing for nagios? cp & sed :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at atopia.net Tue May 18 12:27:54 2010 From: matt at atopia.net (Matt Juszczak) Date: Tue, 18 May 2010 12:27:54 -0400 (EDT) Subject: [nycbug-talk] Cacti Alternatives? In-Reply-To: References: Message-ID: > Normally you would run autom8 and discovery in conjunction. discovery finds nodes and automate adds graphs to them. If > you ldap code discovers graphs you do not need discovery and autom8 takes care of the rest. Well, we can't "discover" nodes because we don't have a private LAN :) The LAN is shared with other servers. So we have to inventory servers and get creative. > >> What did you end up doing for nagios? > > cp & sed :) Ah okay :) Thanks! From matt at atopia.net Tue May 18 13:01:54 2010 From: matt at atopia.net (Matt Juszczak) Date: Tue, 18 May 2010 13:01:54 -0400 (EDT) Subject: [nycbug-talk] OT: Puppet Questions Message-ID: Hello fellow puppet savvy sys-admins, I have a few puppet questions, and I'm wondering if anyone knows the answers to any of them. 1) I'm a little confused on the fileserver. I have a file in /etc/puppet called "fileserver.conf". There are no lines in it. The documentation says that if no files are specified in that file, then the default is to "deny" all access to the file server. However, when I do things like: file { "name": source => "puppet:///module/filename" } it works fine. If the default policy of the fileserver is to deny access, why is that file able to be transferred? The main reason I'm concerned about this is because we're on a shared LAN (Cloud environment), and while I have puppet blocked off to the outside, on the LAN it's open to 10.0.0.0/8, which allows hosts that *aren't* ours to connect to it anyway. I manually approve any client SSL certificates with puppet-ca --sign, so that blocks hosts from using puppet when they aren't approved, but what's to stop those hosts from using the fileserver to grab files meant for our instances? Or am I confused, and is the fileserver.conf an added security measure ON TOP of the SSL certificate signing, but without the client certificate being signed, no access is allowed? 2) Nagios: I'm currently using Puppet to generate nagios configuration for our instances. It works well, but I need to also generate nagios host/service information for hosts which *aren't* managed by LDAP. Would there be an easy way to do this inside Nagios (for instance, manually configure those hosts inside the nagios module, so that they are mixed into the Nagios_host and Nagios_service files?) The other option I've been thinking of is putting those host configurations into LDAP (alongside other puppet nodes), and just manually generating nagios configs from LDAP with a cron. It seems like the stored configuration might hold good data too, but again, it would only hold data for instances managed by puppet. Any thoughts? 3) Simulating a private LAN with iptables: the LAN we have is shared (cloud environment). The negative of that is that any other host on the LAN can connect to puppet, internal DNS servers, LDAP, etc. I'm fairly confident in the security of those three tools, so I've considered leaving them fully open to the LAN and just making sure they are configured securely (IE: no anonymous bind for LDAP, etc.). Even if that is the case, I'd still like to lock down certain services on certain boxes to only allow connections from my other boxes (and not the entire LAN). What would the easiest way be to implement an access list that all hosts would maintain? It would be a set of X number of IPs (where X is unknown at this time) that would constantly be maintained automatically (so when a new server is launched, access is granted the next time each existing server checks into puppet). 4) Failover: What are people doing these days for puppet failover? My gut says to keep the configs in SVN, and always have another host on stand by. However, there's an issue with that: the puppet nodes wouldn't be able to just be re-pointed, because the client SSL certificates would be validated by the failover server (and therefore, there would be certificate validation errors). 5) Puppetizing your puppet servers: I've already made the decision NOT to LDAPify my LDAP servers (master and failover), as I wouldn't want LDAP to fail and cause issues getting into the LDAP box. My gut has also told me NOT to puppetize my puppet server (and just keep good backups); however, what have others done? I've seen conflicting documentation and blog entries on the subject. Thanks all! -Matt From matt at atopia.net Sat May 29 00:32:59 2010 From: matt at atopia.net (Matt Juszczak) Date: Sat, 29 May 2010 00:32:59 -0400 (EDT) Subject: [nycbug-talk] BSDJobs.net re-write Message-ID: Hi all, As many of you know, I started BSDJobs.net about 3 years ago. The site is still used today, but I haven't done anything new with it in the past few months, primarily because the code is procedural and I have always wanted it to be re-written from scratch first. Now is the time to do so. I've recently acquired some small (but reasonable) funding to pay someone to re-create the existing site from scratch. I can handle the database side of things, but that person would be responsible for the code + views/design. Once the site is re-written, I'd like to give SVN commit access to some of those who want it, so that more people can develop on the project and continue to help it grow. Before I reached out to some of my developer friends, I wanted to see if anyone on list was interesting in taking this on. Your name would forever be in the credits of the site, you would receive a "reasonable" sum for the work you do, and I would hope that you would still make contributions to the project (unpaid of course) after the initial re-write is complete. Thanks in advance for any interest. Matt