[nycbug-talk] 5.4 Jails, nullfs or unionfs?

[Isaac Levy]

Hi Charles,

Sorry to let this thread slide for you too- busy end to this week here.

On Apr 21, 2005, at 12:19 AM, Charles Sprickman wrote:

> On Tue, 19 Apr 2005, Isaac Levy wrote:
>
>> Turning on the -v flag here,
>
> and how!  Thanks for the excellent overview.  That should all be on a 
> webpage somewhere.

<cough>somethin's brewing!</cough>

I'm not good at the shameless self-promotion, but I'll be speaking on 
jails at BSDCan!  You going?!

http://www.bsdcan.org/2005/activity.php?id=58

> [big snip]
>
>> In FreeBSD 5.x devices are mounted using mount_devfs.  This is 
>> terrific for jailing, insomuch as the start scripts for a given jail 
>> can contain flags to mount_devfs to hide various devices, and the 
>> jail never gets them- it's that simple.
>
> Just to add to this, /etc/defaults/devfs.rules has a ruleset (#4, at 
> the bottom) that is given as an example for a jailed system.

NIIICE.  Man I'm exited to get on with 5.x in production eventually 
here!


>> Now, with regard to nullfs and unionfs, I have *no idea* what the 
>> state of these are for FreeBSD, or for jails- but I'm not personally 
>> aware of any manditory use cases for these in jails to begin with- 
>> though I can think of things which would become nicer to manage, (a 
>> single update to a ports tree for all systems perhaps, or a single 
>> user-land image template for massively parallel jailed clusters, 
>> etc...), but these cases, to me, seem to be better suited to chrooted 
>> enviornments, because of the implied homogeneity- so I'm stumped, 
>> (and looking for a reason to get exited about nullfs or unionfs!)...
>
> I currently don't do anything with jails, or 5.x for that matter.  But 
> in looking for new services to offer and new business models as far as 
> "dedicated/shared" web hosting and all the permutations of that that 
> stem from multiple "virtual hosts" on one big box, I'm starting to 
> wonder about what the easiest way to manage all this is.

--
Most scales of operation:

Easy: be extremely anal-rettentive, and make yourself packages using 
tarballs- and scripts to automate unpacking/managing the data in the 
jails.  (be sure to unmount devfs and procfs as stated in my last post 
to this thread)

Easiest: hire me as a consultant or system architect
(figured while I'm on the shameless self-promotion kick ;)


--
MEGA-HUGE scales of operation:

Easy: make FreeBSD Ports of your jail templates and components, suited 
to your environment or particular hosted product offerings. (I'm 
talking about making ports specific to your setup, as it would be 
insanely presumptious [borderline irresponsible?] to put these kinds of 
ports into the FreeBSD Ports collection at large)

Easiest: hire Yarema and I to do the above, Yarema on ports, me on 
jails... :)
(ok, ok- that's enough of the shameless self promotion- sorry)


> It seems like the dumb-simple way is to come up with a set of configs 
> for each jail and then just do an installworld into the jail from the 
> main host.

Yes, this would work, but, that's only half the battle still- in 
getting things installed- (configs? basic packages? etc...) and I've 
found that in practice, that strategy soaks more system resources than 
just untarring.


> What interested me about nullfs or mount_union is that you could 
> really save a ton of space and work if you had all of the stuff needed 
> by all jails shared read-only off of the "host" system.  From looking 
> at the archives, people do this with nullfs quite a bit.  The gotcha 
> here of course is that both nullfs and unionfs have rather stern 
> warnings in their manpages about actually using them for anything 
> important.

Did some research on nullfs and unionfs, and from what I gather,

--
While I understand the nullfs coolness, and space savings, I'll 
personally stick to my guns about 'harddrives being cheaper than one's 
time'.  Nullfs is a really cool, but I also see that it's fallen into 
some sort of disrepair for 5.x alltogether- (perhaps a side effect of 
harddrive cost?)- which makes me run from it like the plague.

--
With unionfs, I don't comprehend the practical savings- upgrading the 
unionfs mount seems to me to be a nightmare mess, no?
i.e. my understanding of unionfs is that local changes to a unionfs 
mount keep persistent state- i.e. they become 'data' that remains (as 
metadata, from the backend perspective).  So even so much as touching a 
file would change it, and cause it to persist.

So then, in the event of an upgrade, lets say an ssl vulnerability 
upgrade for this argument, if either the jailed admins have 
modified/touched the ssl components, or worst case- the host has 
already been compromised, and ssl has been so much as touched by the 
attackers, there's really little way to go in and do anything except 
nuke all the localized changes to the union mount and start from zero- 
which TOTALLY bites.

I've lived through this exact scenario several times at my old jail 
based hosting company (anyone remember the back-to-back vulnerabilities 
in 2003?)
In the ssl situation for instance, we successfully patched over 480 
jailed systems in just under 2 hours of work, most of that time being 
spent *slowly and maticulously* testing the upgrades in cloned jails.
The actual untarring and -HUP'ing of everything was the easy part- (and 
in that case, it was most effective to simply restart all the jails- 
and didn't make any clients mad, because we were finished before the 
rest of the world got around to start panicking).  Also, more than 
helpful is to be able to trust the tools being used at times like that- 
and tar is fairly trustworthy ;)


! ! ! ! ! ! !
Now, I could totally be missing something here because I chose not to 
work this way long ago, and my hands-on knowledge of nullfs and unionfs 
is nill, so Charles- if you or anyone wants to tell me different, I'm 
totally all ears- even to just discuss this particular point further... 
  But until then I'll steer away from these practices until I need them 
and they're stable.
! ! ! ! ! ! !


> It's odd that there's not more info out there on the subject, as I 
> have to imagine that many of these $30/month "dedicated server" folks 
> rely on FreeBSD and jails to make their business work.

Many of these companies use indeed use jailing- too bad that so many of 
them try to hide that they're just jailing and make something magical 
about it... (i.e. I'm always looking to find more jailing hosting 
providers!)


> I find it hard to believe they do 100 installworlds to update a box 
> full of jails. :)

Nope- tarballing packages (or using cvs from clean builds) is very 
common with oldschool jailing practitioners.


> Maybe we should lure "scrappy at hub.org" down for a talk at one of our 
> meetings.  He seems to be the main bugfinder in the area and it would 
> be interesting to hear how he goes about all this stuff.

Does scrappy work on nullfs stuff?
+1 on this idea!


>   It seems one of his long-running issues that occured as a result of 
> "stacking" filesystems was that his machines would run out of vnodes, 
> or worse, something was leaking vnodes...
>
> My first foray into 5.x and jails is going to be a new shell server 
> for a client.  If you believe that security comes from layering 
> protection, putting the shell users in a jail makes sense.  And then 
> when I think of what else I can do with this thing that will also be a 
> repository for lots of shareable content (personal pages, blogs, 
> possibly an AFP/SMB-accessible backup service, etc...) segregating 
> some of these services sounds like a good idea.

Heck yeah!  This is an excellent use.  Have fun!


> Anyhow, thanks for all the info, that message goes in the permanent 
> collection.  You should think about doing a DN on jail changes in 5.x. 
> 5.4 looks like it's getting to the point where more people may 
> entertain running it in a production environment.

Well, I'm glad the stack of esoteric knowledge lying around in my brain 
helped you out, and perhaps this is a good idea to post to DN- there's 
also some other fun coming after BSDCan, I'm sure...

<cough>somethin's brewing!</cough>

Rocket-
.ike