[nycbug-talk] jails and shared filesystems

[Charles Sprickman]

On Thu, 16 Jun 2005, Isaac Levy wrote:

> Hi Charles, All,
>
> On Jun 15, 2005, at 2:43 AM, Charles Sprickman wrote:
>
>> Hey all,
>> 
>> I'm hoping some of our resident jail experts can help out with this 
>> issue...
>> 
>> I've been playing around with 5.4 and jails for some time now, and for the 
>> most part everything is really nice; the addition of jail configs to 
>> rc.conf is excellent.
>
> Dude- can you elaborate on this one, (sorry for the tangent here)?
>
> I've not had/made time to dive into the boot/rc configs for jailing, as I've 
> always avoided starting jails on boot like the plague based on problems 
> experienced long ago during the height of the 4.x era...
>
> - How do you like the rc.conf stuff for setting up jails?

Well, not having used 4.x jails, or any jails for that matter, I like it 
very much.  I posted the summary a few weeks(?) ago: 
http://lists.nycbug.org/pipermail/talk/2005-May/005569.html

>From what I've read about the 4.x stuff, this trumps it.

> - What facility, if any, is available to mitigate all the nasty problems 
> which can raise their head when a single jail barfs during system boot?

Again, not much experience there, but I have broken some test jails and 
running the /etc/rc.d/jail jailX stop|start did not hang, it just spit out 
an error and that was that.

> - Is there anything in the rc system that helps the system deal with the fact 
> it could be trying to simultaneously start up *many* different daemon apps, 
> (simultaneously!) while each jail starts?

Probably not, and I'm still not familiar enough with rcNG to really know 
where in the boot process the jail startup happens.

> - Any comments on all the buttons/knobs/bells/whistles which are noteworthy?

That link above pretty much covers all the cool stuff I found. :)

> Thx. for any comments here, (from anyone actually!), Charles, back to your 
> question,
>
>> 
>> Here's my problem/question:
>> 
>> I will only be running maybe 4 or 5 jails on this box.  Two or more of 
>> these jails may need to share some files.  To add to the confusion, these 
>> two or three jails are somewhat unpredictable in their growth.  I've got 
>> about 270GB to work with.  While testing things, I just took that whole 
>> partition and called it "/jails" and then made "/jails/jail1", 
>> "/jails/jail2", etc.  Technically, that works.
>
> Solid plan actually,

OK, I'm just trying to wrap my head around this, other than issues of 
disk-fillage, which I honestly do not think will be an issue in my 
application, are there any advantages in trying to partition things up?

>> But backing up and restoring one giant partition kind of makes me think 
>> twice about this.
>
> Yep.  What about breaking it up by userland disk contents, instead of the 
> whole disk being backed up?  I've actually come to love an aspect of 
> FreeBSD's relative userland consistency, i.e. one can back up /usr and /etc 
> from a jail be in really great shape in most restore/upgrade contexts...

Hmmm, good idea, why back up each jail beyond config and data?  Both 
Amanda and Flexbackup can deal with methods other than dump, so that kind 
of takes care of that issue.

>> What other options might I have?
>
> I have no word yet on nullfs being fixed (anyone? anyone?)- as of BSDCan it 
> was still widely known to be broken and in need of a maintainer.  I'm not 
> sure if this situation has changed.  DevFS, similar problems...

No idea, and it seems nullfs still has performance issues as well.

> --
> As another tangent (on my brain recently), you could locally mount remote raw 
> disk partitions using Geom Gate - (it's been committed to FreeBSD source 
> since 5.4, I believe, and I've been futzing with it in the BKNY lab...).
>
> 2003 initial Release:
> http://kerneltrap.org/node/1003
>
> 2004 Commited to FreeBSD:
> http://kerneltrap.org/news/freebsd?from=20
>
> ggated(8), ggatec(8), ggatel(8) are the utilities to use Geom Gate.
> http://www.freebsd.org/cgi/man.cgi? 
> query=ggated&sektion=8&apropos=0&manpath=FreeBSD+5.4-RELEASE

Hmmmm...

> Anyhow, just thought I'd throw this in, as you could mount remote disk 
> partitions with what seems to be something *way* faster and more bulletproof 
> than nfs- just by throwing some copper gig-e nics and crossover cables at the 
> problem...

I think I'll play around with ggate this week then.  It can't be worse 
than NFS.

>> I was thinking of grabbing 10-20GB and making that "/jails" and letting 
>> that be where my base system for each jail is installed.  This would also 
>> be "flat" and would hold any additional software.  I was then going to take 
>> the remaining space, and have each jail "mount" that, but apparently that's 
>> not really possible.
>
> Well, the jails themselves can't mount mount the volumes, but you could mount 
> them from the host server- just by making the mountpoint exist in the jail 
> directory trees.

Apparently, they can, I don't know if this is something new, but it's in 
the 5.x rc.d/jail script, and uses an /etc/fstab.jailname file to set up 
the mounts.  I assume this is really just an automated/jail-specific way 
of doing what you mention above.  But without resorting to 
nullfs/nfs/ggate trickery you can't have the mount appear in more than one 
jail.

> I mean it sounds like you've got a case for getting down with filesystem 
> quotas, imposed from the host system, on a big flat slice...  (Feels sanest 
> to me in this case).

Yeah, have to try that again.  Adding/deleting partitions in 5.x is pretty 
tough now.  It took me hours to find the geom debug flag to set to allow 
disklabel to work on a disk that has mounted partitions.  No manpage info, 
nothing, just have to Google until you find the thread where everyone 
yells at PHK about it.  I even managed to panic the machine in the 
process.

>> Any ideas?  I'm thinking just making one giant partition is what I'll end 
>> up doing, but I wanted to run this by folks first.
>
> From what I'm hearing in your post, I personally believe one giant partition, 
> and perhaps some disk quota use, is indeed sanest.  If your going for 
> disaster-recovery for the jails, the quickest/simplest strategy involves 
> making 4 tarballs:
>
> 1) a 'base' jail system tarball, the same thing for each jail, which can be 
> rolled-out very quickly to make a new jail (on different hardware box even if 
> necessary)
>
> 2) a 'base' config tarball, perhaps /etc/rc.conf and /etc.resolv.conf
>
> 3) /usr/*stuff* from each jailed system
>
> 4) start scripts and misc. for the jails
>
>> ps - next up, quotas!  that looks fun...
>
> Heh- here we go...  Anyone have any *serious* quota experience to share?  (my 
> experience has been that when employed, quotas simply work as the man page 
> says... transparently, nothin' to write home about...  Any of the 
> Good-bad-Ugly out there with regard to quotas?)

It's pretty messy, is what I'm seeing at first glance.  You have to have 
all your jailed users in the host's passwd db to make things work in a 
sane manner.  But I've just barely started reading about that.

Thanks,

Charles

> Rocket-
> .ike
>
>