[nycbug-talk] jails and shared filesystems

[Isaac Levy]

Hi Charles, All,

On Jun 15, 2005, at 2:43 AM, Charles Sprickman wrote:

> Hey all,
>
> I'm hoping some of our resident jail experts can help out with this  
> issue...
>
> I've been playing around with 5.4 and jails for some time now, and for  
> the most part everything is really nice; the addition of jail configs  
> to rc.conf is excellent.

Dude- can you elaborate on this one, (sorry for the tangent here)?

I've not had/made time to dive into the boot/rc configs for jailing, as  
I've always avoided starting jails on boot like the plague based on  
problems experienced long ago during the height of the 4.x era...

- How do you like the rc.conf stuff for setting up jails?

- What facility, if any, is available to mitigate all the nasty  
problems which can raise their head when a single jail barfs during  
system boot?

- Is there anything in the rc system that helps the system deal with  
the fact it could be trying to simultaneously start up *many* different  
daemon apps, (simultaneously!) while each jail starts?

- Any comments on all the buttons/knobs/bells/whistles which are  
noteworthy?

Thx. for any comments here, (from anyone actually!), Charles, back to  
your question,

>
> Here's my problem/question:
>
> I will only be running maybe 4 or 5 jails on this box.  Two or more of  
> these jails may need to share some files.  To add to the confusion,  
> these two or three jails are somewhat unpredictable in their growth.   
> I've got about 270GB to work with.  While testing things, I just took  
> that whole partition and called it "/jails" and then made  
> "/jails/jail1", "/jails/jail2", etc.  Technically, that works.

Solid plan actually,

> But backing up and restoring one giant partition kind of makes me  
> think twice about this.

Yep.  What about breaking it up by userland disk contents, instead of  
the whole disk being backed up?  I've actually come to love an aspect  
of FreeBSD's relative userland consistency, i.e. one can back up /usr  
and /etc from a jail be in really great shape in most restore/upgrade  
contexts...

Also, not sure if disk-resource based attacks are a threat here (many  
contexts it can create more problems to try to address this- right),  
but...

>
> What other options might I have?

I have no word yet on nullfs being fixed (anyone? anyone?)- as of  
BSDCan it was still widely known to be broken and in need of a  
maintainer.  I'm not sure if this situation has changed.  DevFS,  
similar problems...

--
As another tangent (on my brain recently), you could locally mount  
remote raw disk partitions using Geom Gate - (it's been committed to  
FreeBSD source since 5.4, I believe, and I've been futzing with it in  
the BKNY lab...).

2003 initial Release:
http://kerneltrap.org/node/1003

2004 Commited to FreeBSD:
http://kerneltrap.org/news/freebsd?from=20

ggated(8), ggatec(8), ggatel(8) are the utilities to use Geom Gate.
http://www.freebsd.org/cgi/man.cgi? 
query=ggated&sektion=8&apropos=0&manpath=FreeBSD+5.4-RELEASE

- NOTE ON PRODUCTION STABILITY -
Some folks I met in Canada are running Ggate stuff in production  
(conservatively), and report it as rock solid.  (I met the guy Pawel  
who wrote it, and he's razor sharp- and very actively continuing Geom  
work, so I'd personally lean towards trusting this stuff in production  
[but my conservative side wants to give it another year or two to  
mature].)  The Geom subsystem is *the* way to do disk stuff now, so  
weather or not it's as solid as ye' old 4.x disk drivers yet, it's the  
de-facto, and just going to get faster/better etc...

Anyhow, just thought I'd throw this in, as you could mount remote disk  
partitions with what seems to be something *way* faster and more  
bulletproof than nfs- just by throwing some copper gig-e nics and  
crossover cables at the problem...

(I was thinking about this for mounting /usr/ports as readonly across  
jailing boxen, and using localhost to repeatedly mount the disk for  
local jails (orchestrating all the mounting from the host systems), and  
have ports identical and up to date for all jailed servers across the  
lan...
--


> I was thinking of grabbing 10-20GB and making that "/jails" and  
> letting that be where my base system for each jail is installed.  This  
> would also be "flat" and would hold any additional software.  I was  
> then going to take the remaining space, and have each jail "mount"  
> that, but apparently that's not really possible.

Well, the jails themselves can't mount mount the volumes, but you could  
mount them from the host server- just by making the mountpoint exist in  
the jail directory trees.

I mean it sounds like you've got a case for getting down with  
filesystem quotas, imposed from the host system, on a big flat slice...  
  (Feels sanest to me in this case).


> At this point is where the jail/host thing gets confusing.  You can  
> mount one filesystem once using the "/etc/fstab.jailx" method, but  
> then you cannot share that space with other jails.

?  I'm confused, but I'm guessing this is about partitioning each  
jailed system (whomever setup the /etc/fstab.jailx thing beinb very  
conscious of mitigating resource-based problems, and being all *nix-fu  
with partitioning...)  Hrm.  I really need to make time to see what  
direction the rc jailing stuff is going...

>
> This brings me back to hacks like mount_null or NFS that I want to  
> avoid.

or ggated for that matter...  I guess it all adds complexity.

>
> Any ideas?  I'm thinking just making one giant partition is what I'll  
> end up doing, but I wanted to run this by folks first.

 From what I'm hearing in your post, I personally believe one giant  
partition, and perhaps some disk quota use, is indeed sanest.  If your  
going for disaster-recovery for the jails, the quickest/simplest  
strategy involves making 4 tarballs:

1) a 'base' jail system tarball, the same thing for each jail, which  
can be rolled-out very quickly to make a new jail (on different  
hardware box even if necessary)

2) a 'base' config tarball, perhaps /etc/rc.conf and /etc.resolv.conf

3) /usr/*stuff* from each jailed system

4) start scripts and misc. for the jails

--
If you keep consistent religious tar-balls in a manner relevant to your  
setup, you can recover from complete failure in as much time as it  
takes to unpack the tarballs... without having to deal with backing up  
the *entire* disk partiton.  But backup strategies are a bigger topic  
than my bastardized recommendation here, in many cases.

>
> Thanks,
>
> Charles
>
> ps - next up, quotas!  that looks fun...

Heh- here we go...  Anyone have any *serious* quota experience to  
share?  (my experience has been that when employed, quotas simply work  
as the man page says... transparently, nothin' to write home about...   
Any of the Good-bad-Ugly out there with regard to quotas?)

Rocket-
.ike