5.4 jails (was Re: [nycbug-talk] abusing a server)
Charles Sprickman
spork
Fri May 27 23:41:57 EDT 2005
I'm keeping Ike's excellent info below for context.
Anyhow after poking around in /etc/defaults/rc.conf, I thought I'd share
some stuff I found there. While the current jail(8) manpage is excellent,
it's still referencing 4.x, and none of the new stuff is included there.
So setting up jails looks to be much easier in 5.x. In the host's rc.conf
you can enable many of the common jail settings now:
# jail stuff - general
jail_enable="YES"
jail_list="jail1" (you can add more, seperate with spaces)
jail_set_hostname_allow="NO"
jail_socket_unixiproute_only="YES"
jail_sysvipc_allow="NO"
# note that last few set some general "security.jail.*" sysctls.
# use the names from jail_list above between "jail_ _optionname"
# jail stuff - per jail settings
jail_jail1_rootdir="/jails/jail1"
jail_jail1_hostname="blah.bway.net"
jail_jail1_ip="216.220.107.xx"
jail_jail1_exec_start="/bin/sh /etc/rc"
jail_jail1_exec_stop="/bin/sh /etc/rc.shutdown"
jail_jail1_devfs_enable="YES" # see below to set rules
jail_jail1_fdescfs_enable="NO"
jail_jail1_procfs_enable="NO"
jail_jail1_mount_enable="NO"
jail_jail1_devfs_ruleset="devfsrules_jail" # from default
/etc/defaults/devfs.rules
Then a simple config in jail1's rc.conf:
accounting_enable="YES"
rpcbind_enable="NO"
sshd_enable="YES"
network_interfaces=""
syslogd_enable="NO"
# ports
apache2_enable="YES"
I'm still digging a little bit, it seems the chflags stuff is not yet in
the rc scripts, I'm sure it will appear shortly.
This is very nice as you can start and stop all the jails very easily on a
running system:
root at newblah[/usr/ports/security/sudo]# sh /etc/rc.d/jail start
Configuring jails:.
Starting jails: blah.bway.net.
root at newida[/usr/ports/security/sudo]#
Now it's running:
root at newblah[/usr/ports/security/sudo]# jps -i 6
17152 17151 17148 17146 17145 17144 17143 17142 17129 17113 17101
And to stop it:
root at newblah[/usr/ports/security/sudo]# sh /etc/rc.d/jail stop
Stopping jails: blah.bway.net.
root at newblah[/usr/ports/security/sudo]# jps -i 6
root at newblah[/usr/ports/security/sudo]#
Coming together very nicely. Glad I waited for 5.4. :)
Charles
On Fri, 27 May 2005, Isaac Levy wrote:
> Hi Charles, All,
>
> For the record, I thought I'd toss in a quick checklist of issues to
> mitigate, off the top of my head:
> (note- ikestyle vocabulary follows, please excuse it for what it is)
>
> + Lost Jail Attack
>
> problem: a jailed user changes the jails hostname, making it terribly
> difficult to use tools on the host to locate and manage the jail
>
> solution: host server sysctl.conf variable:
> #security.jail.set_hostname_allowed=0
> ('man 8 jail' for more info)
>
> + Disk based resource attacks
>
> problem: jailed users fill disks
>
> solution: quotas or disk slices for jailed directory trees,
> managed from host server
>
> + Information Leak Attacks
>
> problem: jailed users get information from /dev which is useful
> in compromising the host system (or other jails), using tools like
> df and etc...
>
> solution: 2 parts, get your hands dirty with mount_devfs flags (on
> the host
> server to make certain a jail only gets to see devices it actually
> needs to have access to. mount_devfs can also use macro files in
> FreeBSD 5.4- so this can be simple to manage once you set it up.
>
> Additionally, check out this host server sysctl.conf variable:
> #security.jail.getfsstatroot_only=1
> ('man 8 jail' for more info)
>
> -
> problem: Jailed users *can* have access to more network interfaces
> than
> they actually need to, but this is not the default.
>
> solution: host server sysctl.conf variable:
> #security.jail.socket_unixiproute_only=1
> ('man 8 jail' for more info)
>
> -
> problem: Jailed user can have too much access to information-leaking
> syscalls,
> but this is not the default setup.
>
> solution: host server sysctl.conf variable:
> #security.jail.sysvipc_allowed=0
> ('man 8 jail' for more info)
>
>
> + Various Network Attacks
>
> problem: Users *can* get into trouble if they are allowed to create
> raw
> sockets, in some contexts. In other contexts, it may be important
> that
> users create raw sockets- (running custom network daemons, for
> example).
>
> solution: host server sysctl.conf variable:
> #security.jail.allow_raw_sockets=0
> If you really lock down what network daemons run, (just apache, for
> example),
> it may be wise to disallow raw sockets so someone can't for example
> run an
> irc daemon, or etc...
> ('man 8 jail' for more info)
>
>
> + Jailed Restraint Attack
>
> problem: Users may be able to bypass restrictions on process and
> memory limits,
> (put in place to mitigate fork-bombs and memory hogs), by changing
> login.conf et. al.
> (they are after all root, within their jail).
>
> solution: Restrict jailed users from ever editing files in jail, by
> restricting
> jailed root from calling chflags- (somewhat like running a higher
> secure level
> from within the jail itself)- this way only root on the host server
> can edit or
> modify these files, absolutely. Some files to lock down with
> immutable flags are
> a jail's sysctl.conf, login.conf, and perhaps some configs in /etc if
> relevant in
> your context- (i.e. resolv.conf, etc...)
> check out this host server sysctl.conf variable:
> #security.jail.chflags_allowed=0
> ('man 8 jail' for more info)
>
> -
> problem: Jailed users regularly gone amok, or need to be run in
> maximum security
> environment.
>
> solution: this is NOT fun, and in many contexts impractical, but
> individual jails
> can run at a higher BSD secure level than the host system.
> Configure a JAILED sysctl.conf variable:
> #kern.securelevel
> ('man 8 securelevel' for more info)
> Additionally, be sure to lock down sysctl.conf by setting the above
> mentioned
> #security.jail.chflags_allowed=0 variable in the HOST system.
>
>
>
>> I'm still not certain exactly what all the services offered will be, but at
>> a minimum I'm looking at apache+php, various bits of blog software and
>> shell users that tend to not do much beyond read mail in pine and edit
>> their ~user homepages.
>
> I'd think in that context, for your management sanity, you can really
> lock-down the jails if that's what the users are doing day to day. They
> don't need raw sockets, device access, special syscalls, or the ability for
> root user to use the chflags facility. You could even try running higher
> securelevels in the jails and see if it affects anyone's services.
>
>
>> Any ideas on how I can simulate around 50 logged-in users and a worst-case
>> flood of web traffic to a php-heavy blog? I really want to pummel this
>> thing. If it falls over, all the better as then I'll have something to
>> look at. What are the weak spots? The whole mess is jailed, and I'm
>> making heavy use of login.conf for the shell people so that they can't
>> fork-bomb or try to suck up more memory than they should. I think I'm most
>> interested in doing some very long-running tests...
>
> As good real-time memory/process control is a holy-grail type effort for MANY
> operating systems, and OS's which impliment it often pay severely for it in
> other ways. FreeBSD falls short with everyone else on this issue. Jailed
> users *may* be able to consume more memory or cpu than their fair share.
>
> A reliable way to mitigate this problem in reality, is to create some simple
> shell scripts to be run by cron, which do the following:
>
> - Check processes on the host system
> - nice/renice any processes hogging too much memory or CPU.
> - Check for any processes which have been nice'd/renice'd and eventually
> un-nice/renice them.
>
> In practice, this works terrificly, but the downside is that nobody has
> formalized any prefab tool to do this, to my knowledge.
>
>
>> So far I'm just fiddling with some shell scripts to simulate users futzing
>> around and hitting apache with http_load. So far I'm having more success
>> in breaking http_load than anything else.
>
> /me bows out and hopes someone else comes to the table with some good
> load-testing systems here.
>
> - Sidenote, insomuch as really good load testing comes when servers are being
> hit with packets from all across the internet, has anyone heard/thought of
> any groups of sysadmins who maintain 'test ddos' type networks? I.E. I'd
> gladly put out a jail to a trusted network of people, for use in load testing
> scenarios- if I could likewise have safe/sane access to a couple of hundred
> other boxen for the same purpose... (or like a bit-torrent type idea for load
> testing? :)
>
> Just thinking out loud on that one... I guess to really do this right
> NYC*BUG would have to take over some bot-nets from the spackers- (not a fun
> idea- that'd be all-out war :).
>
> Rocket-
> .ike
>
>
More information about the talk
mailing list