[nycbug-talk] shared hosting
G. Rosamond
george
Thu Jan 27 23:28:49 EST 2005
On Jan 27, 2005, at 5:29 PM, alex at pilosoft.com wrote:
> On Thu, 27 Jan 2005, Isaac Levy wrote:
>
<very big snip>
>
>> It seems to me, as Sunny and I had discussed long ago, that Linux VM
>> systems really need a Kernel available to the instances more than the
>> BSD's, because so much is done in Linux through the Kernel- whereas
>> the
>> converse is true across the BSD's, where much more is accomplished
>> through userland tools and subsystems (if not userland, at least
>> subsystems which aren't in the kernel).
> What in the world are you talking about now? Both Linux and bsd's have
> about the same separation of kernelspace and userspace. Name one
> example.
Woah. . . we can start with the fact that userland for the GNU/Linuxes
are providing some 200 or whatever interpretations of the kernelspace
of Linus. . . and the respective BSDs are dealing with the kernel and
user land in a much more unified manner. . .
>
> <snip>
>>> I beg to differ. Jail-related code is *all over* the OS. Every time a
>>> root privilege is checked, a code-writer needs to think whether one
>>> should also check for 'root but not superuse'.
>>
>> Are you absolutely certain that the code you are referring to is not
>> the
>> foundation for chroot? In BSD's other than FreeBSD, jail is often
>> vocabulary used to discribe chrooted processes- (esp. in the OpenBSD
>> scene).
>>
>> I'll be totally happy to stand down corrected, but at this point, I'm
>> going to need to see some examples in the kernel source- and have
>> started poking around...
>>
>> To my knowledge, after looking up/browsing the source for regular
>> jail,
>> it's even smaller than I'd thought,
> Jail is still chroot on steroids, no matter how much BSD people try to
> claim it isn't.
??? To paraphrase your earlier comments, Alex, it's all apples and
oranges.
chroot and jails may have started with the same vague goal, but that's
a bit of a stretch.
A provider might use chroot to provide services to a provider, but
providing jails is a whole other level of control that a chroot'd
program won't provide. . .
>
> <snip>
>> To my knowledge, that's all there is to jailing, so that's under 16 kb
>> in source code- (again, vs. the 9.2 mb of Xen- so we're actually at
>> the
>> magnitude of about 580 times the code, give or take, for the record.)
> Jail-related code is all over kernel. Everywhere you check for
> superuser,
> you need to check for jailed superuser.
yes, pretty built-in with implementation. . .
>
>> Now, diving deeper into the source, I can't find the jaill-related
>> code
>> which you state is *all-over*.
>>
>> I found about 550 references to jail, mostly contained in the files
>> mentioned above, and some strewn about in the chroot code, in the
>> source
>> for PS, expected places- but haven't taken the time to investigate
>> *too*
>> deeply (but did find a cute EPS diagram which grep ripped 'jail' out
>> of).
> There you go. It's all over. Everywhere kernel needs to check for
> superuser, it needs to check for a jailed superuser.
>
> <snip>
>
>> Well in PS, this makes a heck of a lot of sense, but not from the
>> perspective of a hole as you describe, but in the way which jails are
>> restricted from particular low-level calls, like the calls ps would
>> make
>> to page memory/cpu for processes.
> ps(1) does not page memory or CPU. Kernel does.
>
> Try to get your language correct, like 'system call'.
>
> Yes, in a syscall, kernel needs to verify whether root is a superuser.
> That means at every place where you check for rootness, you need to
> also
> check for jailed root-ness.
>
>> So this patch is to make ps function within the contextual confines
>> of a
>> jail, for practical purposes, and trying to modify this would just
>> break
>> ps for the jail (an inconvenient problem, for sure)..
> No, that patch was just cosmetic to show whether process is jailed or
> not.
>
> There's a bigger issue: ps used to work through direct access of kernel
> virtual memory. If you allow that access to a jailed user, you open
> yourself for information-leakage attacks. So you don't, and instead
> make a
> separate system call (or a /proc-like filesystem like linux) to export
> that information. That's code that might not have been needed if not
> for
> jail.
>
> Also, to me, the fact that jailed user can still see 'ps' for *every*
> process, not just owned by same user is a big information leak. And if
> you
> say that its possible to add code to prevent that leak by having
> /proc-like-filesystem present different views - well, that's even more
> jail-related-code that doesn't need to exist.
This might clear that question up. . . and note this is 4.10, not 5.x.
. .
from a jail. . .
[xxx at mail ~]$ sudo ps -ax
PID TT STAT TIME COMMAND
2171 ?? SsJ 0:01.04 /usr/sbin/syslogd -s
2178 ?? SsJ 0:00.54 /usr/sbin/inetd -wW
2180 ?? IsJ 0:00.66 /usr/sbin/cron
2182 ?? SsJ 0:00.49 /usr/sbin/sshd
2251 ?? SsJ 0:03.48 /usr/local/libexec/postfix/master
2253 ?? SJ 0:01.32 qmgr -l -t fifo -u
2254 ?? SsJ 0:06.94 /usr/local/sbin/httpd -DSSL
2270 ?? SsJ 0:05.40 /usr/bin/perl
/usr/local/lib/webmin/miniserv.pl /usr/lo
2273 ?? IJ 0:00.10 /usr/local/sbin/httpd -DSSL
2274 ?? IJ 0:00.11 /usr/local/sbin/httpd -DSSL
2275 ?? IJ 0:00.01 /usr/local/sbin/httpd -DSSL
2276 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
2277 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
8106 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
8112 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
56717 ?? SJ 0:00.02 pickup -l -t fifo -u
57311 ?? SJ 0:00.02 sshd: xxx [priv] (sshd)
57314 ?? SJ 0:00.01 sshd: xxx at ttyp0 (sshd)
57315 p0 SsJ 0:00.01 -bash (bash)
57316 p0 R+J 0:00.01 ps -ax
From the master box. . .
[xxx at xxx xxx]$ sudo ps -ax
PID TT STAT TIME COMMAND
0 ?? DLs 0:00.00 (swapper)
1 ?? ILs 0:00.05 /sbin/init --
2 ?? DL 0:00.00 (taskqueue)
3 ?? DL 0:00.05 (usb0)
4 ?? DL 0:00.00 (usbtask)
5 ?? DL 0:00.00 (aic_recovery0)
6 ?? DL 0:00.00 (aic_recovery0)
7 ?? DL 0:00.00 (aic_recovery1)
8 ?? DL 0:00.00 (aic_recovery1)
9 ?? DL 0:00.62 (pagedaemon)
10 ?? DL 0:00.00 (vmdaemon)
11 ?? DL 0:02.96 (bufdaemon)
12 ?? DL 0:08.07 (vnlru)
13 ?? DL 6:35.22 (syncer)
32 ?? Is 0:00.00 adjkerntz -i
91 ?? Ss 0:01.39 /usr/sbin/syslogd -s
98 ?? Ss 0:00.54 /usr/sbin/inetd -wW -a xxx.xxx.xxx.xxx
100 ?? Is 0:00.84 /usr/sbin/cron
102 ?? Ss 0:00.53 /usr/sbin/sshd -4
104 ?? Ss 0:01.05 /usr/sbin/usbd
107 ?? Ss 0:09.20 sendmail: accepting connections (sendmail)
110 ?? Is 0:00.16 sendmail: Queue runner at 00:30:00 for
/var/spool/clientmq
134 ?? Ss 0:00.63 moused -p /dev/psm0 -t auto
2171 ?? SsJ 0:01.04 /usr/sbin/syslogd -s
2178 ?? SsJ 0:00.54 /usr/sbin/inetd -wW
2180 ?? IsJ 0:00.66 /usr/sbin/cron
2182 ?? SsJ 0:00.49 /usr/sbin/sshd
2251 ?? SsJ 0:03.48 /usr/local/libexec/postfix/master
2253 ?? SJ 0:01.32 qmgr -l -t fifo -u
2254 ?? SsJ 0:06.95 /usr/local/sbin/httpd -DSSL
2270 ?? SsJ 0:05.40 /usr/bin/perl
/usr/local/lib/webmin/miniserv.pl /usr/lo
2273 ?? IJ 0:00.10 /usr/local/sbin/httpd -DSSL
2274 ?? IJ 0:00.11 /usr/local/sbin/httpd -DSSL
2275 ?? IJ 0:00.01 /usr/local/sbin/httpd -DSSL
2276 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
2277 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
2336 ?? SsJ 0:01.04 /usr/sbin/syslogd -s
2343 ?? SsJ 0:00.54 /usr/sbin/inetd -wW
2345 ?? IsJ 0:00.64 /usr/sbin/cron
2347 ?? SsJ 0:00.46 /usr/sbin/sshd
2416 ?? SsJ 0:03.42 /usr/local/libexec/postfix/master
2418 ?? SJ 0:01.36 qmgr -l -t fifo -u
2419 ?? SsJ 0:07.03 /usr/local/sbin/httpd -DSSL
2434 ?? SsJ 0:05.54 /usr/bin/perl
/usr/local/lib/webmin/miniserv.pl /usr/lo
2437 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2438 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2439 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2440 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2441 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2500 ?? SsJ 0:01.00 /usr/sbin/syslogd -s
2507 ?? SsJ 0:00.57 /usr/sbin/inetd -wW
2509 ?? IsJ 0:00.65 /usr/sbin/cron
2511 ?? SsJ 0:00.48 /usr/sbin/sshd
2580 ?? SsJ 0:03.40 /usr/local/libexec/postfix/master
2582 ?? SJ 0:01.34 qmgr -l -t fifo -u
2583 ?? SsJ 0:06.99 /usr/local/sbin/httpd -DSSL
2598 ?? SsJ 0:05.60 /usr/bin/perl
/usr/local/lib/webmin/miniserv.pl /usr/lo
2601 ?? IJ 0:00.10 /usr/local/sbin/httpd -DSSL
2602 ?? IJ 0:00.10 /usr/local/sbin/httpd -DSSL
2603 ?? IJ 0:00.10 /usr/local/sbin/httpd -DSSL
2604 ?? IJ 0:00.01 /usr/local/sbin/httpd -DSSL
2605 ?? IJ 0:00.11 /usr/local/sbin/httpd -DSSL
2664 ?? SsJ 0:02.12 /usr/sbin/syslogd -s
2671 ?? SsJ 0:00.54 /usr/sbin/inetd -wW
2673 ?? IsJ 0:00.63 /usr/sbin/cron
2675 ?? SsJ 0:00.48 /usr/sbin/sshd
2747 ?? SsJ 0:07.36 /usr/libexec/postfix/master
2749 ?? SJ 0:01.33 qmgr -l -t fifo -u
2750 ?? SsJ 0:06.83 /usr/local/sbin/httpd -DSSL
2766 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2767 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2768 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2769 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2770 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2844 ?? SsJ 0:01.05 /usr/sbin/syslogd -s
2851 ?? SsJ 0:00.52 /usr/sbin/inetd -wW
2853 ?? IsJ 0:00.64 /usr/sbin/cron
2855 ?? SsJ 0:00.48 /usr/sbin/sshd
2924 ?? SsJ 0:03.46 /usr/local/libexec/postfix/master
2926 ?? SJ 0:01.33 qmgr -l -t fifo -u
2927 ?? SsJ 0:07.28 /usr/local/sbin/httpd -DSSL
2942 ?? SsJ 0:05.41 /usr/bin/perl
/usr/local/lib/webmin/miniserv.pl /usr/lo
2945 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
2946 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2947 ?? IJ 0:00.01 /usr/local/sbin/httpd -DSSL
2948 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
2949 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
3008 ?? SsJ 0:01.02 /usr/sbin/syslogd -s
3015 ?? SsJ 0:00.52 /usr/sbin/inetd -wW
3017 ?? IsJ 0:00.64 /usr/sbin/cron
3019 ?? SsJ 0:00.48 /usr/sbin/sshd
3088 ?? SsJ 0:03.40 /usr/local/libexec/postfix/master
3090 ?? SJ 0:01.32 qmgr -l -t fifo -u
3091 ?? SsJ 0:06.99 /usr/local/sbin/httpd -DSSL
3106 ?? SsJ 0:05.46 /usr/bin/perl
/usr/local/lib/webmin/miniserv.pl /usr/lo
3109 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
3110 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
3111 ?? IJ 0:00.01 /usr/local/sbin/httpd -DSSL
3112 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
3113 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
3172 ?? SsJ 0:01.00 /usr/sbin/syslogd -s
3179 ?? SsJ 0:00.55 /usr/sbin/inetd -wW
3181 ?? IsJ 0:00.64 /usr/sbin/cron
3183 ?? SsJ 0:00.45 /usr/sbin/sshd
3252 ?? SsJ 0:03.41 /usr/local/libexec/postfix/master
3254 ?? SJ 0:01.30 qmgr -l -t fifo -u
3255 ?? SsJ 0:06.74 /usr/local/sbin/httpd -DSSL
3270 ?? SsJ 0:05.59 /usr/bin/perl
/usr/local/lib/webmin/miniserv.pl /usr/lo
3273 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
3274 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
3275 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
3276 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
3277 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
3280 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
3281 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
8106 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
8112 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
9265 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
9266 ?? IJ 0:00.02 /usr/local/sbin/httpd -DSSL
9267 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
9268 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
9269 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
39540 ?? IJ 0:00.00 /usr/local/sbin/httpd -DSSL
56658 ?? SJ 0:00.03 proxymap -t unix -u
56713 ?? SJ 0:00.02 pickup -l -t fifo -u
56717 ?? SJ 0:00.02 pickup -l -t fifo -u
56718 ?? SJ 0:00.02 pickup -l -t fifo -u
56719 ?? SJ 0:00.02 pickup -l -t fifo -u
56721 ?? SJ 0:00.02 pickup -l -t fifo -u
56772 ?? SJ 0:00.02 pickup -l -t fifo -u
56774 ?? SJ 0:00.02 pickup -l -t fifo -u
57319 ?? S 0:00.02 sshd: xxx [priv] (sshd)
57322 ?? S 0:00.01 sshd: xxx at ttyp0 (sshd)
57323 p0 Ss 0:00.01 -bash (bash)
57325 p0 R+ 0:00.01 ps -ax
16536 v0 Is+ 0:00.00 /usr/libexec/getty Pc ttyv0
142 v1 Is+ 0:00.00 /usr/libexec/getty Pc ttyv1
143 v2 Is+ 0:00.00 /usr/libexec/getty Pc ttyv2
144 v3 Is+ 0:00.00 /usr/libexec/getty Pc ttyv3
145 v4 Is+ 0:00.00 /usr/libexec/getty Pc ttyv4
146 v5 Is+ 0:00.00 /usr/libexec/getty Pc ttyv5
147 v6 Is+ 0:00.00 /usr/libexec/getty Pc ttyv6
148 v7 Is+ 0:00.00 /usr/libexec/getty Pc ttyv7
>
> There are similar things that also need to work by directly accessing
> kernel memory. In jail, they won't work. That's not cool.
>
>> <Total sidenote- top does not work in jails out of the box, for the
>> this
>> very reason, (paging memory) and this gives me a nice lead to actually
>> get around to hacking a patch to make top work in jails- what a
>> convienence that'd be... (but talk is cheap, so I digress)...>
> Yes, cause top works directly by accessing kernel virtual memory.
>
>> Alex- show me the code. I'm willing to step down on this issue, but
>> currently believe I am correct, and it's worth stating it here to not
>> do
>> any disservice to jailing.
> Which code? I'm just showing you examples. Just because it doesn't say
> "jail" in the source it doesn't mean it isn't necessitated by jail.
>
>>> Yes, but in Xen, you have to hax0r two environments before you can
>>> even *get* to the Host environment!
>>
>> That point not understood on this end of the wire, here's why: If a
>> cracker gets through the Xen enviornment and into the host, how is the
>> host *not* then compromised? I mean, it's still all the same
>> hardware,
>> still the same system.
> Again, you are either not getting it or intentionally misunderstanding
> my
> words. Assume you have a bug in a certain syscall kernel that causes
> OS to
> crash when it is executed with certain parameters. With jail, you are
> immediately screwed.
>
>> I would additionally argue that a hole *could* be burrowed right
>> through
>> the OS instance, (I'd start digging at the kernel mods for Xen
>> modified
>> system calls)- and burrow straight into the host OS, precisely the
>> same
>> threat model that jail(2) faces. Layers in between don't matter, big
>> picture of this complexity, it's still the same hardware.
>>
>> (/me opens the door for Sunny here hoping he'll pipe in with some
>> words
>> on yummy hardware separation tech?)
> No it isn't. Xen runs code in a different security ring. (Ring 0, ring
> 1,
> ring 2, ring 3). Read up on intel security contexts. When you are
> running
> inside a guest OS on Xen, your syscalls are handled by the "guest OS"
> kernel. Guest OS can make a syscall into Xen. Xen can make calls into
> Host
> kernel. There is no direct way for a guest OS to make a syscall into
> Host
> kernel. Period.
>
> <snip>
>> Can you tell me *why* my abstract opinion presented, is not correct?
>>
>> With both Xen or Jail(8), restricting resources, is, restricting
>> resources. Or is there something I'm missing here?
> Yes, you are. You can't restrict global kernel resources with jail,
> because you can't, kernel doesn't account for those (and accounting
> for it
> would take hit on performance and be "more code to maintain"). You
> cannot
> say that jailed user "cannot use more than 512M of *ANY KIND OF
> MEMORY*".
> Not possible.
>
>>>> It is not however necessary in the context of my managed virtualized
>>>> servers, that my jails have a kernel- and actually is something I
>>>> prefer, in the context of type of app/web development I do. It's
>>>> that simple- and there really isn't anything you've stated here Alex
>>>> which constructively changes my preference- (though I'm open to
>>>> change my mind if there's something that I'm missing here??)
>>> When one of your users starts to continuously locally-ddos your
>>> machine, you might change your mind on VMs.
>>
>> Do you now mean network-based DDoS over the local network or
>> localhost?
> Local DDoS.
>
>> Would this not be mitigated by running various forms of bandwidth
>> shaping and packet filtering, thoughtfully taking into consideration
>> communications between abstracted interfaces? (I've worked in big
>> jail
>> clusters which were NAT'd on the host server, routing was very
>> modular,
>> controllable, predictable... well worth the added overhead and
>> complexity...)
> No.
>
>> Ok, so to totally put a halt to the FUD about features here, and after
>> reading up on Xen, I don't see any features which are not covered when
>> running jails. Here's why:
>>
>>> You are able to use Xen to control damage from resource-based attacks
>>> far more effectively than jail - for example, by telling Xen not to
>>> allocate more than 256M to a given OS, no matter what. You just can't
>>> do that with jail.
>>
>> Actually, yes- you can do this with jailed systems, though it's done
>> differently, insomuch as the scope of jailing as a whole is totally
>> different, it leverages basic, time-tested, expendable, replacable
>> unix
>> utilities to cover the features which Xen provides for itself. There
>> is
>> no 'can't do' anything here, basic examples would be:
> Jesus christ. I feel that I'm arguing with a GenToo user who just been
> told that FreeBSD is much cooler. You have no understanding of how the
> kernel works.
Be nice. . .
<snipped further. . .>
g
More information about the talk
mailing list