[nycbug-talk] shared hosting

G. Rosamond george
Thu Jan 27 23:28:49 EST 2005


On Jan 27, 2005, at 5:29 PM, alex at pilosoft.com wrote:

> On Thu, 27 Jan 2005, Isaac Levy wrote:
>


<very big snip>

>
>> It seems to me, as Sunny and I had discussed long ago, that Linux VM
>> systems really need a Kernel available to the instances more than the
>> BSD's, because so much is done in Linux through the Kernel- whereas 
>> the
>> converse is true across the BSD's, where much more is accomplished
>> through userland tools and subsystems (if not userland, at least
>> subsystems which aren't in the kernel).
> What in the world are you talking about now? Both Linux and bsd's have
> about the same separation of kernelspace and userspace. Name one 
> example.

Woah. . . we can start with the fact that userland for the GNU/Linuxes 
are providing some 200 or whatever interpretations of the kernelspace 
of Linus. . . and the respective BSDs are dealing with the kernel and 
user land in a much more unified manner. . .

>
> <snip>
>>> I beg to differ. Jail-related code is *all over* the OS. Every time a
>>> root privilege is checked, a code-writer needs to think whether one
>>> should also check for 'root but not superuse'.
>>
>> Are you absolutely certain that the code you are referring to is not 
>> the
>> foundation for chroot?  In BSD's other than FreeBSD, jail is often
>> vocabulary used to discribe chrooted processes- (esp. in the OpenBSD
>> scene).
>>
>> I'll be totally happy to stand down corrected, but at this point, I'm
>> going to need to see some examples in the kernel source- and have
>> started poking around...
>>
>> To my knowledge, after looking up/browsing the source for regular 
>> jail,
>> it's even smaller than I'd thought,
> Jail is still chroot on steroids, no matter how much BSD people try to
> claim it isn't.

???  To paraphrase your earlier comments, Alex, it's all apples and 
oranges.

chroot and jails may have started with the same vague goal, but that's 
a bit of a stretch.

A provider might use chroot to provide services to a provider, but 
providing jails is a whole other level of control that a chroot'd 
program won't provide. . .

>
> <snip>
>> To my knowledge, that's all there is to jailing, so that's under 16 kb
>> in source code- (again, vs. the 9.2 mb of Xen- so we're actually at 
>> the
>> magnitude of about 580 times the code, give or take, for the record.)
> Jail-related code is all over kernel. Everywhere you check for 
> superuser,
> you need to check for jailed superuser.

yes, pretty built-in with implementation. . .

>
>> Now, diving deeper into the source, I can't find the jaill-related 
>> code
>> which you state is *all-over*.
>>
>> I found about 550 references to jail, mostly contained in the files
>> mentioned above, and some strewn about in the chroot code, in the 
>> source
>> for PS, expected places- but haven't taken the time to investigate 
>> *too*
>> deeply (but did find a cute EPS diagram which grep ripped 'jail' out
>> of).
> There you go. It's all over. Everywhere kernel needs to check for
> superuser, it needs to check for a jailed superuser.
>
> <snip>
>
>> Well in PS, this makes a heck of a lot of sense, but not from the
>> perspective of a hole as you describe, but in the way which jails are
>> restricted from particular low-level calls, like the calls ps would 
>> make
>> to page memory/cpu for processes.
> ps(1) does not page memory or CPU. Kernel does.
>
> Try to get your language correct, like 'system call'.
>
> Yes, in a syscall, kernel needs to verify whether root is a superuser.
> That means at every place where you check for rootness, you need to 
> also
> check for jailed root-ness.
>
>> So this patch is to make ps function within the contextual confines 
>> of a
>> jail, for practical purposes, and trying to modify this would just 
>> break
>> ps for the jail (an inconvenient problem, for sure)..
> No, that patch was just cosmetic to show whether process is jailed or
> not.
>
> There's a bigger issue: ps used to work through direct access of kernel
> virtual memory. If you allow that access to a jailed user, you open
> yourself for information-leakage attacks. So you don't, and instead 
> make a
> separate system call (or a /proc-like filesystem like linux) to export
> that information. That's code that might not have been needed if not 
> for
> jail.
>
> Also, to me, the fact that jailed user can still see 'ps' for *every*
> process, not just owned by same user is a big information leak. And if 
> you
> say that its possible to add code to prevent that leak by having
> /proc-like-filesystem present different views - well, that's even more
> jail-related-code that doesn't need to exist.


This might clear that question up. . . and note this is 4.10, not 5.x. 
. .

from a jail. . .

[xxx at mail ~]$ sudo ps -ax
   PID  TT  STAT      TIME COMMAND
  2171  ??  SsJ    0:01.04 /usr/sbin/syslogd -s
  2178  ??  SsJ    0:00.54 /usr/sbin/inetd -wW
  2180  ??  IsJ    0:00.66 /usr/sbin/cron
  2182  ??  SsJ    0:00.49 /usr/sbin/sshd
  2251  ??  SsJ    0:03.48 /usr/local/libexec/postfix/master
  2253  ??  SJ     0:01.32 qmgr -l -t fifo -u
  2254  ??  SsJ    0:06.94 /usr/local/sbin/httpd -DSSL
  2270  ??  SsJ    0:05.40 /usr/bin/perl 
/usr/local/lib/webmin/miniserv.pl /usr/lo
  2273  ??  IJ     0:00.10 /usr/local/sbin/httpd -DSSL
  2274  ??  IJ     0:00.11 /usr/local/sbin/httpd -DSSL
  2275  ??  IJ     0:00.01 /usr/local/sbin/httpd -DSSL
  2276  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  2277  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  8106  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  8112  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
56717  ??  SJ     0:00.02 pickup -l -t fifo -u
57311  ??  SJ     0:00.02 sshd: xxx [priv] (sshd)
57314  ??  SJ     0:00.01 sshd: xxx at ttyp0 (sshd)
57315  p0  SsJ    0:00.01 -bash (bash)
57316  p0  R+J    0:00.01 ps -ax

 From the master box. . .


[xxx at xxx xxx]$ sudo ps -ax

   PID  TT  STAT      TIME COMMAND
     0  ??  DLs    0:00.00  (swapper)
     1  ??  ILs    0:00.05 /sbin/init --
     2  ??  DL     0:00.00  (taskqueue)
     3  ??  DL     0:00.05  (usb0)
     4  ??  DL     0:00.00  (usbtask)
     5  ??  DL     0:00.00  (aic_recovery0)
     6  ??  DL     0:00.00  (aic_recovery0)
     7  ??  DL     0:00.00  (aic_recovery1)
     8  ??  DL     0:00.00  (aic_recovery1)
     9  ??  DL     0:00.62  (pagedaemon)
    10  ??  DL     0:00.00  (vmdaemon)
    11  ??  DL     0:02.96  (bufdaemon)
    12  ??  DL     0:08.07  (vnlru)
    13  ??  DL     6:35.22  (syncer)
    32  ??  Is     0:00.00 adjkerntz -i
    91  ??  Ss     0:01.39 /usr/sbin/syslogd -s
    98  ??  Ss     0:00.54 /usr/sbin/inetd -wW -a xxx.xxx.xxx.xxx
   100  ??  Is     0:00.84 /usr/sbin/cron
   102  ??  Ss     0:00.53 /usr/sbin/sshd -4
   104  ??  Ss     0:01.05 /usr/sbin/usbd
   107  ??  Ss     0:09.20 sendmail: accepting connections (sendmail)
   110  ??  Is     0:00.16 sendmail: Queue runner at 00:30:00 for 
/var/spool/clientmq
   134  ??  Ss     0:00.63 moused -p /dev/psm0 -t auto
  2171  ??  SsJ    0:01.04 /usr/sbin/syslogd -s
  2178  ??  SsJ    0:00.54 /usr/sbin/inetd -wW
  2180  ??  IsJ    0:00.66 /usr/sbin/cron
  2182  ??  SsJ    0:00.49 /usr/sbin/sshd
  2251  ??  SsJ    0:03.48 /usr/local/libexec/postfix/master
  2253  ??  SJ     0:01.32 qmgr -l -t fifo -u
  2254  ??  SsJ    0:06.95 /usr/local/sbin/httpd -DSSL
  2270  ??  SsJ    0:05.40 /usr/bin/perl 
/usr/local/lib/webmin/miniserv.pl /usr/lo
  2273  ??  IJ     0:00.10 /usr/local/sbin/httpd -DSSL
  2274  ??  IJ     0:00.11 /usr/local/sbin/httpd -DSSL
  2275  ??  IJ     0:00.01 /usr/local/sbin/httpd -DSSL
  2276  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  2277  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  2336  ??  SsJ    0:01.04 /usr/sbin/syslogd -s
  2343  ??  SsJ    0:00.54 /usr/sbin/inetd -wW
  2345  ??  IsJ    0:00.64 /usr/sbin/cron
  2347  ??  SsJ    0:00.46 /usr/sbin/sshd
  2416  ??  SsJ    0:03.42 /usr/local/libexec/postfix/master
  2418  ??  SJ     0:01.36 qmgr -l -t fifo -u
  2419  ??  SsJ    0:07.03 /usr/local/sbin/httpd -DSSL
  2434  ??  SsJ    0:05.54 /usr/bin/perl 
/usr/local/lib/webmin/miniserv.pl /usr/lo
  2437  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2438  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2439  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2440  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2441  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2500  ??  SsJ    0:01.00 /usr/sbin/syslogd -s
  2507  ??  SsJ    0:00.57 /usr/sbin/inetd -wW
  2509  ??  IsJ    0:00.65 /usr/sbin/cron
  2511  ??  SsJ    0:00.48 /usr/sbin/sshd
  2580  ??  SsJ    0:03.40 /usr/local/libexec/postfix/master
  2582  ??  SJ     0:01.34 qmgr -l -t fifo -u
  2583  ??  SsJ    0:06.99 /usr/local/sbin/httpd -DSSL
  2598  ??  SsJ    0:05.60 /usr/bin/perl 
/usr/local/lib/webmin/miniserv.pl /usr/lo
  2601  ??  IJ     0:00.10 /usr/local/sbin/httpd -DSSL
  2602  ??  IJ     0:00.10 /usr/local/sbin/httpd -DSSL
  2603  ??  IJ     0:00.10 /usr/local/sbin/httpd -DSSL
  2604  ??  IJ     0:00.01 /usr/local/sbin/httpd -DSSL
  2605  ??  IJ     0:00.11 /usr/local/sbin/httpd -DSSL
  2664  ??  SsJ    0:02.12 /usr/sbin/syslogd -s
  2671  ??  SsJ    0:00.54 /usr/sbin/inetd -wW
  2673  ??  IsJ    0:00.63 /usr/sbin/cron
  2675  ??  SsJ    0:00.48 /usr/sbin/sshd
  2747  ??  SsJ    0:07.36 /usr/libexec/postfix/master
  2749  ??  SJ     0:01.33 qmgr -l -t fifo -u
  2750  ??  SsJ    0:06.83 /usr/local/sbin/httpd -DSSL
  2766  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2767  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2768  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2769  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2770  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2844  ??  SsJ    0:01.05 /usr/sbin/syslogd -s
  2851  ??  SsJ    0:00.52 /usr/sbin/inetd -wW
  2853  ??  IsJ    0:00.64 /usr/sbin/cron
  2855  ??  SsJ    0:00.48 /usr/sbin/sshd
  2924  ??  SsJ    0:03.46 /usr/local/libexec/postfix/master
  2926  ??  SJ     0:01.33 qmgr -l -t fifo -u
  2927  ??  SsJ    0:07.28 /usr/local/sbin/httpd -DSSL
  2942  ??  SsJ    0:05.41 /usr/bin/perl 
/usr/local/lib/webmin/miniserv.pl /usr/lo
  2945  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  2946  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2947  ??  IJ     0:00.01 /usr/local/sbin/httpd -DSSL
  2948  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  2949  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  3008  ??  SsJ    0:01.02 /usr/sbin/syslogd -s
  3015  ??  SsJ    0:00.52 /usr/sbin/inetd -wW
  3017  ??  IsJ    0:00.64 /usr/sbin/cron
  3019  ??  SsJ    0:00.48 /usr/sbin/sshd
  3088  ??  SsJ    0:03.40 /usr/local/libexec/postfix/master
  3090  ??  SJ     0:01.32 qmgr -l -t fifo -u
  3091  ??  SsJ    0:06.99 /usr/local/sbin/httpd -DSSL
  3106  ??  SsJ    0:05.46 /usr/bin/perl 
/usr/local/lib/webmin/miniserv.pl /usr/lo
  3109  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  3110  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  3111  ??  IJ     0:00.01 /usr/local/sbin/httpd -DSSL
  3112  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  3113  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  3172  ??  SsJ    0:01.00 /usr/sbin/syslogd -s
  3179  ??  SsJ    0:00.55 /usr/sbin/inetd -wW
  3181  ??  IsJ    0:00.64 /usr/sbin/cron
  3183  ??  SsJ    0:00.45 /usr/sbin/sshd
  3252  ??  SsJ    0:03.41 /usr/local/libexec/postfix/master
  3254  ??  SJ     0:01.30 qmgr -l -t fifo -u
  3255  ??  SsJ    0:06.74 /usr/local/sbin/httpd -DSSL
  3270  ??  SsJ    0:05.59 /usr/bin/perl 
/usr/local/lib/webmin/miniserv.pl /usr/lo
  3273  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  3274  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  3275  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  3276  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  3277  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  3280  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  3281  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  8106  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  8112  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  9265  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  9266  ??  IJ     0:00.02 /usr/local/sbin/httpd -DSSL
  9267  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  9268  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
  9269  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
39540  ??  IJ     0:00.00 /usr/local/sbin/httpd -DSSL
56658  ??  SJ     0:00.03 proxymap -t unix -u
56713  ??  SJ     0:00.02 pickup -l -t fifo -u
56717  ??  SJ     0:00.02 pickup -l -t fifo -u
56718  ??  SJ     0:00.02 pickup -l -t fifo -u
56719  ??  SJ     0:00.02 pickup -l -t fifo -u
56721  ??  SJ     0:00.02 pickup -l -t fifo -u
56772  ??  SJ     0:00.02 pickup -l -t fifo -u
56774  ??  SJ     0:00.02 pickup -l -t fifo -u
57319  ??  S      0:00.02 sshd: xxx [priv] (sshd)
57322  ??  S      0:00.01 sshd: xxx at ttyp0 (sshd)
57323  p0  Ss     0:00.01 -bash (bash)
57325  p0  R+     0:00.01 ps -ax
16536  v0  Is+    0:00.00 /usr/libexec/getty Pc ttyv0
   142  v1  Is+    0:00.00 /usr/libexec/getty Pc ttyv1
   143  v2  Is+    0:00.00 /usr/libexec/getty Pc ttyv2
   144  v3  Is+    0:00.00 /usr/libexec/getty Pc ttyv3
   145  v4  Is+    0:00.00 /usr/libexec/getty Pc ttyv4
   146  v5  Is+    0:00.00 /usr/libexec/getty Pc ttyv5
   147  v6  Is+    0:00.00 /usr/libexec/getty Pc ttyv6
   148  v7  Is+    0:00.00 /usr/libexec/getty Pc ttyv7


>
> There are similar things that also need to work by directly accessing
> kernel memory. In jail, they won't work. That's not cool.
>
>> <Total sidenote- top does not work in jails out of the box, for the 
>> this
>> very reason, (paging memory) and this gives me a nice lead to actually
>> get around to hacking a patch to make top work in jails- what a
>> convienence that'd be... (but talk is cheap, so I digress)...>
> Yes, cause top works directly by accessing kernel virtual memory.
>
>> Alex- show me the code.  I'm willing to step down on this issue, but
>> currently believe I am correct, and it's worth stating it here to not 
>> do
>> any disservice to jailing.
> Which code? I'm just showing you examples. Just because it doesn't say
> "jail" in the source it doesn't mean it isn't necessitated by jail.
>
>>> Yes, but in Xen, you have to hax0r two environments before you can
>>> even *get* to the Host environment!
>>
>> That point not understood on this end of the wire, here's why:  If a
>> cracker gets through the Xen enviornment and into the host, how is the
>> host *not* then compromised?  I mean, it's still all the same 
>> hardware,
>> still the same system.
> Again, you are either not getting it or intentionally misunderstanding 
> my
> words. Assume you have a bug in a certain syscall kernel that causes 
> OS to
> crash when it is executed with certain parameters. With jail, you are
> immediately screwed.
>
>> I would additionally argue that a hole *could* be burrowed right 
>> through
>> the OS instance, (I'd start digging at the kernel mods for Xen 
>> modified
>> system calls)- and burrow straight into the host OS, precisely the 
>> same
>> threat model that jail(2) faces.  Layers in between don't matter, big
>> picture of this complexity, it's still the same hardware.
>>
>> (/me opens the door for Sunny here hoping he'll pipe in with some 
>> words
>> on yummy hardware separation tech?)
> No it isn't. Xen runs code in a different security ring. (Ring 0, ring 
> 1,
> ring 2, ring 3). Read up on intel security contexts. When you are 
> running
> inside a guest OS on Xen, your syscalls are handled by the "guest OS"
> kernel. Guest OS can make a syscall into Xen. Xen can make calls into 
> Host
> kernel. There is no direct way for a guest OS to make a syscall into 
> Host
> kernel. Period.
>
> <snip>
>> Can you tell me *why* my abstract opinion presented, is not correct?
>>
>> With both Xen or Jail(8), restricting resources, is, restricting
>> resources. Or is there something I'm missing here?
> Yes, you are. You can't restrict global kernel resources with jail,
> because you can't, kernel doesn't account for those (and accounting 
> for it
> would take hit on performance and be "more code to maintain"). You 
> cannot
> say that jailed user "cannot use more than 512M of *ANY KIND OF 
> MEMORY*".
> Not possible.
>
>>>> It is not however necessary in the context of my managed virtualized
>>>> servers, that my jails have a kernel- and actually is something I
>>>> prefer, in the context of type of app/web development I do.  It's
>>>> that simple- and there really isn't anything you've stated here Alex
>>>> which constructively changes my preference- (though I'm open to
>>>> change my mind if there's something that I'm missing here??)
>>> When one of your users starts to continuously locally-ddos your
>>> machine, you might change your mind on VMs.
>>
>> Do you now mean network-based DDoS over the local network or 
>> localhost?
> Local DDoS.
>
>>   Would this not be mitigated by running various forms of bandwidth
>> shaping and packet filtering, thoughtfully taking into consideration
>> communications between abstracted interfaces?  (I've worked in big 
>> jail
>> clusters which were NAT'd on the host server, routing was very 
>> modular,
>> controllable, predictable... well worth the added overhead and
>> complexity...)
> No.
>
>> Ok, so to totally put a halt to the FUD about features here, and after
>> reading up on Xen, I don't see any features which are not covered when
>> running jails.  Here's why:
>>
>>> You are able to use Xen to control damage from resource-based attacks
>>> far more effectively than jail - for example, by telling Xen not to
>>> allocate more than 256M to a given OS, no matter what. You just can't
>>> do that with jail.
>>
>> Actually, yes- you can do this with jailed systems, though it's done
>> differently, insomuch as the scope of jailing as a whole is totally
>> different, it leverages basic, time-tested, expendable, replacable 
>> unix
>> utilities to cover the features which Xen provides for itself.  There 
>> is
>> no 'can't do' anything here, basic examples would be:
> Jesus christ. I feel that I'm arguing with a GenToo user who just been
> told that FreeBSD is much cooler. You have no understanding of how the
> kernel works.

Be nice. . .

<snipped further. . .>

g





More information about the talk mailing list