[nycbug-talk] shared hosting

Thu Jan 27 16:56:25 EST 2005

Wordup Alex, All,

Based on what Sunny wrote last night, I'm now a big sideline fan of 
Xen, but for the record I believe I need to clarify some of the facts 
here, for the record.

Alex- I don't want to make this a confrontational thread, because I 
find this new tech all really interesting- so I'll just disregard the 
stuff that feels disrespectful to me from your last post, and I'm sorry 
if I came off as disrespectful in my last post to you.
The hair on my back does however raise when I get slapped with what 
sound like absolutist imperatives by folks I don't know too well, for 
things which are fairly abstract.  That's all.

>> I'd think that it should
>> be clearly stated that you'd be running a NetBSD Virtual Machine 
>> image,
>> running inside of XEN, on a Linux distro?
> And why does it matter for you?

>> Same goes for things like the networking stack, which ultimately
>> is tied to IP aliasing features, etc... fundamental stuff.
> I would think that out of all people, BSD users would be loath to 
> equate
> popular to good, thus I won't comment on that.

Notably, with regard to my obviously BSD slanted sentiment, THIS IS A 
BSD TECH LIST.  Quit it.
It's my business if I like the BSD's, and BSD licensing, and Alex, you 
need to not get so uppity about a slant TOWARDS the BSD's here.  In the 
right context, I'd love to discuss Linux design implementation in 
contrast to the BSD's, as well as GPL vs. BSD in practice, BUT- note 
that I'm talking about tech in both of the above snippets- and I don't 
care to slip into *these* kinds of discussions indiscriminately, (or at 
least until we both have a fair amount of vodka in us, face to face 
<g>).

> Nobody is forcing you to do anything. If jail fits your needs, use it.
It's not about that- we all are doing that anyhow- the spirit of my 
previous mail is about discussing these fascinating technologies here.

Sunny and I have a long-standing sparring dialogue based on mutual 
respect, seeing as he and I both are junkies for running virtualized 
OS'es- so I hope our banter wasn't interpreted as anything actually 
confrontational.  (guess I'll try to throw my punches at Sunny in 
private from now on.
[btw- sunny- schoolyard- 3pm- tomorrow- be there] :)

-------------------
That out of the way, Alex, on the technical side I believe you have 
simply mis-represented some of the facts about jailing, so I'm going to 
try to set things straight, and put the code where my mouth is for this 
archived list.  I'll keep my words on Xen to a minimum, insomuch as 
it's totally new to me, but jailing, is not.  On to the fun stuff;

DESIGN SEMANTICS
--
With regard to features and 'better, worse, secure', Alex made some 
pretty bold statements, yet I still have no solid evidence of anything 
other than this:
Xen and jail() simply come at the same problem with different tools, 
from different directions:

Xen's intended scope seems more all-inclusive for the task at hand, 
while the intended scope of jailing is built to leverage existing 
userland tools;

On Jan 26, 2005, at 10:05 PM, alex at pilosoft.com wrote:
>> Jail(2) and Jail(8) combined are
>> much smaller than all the VM systems I've seen.
> Of course, because they aren't VM systems.

I'd argue, for the technical record online with the archive of this 
thread, that Jailing is an kernel/OS-based VM, and that Xen is an 
emulator-based VM.  I say this because various VM schemes are all based 
around 1 idea- providing full OS-level access to mutually untrusted 
uses/processes.
Virtual Machine =! Emulator, IMHO.

> Xen *is* a virtual machine whose intention is to completely separate
> mutually untrusted *operating systems!*

Understood now, and in the context of web services, (something you, 
Sunny, and I all do for a living in various contexts), don't we all 
meet the same end goal with both Xen and jail(8)?

>> I really wasn't referring to vmware or bochs, but was alluding to
>> Jail(8)'ing, which has more to do with the overall history of FreeBSD
>> itself, (not to shabby a rep for stability/security/performance <g>).
> Apples to oranges, again. You can't compare xen to jail. You just 
> can't.
> Different things. Designed to solve different problems.

Seems like- but then what I'm still wondering is, what are the Xen 
advantages when applied to hosting common internet applications?

> If your problem *can* be solved by jail, and you trust jail enough, use
> jail. Xen is designed to solve a different problem.

What different problem?!

I can discuss the implementation for the problem Jailing was designed 
to solve, perhaps somebody could contrast it with Xen's purpose:

 From McKusick's latest book (http://www.mckusick.com/FreeBSDbook.html),

'Two important goals of the jail implementation are to:

1. Retain the semantics of the existing discretionary access-control 
mechanisms

2. Allow each jail to have it's own superuser administrator whose 
activities are limited to the processes, files, and network associated 
with it's jail'

The implimentation scope is actually so small it's really only got 
these two stated design goals, (which themselves are 
dissected/discussed/scrubbed fairly extensively).

>> Do other OS's running unmodified, run in a manner which they will meet
>> production-level expections for use (espeically in the context of
>> internet applications)?
> They *are* modified. Modifications are necessary to gain the highest
> performance under the virtual machine environment.

This all makes sense now- but it still doesn't answer my question, 
(though Sunny basically did),

If I have Xen basics straight, I'm still wondering if the modifications 
are simply a performance-gain issue; do they affect production-level 
expectations (perhaps expectations for use in common internet 
services/applications)?

It seems to me, as Sunny and I had discussed long ago, that Linux VM 
systems really need a Kernel available to the instances more than the 
BSD's, because so much is done in Linux through the Kernel- whereas the 
converse is true across the BSD's, where much more is accomplished 
through userland tools and subsystems (if not userland, at least 
subsystems which aren't in the kernel).

>> That stated, I see no real difference between eating an 'apple or an
>> orange' here, a vulnerability in jail(8) can surface, though to date
>> there has been none- (as it is based on time-tested core subsystems,
>> chroot, IP aliasing, etc...- built from 'simple' stuff).
>>
>> I would think that cracking XEN *may* be more plausable, insomuch as
>> larger systems are often filled with many sprue edges and forgotten
>> pathways...
> I beg to differ. Jail-related code is *all over* the OS. Every time a 
> root
> privilege is checked, a code-writer needs to think whether one should 
> also
> check for 'root but not superuse'.

Are you absolutely certain that the code you are referring to is not 
the foundation for chroot?  In BSD's other than FreeBSD, jail is often 
vocabulary used to discribe chrooted processes- (esp. in the OpenBSD 
scene).

I'll be totally happy to stand down corrected, but at this point, I'm 
going to need to see some examples in the kernel source- and have 
started poking around...

To my knowledge, after looking up/browsing the source for regular jail, 
it's even smaller than I'd thought,

contents of /usr/src/usr.sbin/jail

    161B May  8  2003 Makefile
     14K Jan  2  2002 jail.8
      3K Jan  2  2002 jail.c

To my knowledge, that's all there is to jailing, so that's under 16 kb 
in source code- (again, vs. the 9.2 mb of Xen- so we're actually at the 
magnitude of about 580 times the code, give or take, for the record.)

-
Now, diving deeper into the source, I can't find the jaill-related code 
which you state is *all-over*.

I found about 550 references to jail, mostly contained in the files 
mentioned above, and some strewn about in the chroot code, in the 
source for PS, expected places-  but haven't taken the time to 
investigate *too* deeply (but did find a cute EPS diagram which grep 
ripped 'jail' out of).

Alex- can you point out the type of example you speak of here, or else 
drop this point, because I believe you are wrong?

An example of why I believe you are wrong, one example below,

/usr/src/bin/ps/print.c line 237,

      cp++;
      if (!(flag & P_INMEM))
           *cp++ = 'W';
      if (p->p_nice < NZERO)
           *cp++ = '<';
      else if (p->p_nice > NZERO)
           *cp++ = 'N';
      if (flag & P_TRACED)
           *cp++ = 'X';
      if (flag & P_WEXIT && p->p_stat != SZOMB)
           *cp++ = 'E';
      if (flag & P_PPWAIT)
           *cp++ = 'V';
      if ((flag & P_SYSTEM) || p->p_lock > 0)
           *cp++ = 'L';
      if (KI_EPROC(k)->e_flag & EPROC_SLEADER)
           *cp++ = 's';
      if ((flag & P_CONTROLT) && KI_EPROC(k)->e_pgid == 
KI_EPROC(k)->e_tpgid)
           *cp++ = '+';
 >>>> if (flag & P_JAILED) 
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
           *cp++ = 'J';
      *cp = '\0';
      (void)printf("%-*s", v->width, buf);
}

Well in PS, this makes a heck of a lot of sense, but not from the 
perspective of a hole as you describe, but in the way which jails are 
restricted from particular low-level calls, like the calls ps would 
make to page memory/cpu for processes.

So this patch is to make ps function within the contextual confines of 
a jail, for practical purposes, and trying to modify this would just 
break ps for the jail (an inconvenient problem, for sure)..

Alex- show me the code.  I'm willing to step down on this issue, but 
currently believe I am correct, and it's worth stating it here to not 
do any disservice to jailing.

>> Xen is much larger than
>> jail(8) altogether.
> Apples and oranges. Xen does much more, thus it is necessarily more
> complicated.

Understood in the context of the scope of the systems.

So apples and oranges, yes- and the implimentation differences are 
understood in context.

But I'll continue with the parts that make it all common fruit of a 
particular scale and form, (it's not size that matters, really)...

SECURITY CONCERNS
--
>>>> jail(8) is fundamentally ridiculously simple by design, which I see
>>>> as an important factor when working with the complexity which 
>>>> arises,
>>>> in the context of virtualizing services as complex as Operating
>>>> Systems.
>>> And ridiculously insecure.
>>
>> Ok, Alex, I hate to call you out, but that statement is a bit
>> far-fetched, I have to say, insomuch as your next statement correctly
>> summarizes the security context for jail(8),
>>
>>> If there is a kernel bug in host OS, very high chance you can be
>>> bitten by it if are running in a jail(8).
>>
>> you are talking about the security of the Host enviornment, which is
>> likewise an issue in an XEN enviornment.
> Yes, but in Xen, you have to hax0r two environments before you can even
> *get* to the Host environment!

That point not understood on this end of the wire, here's why:  If a 
cracker gets through the Xen enviornment and into the host, how is the 
host *not* then compromised?  I mean, it's still all the same hardware, 
still the same system.
I would additionally argue that a hole *could* be burrowed right 
through the OS instance, (I'd start digging at the kernel mods for Xen 
modified system calls)- and burrow straight into the host OS, precisely 
the same threat model that jail(2) faces.  Layers in between don't 
matter, big picture of this complexity, it's still the same hardware.

(/me opens the door for Sunny here hoping he'll pipe in with some words 
on yummy hardware separation tech?)

>>> If there is local ddos exploit, you will be able to exploit it in
>>> jail(8).
>>
>> What do you mean by local ddos exploit?  Local resource-based attacks
>> like fork bombs or memory hogs?
> Good example, yes.
>>
>> If so, I've not really seen any successful, comprehensive, or easy
>> solutions to this problem on ANY platform.  In past posts here
>> discussing jail(8)'ing, good strategies using SecureLevels in FreeBSD
>> have been discussed, and I'm sure there are many good ways to skin 
>> this
>> cat on various platforms, in different application contexts- but
>> fundamentally local resouce-based attacks problem ALWAYS has been, and
>> ALWAYS will be problems which are about 
>> use/features/freedom/flexibility
>> vs. security.
>>
>> Because of that, recource-based attacks always need to be addressed in
>> the context of a given situation or big-picture enviornment- and NO OS
>> platform solves this problem once and for all.  (All of us in Open
>> Source UNIX-land, BSD and Linux alike, take for granted that we all 
>> have
>> such sophisticated and mature tools to help manage these problems in
>> various ways IMHO.)

> That's not correct.

Can you tell me *why* my abstract opinion presented, is not correct?

With both Xen or Jail(8), restricting resources, is, restricting 
resources.
Or is there something I'm missing here?

>> It is not however necessary in the context of my managed virtualized
>> servers, that my jails have a kernel- and actually is something I
>> prefer, in the context of type of app/web development I do.  It's that
>> simple- and there really isn't anything you've stated here Alex which
>> constructively changes my preference- (though I'm open to change my 
>> mind
>> if there's something that I'm missing here??)
> When one of your users starts to continuously locally-ddos your 
> machine,
> you might change your mind on VMs.

Do you now mean network-based DDoS over the local network or localhost? 
  Would this not be mitigated by running various forms of bandwidth 
shaping and packet filtering, thoughtfully taking into consideration 
communications between abstracted interfaces?  (I've worked in big jail 
clusters which were NAT'd on the host server, routing was very modular, 
controllable, predictable... well worth the added overhead and 
complexity...)

PRACTICAL JAILING FEATURES
--

Ok, so to totally put a halt to the FUD about features here, and after 
reading up on Xen, I don't see any features which are not covered when 
running jails.  Here's why:

> You are able to use Xen to control damage from resource-based attacks 
> far
> more effectively than jail - for example, by telling Xen not to 
> allocate
> more than 256M to a given OS, no matter what. You just can't do that 
> with
> jail.

Actually, yes- you can do this with jailed systems, though it's done 
differently, insomuch as the scope of jailing as a whole is totally 
different, it leverages basic, time-tested, expendable, replacable unix 
utilities to cover the features which Xen provides for itself.  There  
is no 'can't do' anything here, basic examples would be:

FreeBSD 4.x

- master system running securelevel 0
   (securelevel first appeared as init utility in Version 6 AT&T UNIX)

	- chflags to lock down processes in good ol' login.conf
        by making it immutable (root is really root inside a jail)

	* caveat, securelevel 0 is a restriction which can make
	things surprisingly painful in common use scenarios...

	* Un-caveat!, FreeBSD 5.x impliments securelevel for each jail,
	though I haven't myself gone deep with jailing on 5.x, so many new 
strategy options...

- dummynet or other for traffic shaping (first implimented in 96')
   (I don't know if PF is capable of traffic-shaping on FreeBSD?)

- Quotas, disk partition schemes, or Image-based disk schemes
   for resource-based controls (The quota command appeared in 4.2BSD)

And anything else in the OS can be added to this list to control 
resources in a common manner.

** SERIOUSLY ** What can Xen do that cannot be done with fundamental 
tools on ANY good UNIX?  I am truly interested here.

--
Questions/interests I still have about Xen- but perhaps in the next few 
weeks of reading, I'll learn more about it here:

>> I really wasn't referring to vmware or bochs, but was alluding to
>> Jail(8)'ing, which has more to do with the overall history of FreeBSD
>> itself, (not to shabby a rep for stability/security/performance <g>).
> Apples to oranges, again. You can't compare xen to jail. You just 
> can't.
> Different things. Designed to solve different problems.

Seems like- but then what I'm still wondering is, what are the Xen 
advantages when applied to hosting common internet applications?  What 
does it have over jail(8), feature-wise, or in the fundamental 
differences in use?

> If your problem *can* be solved by jail, and you trust jail enough, use
> jail. Xen is designed to solve a different problem.

What different problem?!  I don't see the different problem, aside from 
kernel-dev abilities?
Jail() 'was designed to solve particular SECURITY problems in ways 
chroot did not address', which ended up manifest as a Virtual Machine.  
It's that simple.
What was Xen intended for in it's design?

--
A few other misc. details snipped in here:

>> Gotcha.  How is this [Xen] different than a nicely configured unix 
>> group?
> It is nothing like it. Compare two different users on same machine to 
> two
> different machines running different operating systems.

Gotcha- my mistake based on the way it was explained.

--------------------

So at the end of this email, all I have found as a major difference is 
the *approach* to confining the VM system, not the feaures, and 
certainly better/worse type arguments don't REALLY apply here.  Period.

In the BSD's, much less is done in the kernel (than is my understanding 
of what is done in Linux kernel)- so from a paradigm perspective, I 
believe we are simply solving similar problem, (Virtualizing an OS to 
provide services to mutually untrusted users), from totally different 
paradigms in UNIX.  That's all.  And I think Xen's side of the solution 
is fascinating- and hope to learn more...

Rocket,
.ike