[nycbug-talk] shared hosting

[Isaac Levy]

Hi Alex, All,

Alex, I'm not meaning any of my disagreements/corrections to your 
writing below as a flame, but am hoping, as usual, to engage your 
comments in the most constructive and positive manner possible.

(In short- I like you and generally agree with what you say, but here I 
don't agree with much you said, especailly in the context of my 
original meaning.)

On Jan 26, 2005, at 7:08 PM, alex at pilosoft.com wrote:

> On Wed, 26 Jan 2005, Isaac Levy wrote:
>
>> I don't mean to be confrontational with what folows, seeing as you and
>> I have a decent dialouge with some lively banter, but I gotta ask you
>> to put your money where your mouth is with XEN here man.
>
> <snip>
>> Question:
>>
>> Sunny- for the purposes of the archive for this BSD oriented list, can
>> you please explain *why* XEN is more powerful/flexible than jail(8)?
> Apples and oranges. Xen runs another OS, jail restricts a "root" user 
> to
> less-than-root privileges in a single OS environment OS.

Gotcha.  How is this different than a nicely configured unix group?

> Can't compare
> that. If anything, it is like comparing separation offered by having
> multiple users to a separation offered by having multiple hosts.

So let me get this straight, XEN is not a Virtual Machine whose 
intention is to *completely* separate mutually untrusted users?  How is 
it then really all that fundamentally different than UNIX privilege 
separation, and why then is it self-described as a 'Virtual Machine'?

Is this a misnomer of a title, or am I missing something?

>
>> I'm not sure, but for a BSD based mailing list, I'd think that it 
>> should
>> be clearly stated that you'd be running a NetBSD Virtual Machine 
>> image,
>> running inside of XEN, on a Linux distro?
> And why does it matter for you?

Woah tiger- because I'm thinking about mabye buying some service from 
Sunny, out of interest in exploring what XEN/Sunny has to offer- as I'm 
interested in the tech.  Sorry if my wording sounded inflammatory, but 
this is in the end, a BSD oriented list.

>
>> - Does XEN support VM's of other OS's, or does only NetBSD fit the 
>> bill
>> due to it's rep. for hardware compatability?  (i.e. to meet some funky
>> stuff in the VM?)
> XEN supports lots of things, including windows.

I just read this:
http://www.cl.cam.ac.uk/Research/SRG/netos/xen/faq.html#a1.3

  1.3 Which OSes run on Xen?

  To achieve such high performance, Xen requires that OSes are ported to 
run on it. So far we have stable ports of Linux 2.4, Linux 2.6, and 
NetBSD. Ports of FreeBSD and Plan 9 are nearing completion.

  1.4 Does Xen support Microsoft Windows?

  Unfortunately there are no plans to support any versions of Windows in 
the near future. Furthermore, a port of Windows would be encumbered by 
licensing issues. Longer term, virtualisation features in 
next-generation CPUs should make it much easier to support unmodified 
OSes: at that time we will reconsider Windows sup
--

Alex- let me then re-phrase my question to be clearer, in the next 
paragraph:

>
>> - Are there any fundamental differences in how the VM accesses 
>> devices,
>> memory, networking, etc... which is different than in jail(8)'ing or
>> User Mode Linux (UML)?
> Xen is *similar* to UML. Xen is also similar to vmware/bochs. Xen is
> somewhere in the middle between vmware/bochs and UML - in other words,
> explicit support for Xen is necessary for a 'guest OS' (like UML but 
> not
> like vmware) and explicit support for Xen is necessary for host (like
> vmware but not like UML). Those tradeoffs give Xen significant 
> performance
> boost compared to either vmware or UML.

Do other OS's running unmodified, run in a manner which they will meet 
production-level expections for use (espeically in the context of 
internet applications)?


>
>> - How long has XEN existed, and beyond the commercial backing, what 
>> kind
>> of history does XEN have for stability and maturity in 
>> production-level
>> environments, especially large-scale systems running on the public
>> internet?
> Meh. What kind of history does vmware or bochs have? apples to apples,
> please.

I really wasn't referring to vmware or bochs, but was alluding to 
Jail(8)'ing, which has more to do with the overall history of FreeBSD 
itself, (not to shabby a rep for stability/security/performance <g>).

>
>> jail(8) is fundamentally ridiculously simple by design, which I see as
>> an important factor when working with the complexity which arises, in
>> the context of virtualizing services as complex as Operating Systems.
> And ridiculously insecure.

Ok, Alex, I hate to call you out, but that statement is a bit 
far-fetched, I have to say, insomuch as your next statement correctly 
summarizes the security context for jail(8),

> If there is a kernel bug in host OS, very high
> chance you can be bitten by it if are running in a jail(8).

you are talking about the security of the Host enviornment, which is 
likewise an issue in an XEN enviornment.

(Additionally, not meaning this to be flame bait here, but seriously 
when was the last time this issue became significant on any BSD OS?  Or 
Linux OS?  I mean this is a whole other discussion.)


> If there is
> local ddos exploit, you will be able to exploit it in jail(8).

What do you mean by local ddos exploit?  Local resource-based attacks 
like fork bombs or memory hogs?

If so, I've not really seen any successful, comprehensive, or easy 
solutions to this problem on ANY platform.  In past posts here 
discussing jail(8)'ing, good strategies using SecureLevels in FreeBSD 
have been discussed, and I'm sure there are many good ways to skin this 
cat on various platforms, in different application contexts- but 
fundamentally local resouce-based attacks problem ALWAYS has been, and 
ALWAYS will be problems which are about 
use/features/freedom/flexibility vs. security.

Because of that, recource-based attacks always need to be addressed in 
the context of a given situation or big-picture enviornment- and NO OS 
platform solves this problem once and for all.  (All of us in Open 
Source UNIX-land, BSD and Linux alike, take for granted that we all 
have such sophisticated and mature tools to help manage these problems 
in various ways IMHO.)

> With Xen,
> to do that, you need to 1) find a bug in guest kernel that would allow 
> you
> to execute code in Xen context 2) find a bug in Xen that would allow 
> you
> to execute code in host context 3) find a bug in host kernel that would
> screw up the machine.

I don't see this as any more or less secure than Jailing, insomuch as I 
believe I've nuked your previous statement about the insecurity of 
jail(8), (in the context of jail(8) being more or less secure based on 
host OS vulnerability- or resource based attacks), but I can speak from 
my own experiences on my philosophy:

Large complex systems are more prone to catastrophic failure.  Simple 
systems can be used effectively to create highly scalable systems, 
which themselves become complex.
Therefore, I see managing complex systems with complex tools as being a 
bad thing, in general.

--
That stated, I see no real difference between eating an 'apple or an 
orange' here, a vulnerability in jail(8) can surface, though to date 
there has been none- (as it is based on time-tested core subsystems, 
chroot, IP aliasing, etc...- built from 'simple' stuff).

I would think that cracking XEN *may* be more plausable, insomuch as 
larger systems are often filled with many sprue edges and forgotten 
pathways...

Additionally, I feel security is enhanced by shared agendas.  In the 
history of computintg, Virtual Machines have always had a relatively 
limited audience, throughout their continued rises and falls in 
popularity over time.

That stated, a few words in support of jail(8)'ing, in the context of 
the expectations for web hosting;  Dedicated systems for machine 
emulation have less developers dedicated to the problem/idea, when 
comparing any of the popularized VM systems to jail(8)'ing- e.g., I'd 
think there are a LOT more people who are concerned that the FreeBSD 
Kernel runs ROCK SOLID, than there are for the whole of the history of 
people involved with virtual machines and emulation systems, (any 
living Kernel, for that matter have more people/resources involved who 
REALLY care).  Same goes for things like the networking stack, which 
ultimately is tied to IP aliasing features, etc... fundamental stuff.

Put directly, there's more people in the world who care about the 
kernel and networking stack in FreeBSD alone (the simple building 
blocks for jail(8)), than the engineering core at Intel and IBM put 
together, (including some actual engineers at Intel and IBM).

--
So, all of that said, I'm a big fan of tying my systems into 
technologies which are supported by resources more vast than I can 
individually comprehend- and have experienced extreme pain when I tie 
myself into any solution whose developer base is proportionally smaller 
than the often large amounts of code they produce.

It seems that my sentiment is echoed quite directly by some folks in 
the Linux world here on KernelTrap,

http://kerneltrap.org/node/3981


>
>> How does XEN help an administrator manage the complexity, and how
>> complex is the actual virtualization mechanism itself?  (i.e. is it a
>> 'large' software like VMWare and the like? [I'm asking in the context 
>> of
>> the relatively few lines of kernel code that make up the whole of
>> Jail(8)])
> You'd be surprised how many lines of kernel code actually deal with 
> "root
> but not really superuser thanks to possibility of being in jail". It is
> not a few.

Ok- for the whole of the *userland* components of Jail(8), yes.  It is 
more than a few, lets get the facts down though:

--
The fundamental Jail(2) call, the actual kernel code which does 
everything here, is merely a small pointer which leverages chroot- 
(which exists as an elementary component in every decent unix I know 
of).
With that, I'm not sure about the size of the source code for jailing, 
but I just scoped out Robert Watson's full from-scratch 
reimplementation of jail(2), which is under 2200 lines, available here:

http://www.watson.org/~robert/freebsd/jailng/

Watson's jailctl utility itself is a 4kb tarball- again, not much code 
in comparison to 2.1 mb tarball of XEN, (which itself is relatively 
small, compared to an entire OS),

The full rewrite code for jail is 64kb, the full uncompressed code for 
Xen is 9.8mb,

XEN is roughly an order of magnitude of between 150-200 times the code 
that goes into Jail(2) and jail(8), on quick glance- so I don't see why 
this is is really worth discussing- technically, Xen is much larger 
than jail(8) altogether.


You do know your stuff, and knowing about you I'd bet you've spent more 
time actually hacking the kernels here than I have, but watch the facts 
(and the sprit of the argument)...  Jail(2) and Jail(8) combined are 
much smaller than all the VM systems I've seen.


>
> Again, apples and oranges. To virtualize a x86 processor takes lots of
> code.

Indeed, agreed on apples vs. oranges.  But, again to re-iterate my 
position on these issues, I'm not meaning to trash-talk Xen, or UML, or 
Linux.  I am meaning to clarify expectations for use, and context- 
especially when dealing with services running on the most hostile and 
bustling network ever, the internet.

Xen, UML, and other virtulizations have other strengths in various 
contexts/cultures that Jail(8) does not.  For instance, the ability to 
run a full kernel, in the context of application and kernel 
development, is pretty insanely cool.  One out of many fundamentally 
different ideas.

It is not however necessary in the context of my managed virtualized 
servers, that my jails have a kernel- and actually is something I 
prefer, in the context of type of app/web development I do.  It's that 
simple- and there really isn't anything you've stated here Alex which 
constructively changes my preference- (though I'm open to change my 
mind if there's something that I'm missing here??)


> -alex
>

Rocket-
.ike