[nycbug-talk] shared hosting
Wed Jan 26 21:23:26 EST 2005
Hi Alex, All,
Alex, I'm not meaning any of my disagreements/corrections to your
writing below as a flame, but am hoping, as usual, to engage your
comments in the most constructive and positive manner possible.
(In short- I like you and generally agree with what you say, but here I
don't agree with much you said, especailly in the context of my
On Jan 26, 2005, at 7:08 PM, alex at pilosoft.com wrote:
> On Wed, 26 Jan 2005, Isaac Levy wrote:
>> I don't mean to be confrontational with what folows, seeing as you and
>> I have a decent dialouge with some lively banter, but I gotta ask you
>> to put your money where your mouth is with XEN here man.
>> Sunny- for the purposes of the archive for this BSD oriented list, can
>> you please explain *why* XEN is more powerful/flexible than jail(8)?
> Apples and oranges. Xen runs another OS, jail restricts a "root" user
> less-than-root privileges in a single OS environment OS.
Gotcha. How is this different than a nicely configured unix group?
> Can't compare
> that. If anything, it is like comparing separation offered by having
> multiple users to a separation offered by having multiple hosts.
So let me get this straight, XEN is not a Virtual Machine whose
intention is to *completely* separate mutually untrusted users? How is
it then really all that fundamentally different than UNIX privilege
separation, and why then is it self-described as a 'Virtual Machine'?
Is this a misnomer of a title, or am I missing something?
>> I'm not sure, but for a BSD based mailing list, I'd think that it
>> be clearly stated that you'd be running a NetBSD Virtual Machine
>> running inside of XEN, on a Linux distro?
> And why does it matter for you?
Woah tiger- because I'm thinking about mabye buying some service from
Sunny, out of interest in exploring what XEN/Sunny has to offer- as I'm
interested in the tech. Sorry if my wording sounded inflammatory, but
this is in the end, a BSD oriented list.
>> - Does XEN support VM's of other OS's, or does only NetBSD fit the
>> due to it's rep. for hardware compatability? (i.e. to meet some funky
>> stuff in the VM?)
> XEN supports lots of things, including windows.
I just read this:
1.3 Which OSes run on Xen?
To achieve such high performance, Xen requires that OSes are ported to
run on it. So far we have stable ports of Linux 2.4, Linux 2.6, and
NetBSD. Ports of FreeBSD and Plan 9 are nearing completion.
1.4 Does Xen support Microsoft Windows?
Unfortunately there are no plans to support any versions of Windows in
the near future. Furthermore, a port of Windows would be encumbered by
licensing issues. Longer term, virtualisation features in
next-generation CPUs should make it much easier to support unmodified
OSes: at that time we will reconsider Windows sup
Alex- let me then re-phrase my question to be clearer, in the next
>> - Are there any fundamental differences in how the VM accesses
>> memory, networking, etc... which is different than in jail(8)'ing or
>> User Mode Linux (UML)?
> Xen is *similar* to UML. Xen is also similar to vmware/bochs. Xen is
> somewhere in the middle between vmware/bochs and UML - in other words,
> explicit support for Xen is necessary for a 'guest OS' (like UML but
> like vmware) and explicit support for Xen is necessary for host (like
> vmware but not like UML). Those tradeoffs give Xen significant
> boost compared to either vmware or UML.
Do other OS's running unmodified, run in a manner which they will meet
production-level expections for use (espeically in the context of
>> - How long has XEN existed, and beyond the commercial backing, what
>> of history does XEN have for stability and maturity in
>> environments, especially large-scale systems running on the public
> Meh. What kind of history does vmware or bochs have? apples to apples,
I really wasn't referring to vmware or bochs, but was alluding to
Jail(8)'ing, which has more to do with the overall history of FreeBSD
itself, (not to shabby a rep for stability/security/performance <g>).
>> jail(8) is fundamentally ridiculously simple by design, which I see as
>> an important factor when working with the complexity which arises, in
>> the context of virtualizing services as complex as Operating Systems.
> And ridiculously insecure.
Ok, Alex, I hate to call you out, but that statement is a bit
far-fetched, I have to say, insomuch as your next statement correctly
summarizes the security context for jail(8),
> If there is a kernel bug in host OS, very high
> chance you can be bitten by it if are running in a jail(8).
you are talking about the security of the Host enviornment, which is
likewise an issue in an XEN enviornment.
(Additionally, not meaning this to be flame bait here, but seriously
when was the last time this issue became significant on any BSD OS? Or
Linux OS? I mean this is a whole other discussion.)
> If there is
> local ddos exploit, you will be able to exploit it in jail(8).
What do you mean by local ddos exploit? Local resource-based attacks
like fork bombs or memory hogs?
If so, I've not really seen any successful, comprehensive, or easy
solutions to this problem on ANY platform. In past posts here
discussing jail(8)'ing, good strategies using SecureLevels in FreeBSD
have been discussed, and I'm sure there are many good ways to skin this
cat on various platforms, in different application contexts- but
fundamentally local resouce-based attacks problem ALWAYS has been, and
ALWAYS will be problems which are about
use/features/freedom/flexibility vs. security.
Because of that, recource-based attacks always need to be addressed in
the context of a given situation or big-picture enviornment- and NO OS
platform solves this problem once and for all. (All of us in Open
Source UNIX-land, BSD and Linux alike, take for granted that we all
have such sophisticated and mature tools to help manage these problems
in various ways IMHO.)
> With Xen,
> to do that, you need to 1) find a bug in guest kernel that would allow
> to execute code in Xen context 2) find a bug in Xen that would allow
> to execute code in host context 3) find a bug in host kernel that would
> screw up the machine.
I don't see this as any more or less secure than Jailing, insomuch as I
believe I've nuked your previous statement about the insecurity of
jail(8), (in the context of jail(8) being more or less secure based on
host OS vulnerability- or resource based attacks), but I can speak from
my own experiences on my philosophy:
Large complex systems are more prone to catastrophic failure. Simple
systems can be used effectively to create highly scalable systems,
which themselves become complex.
Therefore, I see managing complex systems with complex tools as being a
bad thing, in general.
That stated, I see no real difference between eating an 'apple or an
orange' here, a vulnerability in jail(8) can surface, though to date
there has been none- (as it is based on time-tested core subsystems,
chroot, IP aliasing, etc...- built from 'simple' stuff).
I would think that cracking XEN *may* be more plausable, insomuch as
larger systems are often filled with many sprue edges and forgotten
Additionally, I feel security is enhanced by shared agendas. In the
history of computintg, Virtual Machines have always had a relatively
limited audience, throughout their continued rises and falls in
popularity over time.
That stated, a few words in support of jail(8)'ing, in the context of
the expectations for web hosting; Dedicated systems for machine
emulation have less developers dedicated to the problem/idea, when
comparing any of the popularized VM systems to jail(8)'ing- e.g., I'd
think there are a LOT more people who are concerned that the FreeBSD
Kernel runs ROCK SOLID, than there are for the whole of the history of
people involved with virtual machines and emulation systems, (any
living Kernel, for that matter have more people/resources involved who
REALLY care). Same goes for things like the networking stack, which
ultimately is tied to IP aliasing features, etc... fundamental stuff.
Put directly, there's more people in the world who care about the
kernel and networking stack in FreeBSD alone (the simple building
blocks for jail(8)), than the engineering core at Intel and IBM put
together, (including some actual engineers at Intel and IBM).
So, all of that said, I'm a big fan of tying my systems into
technologies which are supported by resources more vast than I can
individually comprehend- and have experienced extreme pain when I tie
myself into any solution whose developer base is proportionally smaller
than the often large amounts of code they produce.
It seems that my sentiment is echoed quite directly by some folks in
the Linux world here on KernelTrap,
>> How does XEN help an administrator manage the complexity, and how
>> complex is the actual virtualization mechanism itself? (i.e. is it a
>> 'large' software like VMWare and the like? [I'm asking in the context
>> the relatively few lines of kernel code that make up the whole of
> You'd be surprised how many lines of kernel code actually deal with
> but not really superuser thanks to possibility of being in jail". It is
> not a few.
Ok- for the whole of the *userland* components of Jail(8), yes. It is
more than a few, lets get the facts down though:
The fundamental Jail(2) call, the actual kernel code which does
everything here, is merely a small pointer which leverages chroot-
(which exists as an elementary component in every decent unix I know
With that, I'm not sure about the size of the source code for jailing,
but I just scoped out Robert Watson's full from-scratch
reimplementation of jail(2), which is under 2200 lines, available here:
Watson's jailctl utility itself is a 4kb tarball- again, not much code
in comparison to 2.1 mb tarball of XEN, (which itself is relatively
small, compared to an entire OS),
The full rewrite code for jail is 64kb, the full uncompressed code for
Xen is 9.8mb,
XEN is roughly an order of magnitude of between 150-200 times the code
that goes into Jail(2) and jail(8), on quick glance- so I don't see why
this is is really worth discussing- technically, Xen is much larger
than jail(8) altogether.
You do know your stuff, and knowing about you I'd bet you've spent more
time actually hacking the kernels here than I have, but watch the facts
(and the sprit of the argument)... Jail(2) and Jail(8) combined are
much smaller than all the VM systems I've seen.
> Again, apples and oranges. To virtualize a x86 processor takes lots of
Indeed, agreed on apples vs. oranges. But, again to re-iterate my
position on these issues, I'm not meaning to trash-talk Xen, or UML, or
Linux. I am meaning to clarify expectations for use, and context-
especially when dealing with services running on the most hostile and
bustling network ever, the internet.
Xen, UML, and other virtulizations have other strengths in various
contexts/cultures that Jail(8) does not. For instance, the ability to
run a full kernel, in the context of application and kernel
development, is pretty insanely cool. One out of many fundamentally
It is not however necessary in the context of my managed virtualized
servers, that my jails have a kernel- and actually is something I
prefer, in the context of type of app/web development I do. It's that
simple- and there really isn't anything you've stated here Alex which
constructively changes my preference- (though I'm open to change my
mind if there's something that I'm missing here??)
More information about the talk