[nycbug-talk] shared hosting

[alex at pilosoft.com]

On Wed, 26 Jan 2005, Isaac Levy wrote:

> >> Sunny- for the purposes of the archive for this BSD oriented list, can
> >> you please explain *why* XEN is more powerful/flexible than jail(8)?
> > Apples and oranges. Xen runs another OS, jail restricts a "root" user
> > to less-than-root privileges in a single OS environment OS.
> 
> Gotcha.  How is this different than a nicely configured unix group?
It is nothing like it. Compare two different users on same machine to two 
different machines running different operating systems. 

Again, one is not like the other.

> > Can't compare that. If anything, it is like comparing separation
> > offered by having multiple users to a separation offered by having
> > multiple hosts.
> 
> So let me get this straight, XEN is not a Virtual Machine whose
> intention is to *completely* separate mutually untrusted users?  How is
> it then really all that fundamentally different than UNIX privilege
> separation, and why then is it self-described as a 'Virtual Machine'?
Xen *is* a virtual machine whose intention is to completely separate 
mutually untrusted *operating systems!*

<snip - i was wrong about Windows port of Xen>

> >> - Are there any fundamental differences in how the VM accesses
> >> devices, memory, networking, etc... which is different than in
> >> jail(8)'ing or User Mode Linux (UML)?
> > Xen is *similar* to UML. Xen is also similar to vmware/bochs. Xen is
> > somewhere in the middle between vmware/bochs and UML - in other words,
> > explicit support for Xen is necessary for a 'guest OS' (like UML but
> > not like vmware) and explicit support for Xen is necessary for host
> > (like vmware but not like UML). Those tradeoffs give Xen significant
> > performance boost compared to either vmware or UML.
> 
> Do other OS's running unmodified, run in a manner which they will meet
> production-level expections for use (espeically in the context of
> internet applications)?
They *are* modified. Modifications are necessary to gain the highest 
performance under the virtual machine environment.

> >> - How long has XEN existed, and beyond the commercial backing, what
> >> kind of history does XEN have for stability and maturity in
> >> production-level environments, especially large-scale systems running
> >> on the public internet?
> > Meh. What kind of history does vmware or bochs have? apples to apples,
> > please.
> 
> I really wasn't referring to vmware or bochs, but was alluding to
> Jail(8)'ing, which has more to do with the overall history of FreeBSD
> itself, (not to shabby a rep for stability/security/performance <g>).
Apples to oranges, again. You can't compare xen to jail. You just can't. 
Different things. Designed to solve different problems. 

If your problem *can* be solved by jail, and you trust jail enough, use 
jail. Xen is designed to solve a different problem. 

> >> jail(8) is fundamentally ridiculously simple by design, which I see
> >> as an important factor when working with the complexity which arises,
> >> in the context of virtualizing services as complex as Operating
> >> Systems.
> > And ridiculously insecure.
> 
> Ok, Alex, I hate to call you out, but that statement is a bit
> far-fetched, I have to say, insomuch as your next statement correctly
> summarizes the security context for jail(8),
> 
> > If there is a kernel bug in host OS, very high chance you can be
> > bitten by it if are running in a jail(8).
> 
> you are talking about the security of the Host enviornment, which is
> likewise an issue in an XEN enviornment.
Yes, but in Xen, you have to hax0r two environments before you can even 
*get* to the Host environment!

> (Additionally, not meaning this to be flame bait here, but seriously
> when was the last time this issue became significant on any BSD OS?  Or
> Linux OS?  I mean this is a whole other discussion.)
> 
> 
> > If there is local ddos exploit, you will be able to exploit it in
> > jail(8).
> 
> What do you mean by local ddos exploit?  Local resource-based attacks
> like fork bombs or memory hogs?
Good example, yes.
> 
> If so, I've not really seen any successful, comprehensive, or easy
> solutions to this problem on ANY platform.  In past posts here
> discussing jail(8)'ing, good strategies using SecureLevels in FreeBSD
> have been discussed, and I'm sure there are many good ways to skin this
> cat on various platforms, in different application contexts- but
> fundamentally local resouce-based attacks problem ALWAYS has been, and
> ALWAYS will be problems which are about use/features/freedom/flexibility
> vs. security.
> 
> Because of that, recource-based attacks always need to be addressed in
> the context of a given situation or big-picture enviornment- and NO OS
> platform solves this problem once and for all.  (All of us in Open
> Source UNIX-land, BSD and Linux alike, take for granted that we all have
> such sophisticated and mature tools to help manage these problems in
> various ways IMHO.)
That's not correct. There are resource-based attacks and there are local 
DDoS exploits and similar. You are vulnerable to both with jail. You are 
not vulnerable to exploits with Xen, (or rather, the only thing local ddos 
exploit on Xen would do is kill your guest instance). 

You are able to use Xen to control damage from resource-based attacks far
more effectively than jail - for example, by telling Xen not to allocate
more than 256M to a given OS, no matter what. You just can't do that with 
jail. 

> > With Xen, to do that, you need to 1) find a bug in guest kernel that
> > would allow you to execute code in Xen context 2) find a bug in Xen
> > that would allow you to execute code in host context 3) find a bug in
> > host kernel that would screw up the machine.
> 
> I don't see this as any more or less secure than Jailing, insomuch as I
> believe I've nuked your previous statement about the insecurity of
> jail(8), (in the context of jail(8) being more or less secure based on
> host OS vulnerability- or resource based attacks), but I can speak from
> my own experiences on my philosophy:
See above.

> Large complex systems are more prone to catastrophic failure.  Simple
> systems can be used effectively to create highly scalable systems, which
> themselves become complex. Therefore, I see managing complex systems
> with complex tools as being a bad thing, in general.
That all depends. All things being equal, I would also prefer a simpler 
system. However, all things are not equal in this case, and Xen gives you 
far more than jail. 

> That stated, I see no real difference between eating an 'apple or an
> orange' here, a vulnerability in jail(8) can surface, though to date
> there has been none- (as it is based on time-tested core subsystems,
> chroot, IP aliasing, etc...- built from 'simple' stuff).
> 
> I would think that cracking XEN *may* be more plausable, insomuch as
> larger systems are often filled with many sprue edges and forgotten
> pathways...
I beg to differ. Jail-related code is *all over* the OS. Every time a root 
privilege is checked, a code-writer needs to think whether one should also 
check for 'root but not superuse'.

> Additionally, I feel security is enhanced by shared agendas.  In the
> history of computintg, Virtual Machines have always had a relatively
> limited audience, throughout their continued rises and falls in
> popularity over time.
Limited for you may mean a general population who can't afford a
VM-capable system wasn't using VM. IBM mainframes *by nature* run VM and
run every OS under VM. (In the above context VM means IBM's virtual
machine environment). All of your credit cards are likely to be processed
by something that runs under a VM.

> That stated, a few words in support of jail(8)'ing, in the context of
> the expectations for web hosting;  Dedicated systems for machine
> emulation have less developers dedicated to the problem/idea, when
> comparing any of the popularized VM systems to jail(8)'ing- e.g., I'd
> think there are a LOT more people who are concerned that the FreeBSD
> Kernel runs ROCK SOLID, than there are for the whole of the history of
> people involved with virtual machines and emulation systems, (any living
> Kernel, for that matter have more people/resources involved who REALLY
> care).  Same goes for things like the networking stack, which ultimately
> is tied to IP aliasing features, etc... fundamental stuff.
I would think that out of all people, BSD users would be loath to equate
popular to good, thus I won't comment on that. And as far as "rock solid",
you have absolutely no idea how much runs under VM environments on
mainframes.

Yes, VM technology on x86 might not be the most mature. But it is the 
Right Thing to do.

> Put directly, there's more people in the world who care about the kernel
> and networking stack in FreeBSD alone (the simple building blocks for
> jail(8)), than the engineering core at Intel and IBM put together,
> (including some actual engineers at Intel and IBM).
Again, you probably have no idea how many people *understand* networking 
stack on BSD (under 100), and how many people work at IBM and Intel 
engineering (under a hundred thousand).

> So, all of that said, I'm a big fan of tying my systems into
> technologies which are supported by resources more vast than I can
> individually comprehend- and have experienced extreme pain when I tie
> myself into any solution whose developer base is proportionally smaller
> than the often large amounts of code they produce.
Nobody is forcing you to do anything. If jail fits your needs, use it. 

<snip>

> >> How does XEN help an administrator manage the complexity, and how
> >> complex is the actual virtualization mechanism itself?  (i.e. is it a
> >> 'large' software like VMWare and the like? [I'm asking in the context
> >> of the relatively few lines of kernel code that make up the whole of
> >> Jail(8)])
> > You'd be surprised how many lines of kernel code actually deal with
> > "root but not really superuser thanks to possibility of being in
> > jail". It is not a few.
> 
> Ok- for the whole of the *userland* components of Jail(8), yes.  It is 
> more than a few, lets get the facts down though:
> 
> The fundamental Jail(2) call, the actual kernel code which does
> everything here, is merely a small pointer which leverages chroot-
> (which exists as an elementary component in every decent unix I know
> of). With that, I'm not sure about the size of the source code for
> jailing, but I just scoped out Robert Watson's full from-scratch
> reimplementation of jail(2), which is under 2200 lines, available here:
> 
> http://www.watson.org/~robert/freebsd/jailng/
This patch does not contain the hooks that are sprinklered all over the 
source code to check for jail-ness. I submit that doing the checking in 
this fashion is unsafe and prone to error. Tomorrow, someone will add a 
syscall to (i dunno, format the harddrive) and check for root, forgetting 
to check for jail'd root.

> XEN is roughly an order of magnitude of between 150-200 times the code
> that goes into Jail(2) and jail(8), on quick glance- so I don't see why
> this is is really worth discussing- technically, Xen is much larger than
> jail(8) altogether.
Apples and oranges. Xen does much more, thus it is necessarily more 
complicated.

> You do know your stuff, and knowing about you I'd bet you've spent more
> time actually hacking the kernels here than I have, but watch the facts
> (and the sprit of the argument)...  Jail(2) and Jail(8) combined are
> much smaller than all the VM systems I've seen.
Of course, because they aren't VM systems. Entirely different things, 
made for entirely different task. I say again, if what you are attempting 
to do can be done with jail, you probably do not want Xen.

> > Again, apples and oranges. To virtualize a x86 processor takes lots of
> > code.
> 
> Indeed, agreed on apples vs. oranges.  But, again to re-iterate my
> position on these issues, I'm not meaning to trash-talk Xen, or UML, or
> Linux.  I am meaning to clarify expectations for use, and context-
> especially when dealing with services running on the most hostile and
> bustling network ever, the internet.
> 
> Xen, UML, and other virtulizations have other strengths in various
> contexts/cultures that Jail(8) does not.  For instance, the ability to
> run a full kernel, in the context of application and kernel development,
> is pretty insanely cool.  One out of many fundamentally different ideas.
> 
> It is not however necessary in the context of my managed virtualized
> servers, that my jails have a kernel- and actually is something I
> prefer, in the context of type of app/web development I do.  It's that
> simple- and there really isn't anything you've stated here Alex which
> constructively changes my preference- (though I'm open to change my mind
> if there's something that I'm missing here??)
When one of your users starts to continuously locally-ddos your machine,
you might change your mind on VMs.

Bottom line is: with jail, you still gotta trust your users not to fuck up
the environment for everyone else. With VMs, you don't have to trust your
users.

-alex