[nycbug-talk] abusing a server

[Isaac Levy]

Hi Charles, All,

On May 27, 2005, at 12:32 AM, Charles Sprickman wrote:

> Hello all,
>
> I'm getting ready to turn loose my first FreeBSD 5.4 server, and not 
> having used 5.anything in production yet, I'm a bit nervous.

Yay!  I'm exited to say I'm following suit soon myself!


> I'd like to try and simulate some worst case scenarios inside one of 
> the jails.

For the record, I thought I'd toss in a quick checklist of issues to 
mitigate, off the top of my head:
(note- ikestyle vocabulary follows, please excuse it for what it is)

+ Lost Jail Attack

	problem: a jailed user changes the jails hostname, making it terribly
	difficult to use tools on the host to locate and manage the jail

	solution: host server sysctl.conf variable:
	#security.jail.set_hostname_allowed=0
	('man 8 jail' for more info)

+ Disk based resource attacks

	problem: jailed users fill disks

	solution: quotas or disk slices for jailed directory trees,
	managed from host server

+ Information Leak Attacks

	problem: jailed users get information from /dev which is useful
	in compromising the host system (or other jails), using tools like
	df and etc...

	solution: 2 parts, get your hands dirty with mount_devfs flags (on the 
host
	server to make certain a jail only gets to see devices it actually
	needs to have access to.  mount_devfs can also use macro files in
	FreeBSD 5.4- so this can be simple to manage once you set it up.

	Additionally, check out this host server sysctl.conf variable:
	#security.jail.getfsstatroot_only=1
	('man 8 jail' for more info)

	-
	problem: Jailed users *can* have access to more network interfaces than
	they actually need to, but this is not  the default.

	solution: host server sysctl.conf variable:
	#security.jail.socket_unixiproute_only=1
	('man 8 jail' for more info)

	-
	problem: Jailed user can have too much access to information-leaking 
syscalls,
	but this is not the default setup.

	solution: host server sysctl.conf variable:
	#security.jail.sysvipc_allowed=0
	('man 8 jail' for more info)


+ Various Network Attacks

	problem: Users *can* get into trouble if they are allowed to create raw
	sockets, in some contexts.  In other contexts, it may be important that
	users create raw sockets- (running custom network daemons, for 
example).

	solution: host server sysctl.conf variable:
	#security.jail.allow_raw_sockets=0
	If you really lock down what network daemons run, (just apache, for 
example),
	it may be wise to disallow raw sockets so someone can't for example 
run an
	irc daemon, or etc...
	('man 8 jail' for more info)


+ Jailed Restraint Attack

	problem: Users may be able to bypass restrictions on process and 
memory limits,
	(put in place to mitigate fork-bombs and memory hogs), by changing 
login.conf et. al.
	(they are after all root, within their jail).

	solution: Restrict jailed users from ever editing files in jail, by 
restricting
	jailed root from calling chflags- (somewhat like running a higher 
secure level
	from within the jail itself)- this way only root on the host server 
can edit or
	modify these files, absolutely.  Some files to lock down with 
immutable flags are
	a jail's sysctl.conf, login.conf, and perhaps some configs in /etc if 
relevant in
	your context- (i.e. resolv.conf, etc...)
	check out this host server sysctl.conf variable:
	#security.jail.chflags_allowed=0
	('man 8 jail' for more info)

	-
	problem: Jailed users regularly gone amok, or need to be run in 
maximum security
	environment.

	solution: this is NOT fun, and in many contexts impractical, but 
individual jails
	can run at a higher BSD secure level than the host system.
	Configure a JAILED sysctl.conf variable:
	#kern.securelevel
	('man 8 securelevel' for more info)
	Additionally, be sure to lock down sysctl.conf by setting the above 
mentioned
	#security.jail.chflags_allowed=0 variable in the HOST system.



> I'm still not certain exactly what all the services offered will be, 
> but at a minimum I'm looking at apache+php, various bits of blog 
> software and shell users that tend to not do much beyond read mail in 
> pine and edit their ~user homepages.

I'd think in that context, for your management sanity, you can really 
lock-down the jails if that's what the users are doing day to day.  
They don't need raw sockets, device access, special syscalls, or the 
ability for root user to use the chflags facility.  You could even try 
running higher securelevels in the jails and see if it affects anyone's 
services.


> Any ideas on how I can simulate around 50 logged-in users and a 
> worst-case flood of web traffic to a php-heavy blog?  I really want to 
> pummel this thing.  If it falls over, all the better as then I'll have 
> something to look at.  What are the weak spots?  The whole mess is 
> jailed, and I'm making heavy use of login.conf for the shell people so 
> that they can't fork-bomb or try to suck up more memory than they 
> should.  I think I'm most interested in doing some very long-running 
> tests...

As good real-time memory/process control is a holy-grail type effort 
for MANY operating systems, and OS's which impliment it often pay 
severely for it in other ways.  FreeBSD falls short with everyone else 
on this issue.  Jailed users *may* be able to consume more memory or 
cpu than their fair share.

A reliable way to mitigate this problem in reality, is to create some 
simple shell scripts to be run by cron, which do the following:

- Check processes on the host system
- nice/renice any processes hogging too much memory or CPU.
- Check for any processes which have been nice'd/renice'd and 
eventually un-nice/renice them.

In practice, this works terrificly, but the downside is that nobody has 
formalized any prefab tool to do this, to my knowledge.


> So far I'm just fiddling with some shell scripts to simulate users 
> futzing around and hitting apache with http_load.  So far I'm having 
> more success in breaking http_load than anything else.

/me bows out and hopes someone else comes to the table with some good 
load-testing systems here.

- Sidenote, insomuch as really good load testing comes when servers are 
being hit with packets from all across the internet, has anyone 
heard/thought of any groups of sysadmins who maintain 'test ddos' type 
networks?  I.E. I'd gladly put out a jail to a trusted network of 
people, for use in load testing scenarios- if I could likewise have 
safe/sane access to a couple of hundred other boxen for the same 
purpose... (or like a bit-torrent type idea for load testing? :)

Just thinking out loud on that one...  I guess to really do this right 
NYC*BUG would have to take over some bot-nets from the spackers- (not a 
fun idea- that'd be all-out war :).

Rocket-
.ike