[nycbug-talk] Exploring pfSense (and an issue with States)
nycbug at chrisbuechler.com
Tue Aug 15 18:54:28 EDT 2006
Tim Allender wrote:
> After reading the topic for next months meeting, I looked into monowall
> and pfsense.
Very cool! first I've heard of that topic at the meeting.
You're in luck, a core contributing member of both projects happens to
hang out here, though I'm almost half way across the country from NYC
(met Ike and others up at BSDCan the last couple years). :)
> After 10005 states, it went to "Undefined", my shell froze (not
> disconnected) but froze up as if the machine was hung.
> The http server stopped responding.
> All new connection attempts failed. No ping, nothing.
> I figured something like that'd happen. But, I wanted to see for myself
> at an off time.
Yep, once the max number is hit, no new connections will be accepted.
That means you'll lose your web GUI access (with all the ajax, it can
open several dozen states per webGUI session alone), though SSH, as long
as you use keepalives in the client, should not drop (existing
connection, existing state).
> I figured that either the states will expire and everything will be ok
> again. Or, I'll just go in a little early and reboot the box.
> Everything was fine and back to normal in the morning after the states
> had expired.
With most normal traffic, the application will close out the states
itself, so you won't have to wait for the timeout. Behavior of port
scanners will vary.
> So, my experience leaves me with some questions:
> 1. Max number of states:
> I can change the max number of states. But why is 10000 the default? and
> what impact will raising it have?
10,000 is the default because it's more than most networks will need,
and is low enough to make running in 64 MB RAM feasible.
> I figure this states table is stored in memory. What's a reasonable
> maximum for 384 megs?
The general rule of thumb with pf is ~1 KB RAM per state. You could
probably use 300 MB RAM for states alone (depending on what other
features you use), so you could have 300K+ states.
> These states have to be processed, though, so it's
> a processing power limitation too, no?
That wouldn't really be directly related to the number of states,
though. Packets per second, and throughput, are very CPU-dependent.
It's unlikely you could push enough pps or Mbps through a 500 MHz box to
exhaust a 100K state table (which in a typical network with mostly web
and mail traffic would probably mean somewhere around a 100 Mb Internet
pipe, but it varies widely, so it's impossible to accurately guess).
It's fair to say states are only limited by RAM, but your ability to
fill those states is limited by CPU and the quality of your NIC's.
> If I raise it very high, and then under heavy load it runs out of
> memory, what happens?
On a full install, it'll just start swapping to disk, and you'll end up
with the performance problems stemming from that. It's highly unlikely
you'd run out of RAM with 384 MB, unless there was a problem.
> Will pfSense do the smart thing and start dropping the oldest inactive
It won't drop anything prematurely. If you're out of RAM and swap (or
don't have swap), processes will start dying because they're out of
memory, and the whole system will turn into a mess pretty quickly. The
system itself should continue to work, but userland things like the
webGUI, caching DNS server, etc. will die. The system will never "fail
The point isn't what happens when you run out of memory, it's avoiding
that happening in the first place. :) With 384 MB RAM, you'll never
see that happen.
> 2: Time to expire / Peremptory clean up of states:
> Can I change the amount of time states remain in the table, maybe based
> on state type, protocol type or other factors? and what impact would
> that have?
> Is there a way to selectively drop states based on priority as the state
> table approaches capacity?
You can only change the state lifetime globally, but there are several
state-related advanced options on the rules pages. So you can set it up
so, say, outbound HTTP is allowed no more than 10K states of a 30K state
table, and SMTP is allowed 5K, etc. etc. It's very flexible and
powerful with all the advanced options, there are plenty of commercial
enterprise class firewalls that can't do that.
> 3. Hardware
> I like that I can do more with less. But, I'm looking at my options
> here. If I have a choice, and it's reasonable, I'd rather have more than
> Soekris is cool. But their top of the line boxes are only half of what
> this super craptacular box is that I'm working with here.
Yep - your only considerations, if looking at PC vs. embedded, would be
power cost and consumption, heat dissipation, noise, and reliability.
that 500 MHz box probably takes around 75 wt, while a Soekris or WRAP
board will run at around 3-4 wt. It's nowhere near enough of a
difference cost-wise, even if running 24/7/365, to make up for the cost
of the box. If heat and noise are a concern, or high reliability (no
moving parts on embedded, vs. an old PC that could die at any time) then
I'd suggest looking at embedded systems.
You mention Soekris, lately I've preferred PC Engines WRAP systems due
to lower cost for essentially the same thing. A WRAP is the same as a
4801, minus the SFF IDE and PCI slot, but around the same price as a
4501 (if not cheaper). Netgate (www.netgate.com) is my preferred source
in the US.
> What about other barebones embedded architectures? I'm thinking, like,
> Soekris only with PowerPC procs and memory sockets (as opposed to
> soldered memory).
eh, I'd stick with x86 personally. Hacom has several options for mid
range to higher end equipment, I have some of their hardware that
they've donated for m0n0wall and pfsense testing purposes and it's been
I also have one of these:
I got it after a couple other project members messed with it, so I'm not
sure if it actually came over to the US from the UK, or where it came
from (it was donated by LinITX). I know you can get them in the US
though. I use it for my core router at home, routing several VLAN's on
my home network. (just because I can...) :)
> And, why for godsakes do these things never come with gigabit or fe
The vast majority of them don't have the processor power to push 100 Mb,
much less 1 Gb. Through a Soekris 4501, you can get ~17 Mb with
m0n0wall, ~12 Mb with pfsense (the difference entirely due to
performance differences between FreeBSD 4.x and 6.x, stock OS
installations perform identically). A Soekris 4801 or WRAP will get you
in the mid 40 Mb range on FreeBSD 4.x, in the low 30 Mb on 6.x. Your
500 MHz will probably get 50-75 Mb, depending on what kind of NIC's you
have in it. To push gig at wire speed, you need a ~2+ GHz or so, plus
good NIC's and a bus sufficient to holding up to such abuse (i.e. PCI-X
or PCI-e, not 32 bit PCI).
> But, I'd like to break the LAN down into subnets and I'd need to route
> them, at 1 gig+ speeds to the application servers if I can.
The only really good way to do this is to use a L3 switch. No firewall
or router will ever be able to match the kind of performance a L3 switch
will give you. But I know there are people out there running pfsense on
Dell PowerEdge 2850 dual Xeon 3.6 systems, new HP dual Xeons, etc. that
route gig speeds. That's far from a box you can slap together from
spare parts though. Or if it is, can I scavenge through your spare
If you need wire speed gigabit performance, look at a new(er) 1U or 2U
standard server, with onboard gig NIC's.
hope that cleared up more questions than it raises. :)
More information about the talk