[nycbug-talk] Exploring pfSense (and an issue with States)

[Chris Buechler]

Tim Allender wrote:
> After reading the topic for next months meeting, I looked into monowall 
> and pfsense.
>   

Very cool!  first I've heard of that topic at the meeting. 

You're in luck, a core contributing member of both projects happens to 
hang out here, though I'm almost half way across the country from NYC 
(met Ike and others up at BSDCan the last couple years).  :) 

<snip>
> After 10005 states, it went to "Undefined", my shell froze (not 
> disconnected) but froze up as if the machine was hung.
> The http server stopped responding.
>
> All new connection attempts failed. No ping, nothing.
>
> I figured something like that'd happen. But, I wanted to see for myself 
> at an off time.
>   

Yep, once the max number is hit, no new connections will be accepted.  
That means you'll lose your web GUI access (with all the ajax, it can 
open several dozen states per webGUI session alone), though SSH, as long 
as you use keepalives in the client, should not drop (existing 
connection, existing state). 


> I figured that either the states will expire and everything will be ok 
> again. Or, I'll just go in a little early and reboot the box.
> Everything was fine and back to normal in the morning after the states 
> had expired.
>   

With most normal traffic, the application will close out the states 
itself, so you won't have to wait for the timeout.  Behavior of port 
scanners will vary. 

> So, my experience leaves me with some questions:
>
> 1. Max number of states:
> I can change the max number of states. But why is 10000 the default? and 
> what impact will raising it have?
>   

10,000 is the default because it's more than most networks will need, 
and is low enough to make running in 64 MB RAM feasible. 


> I figure this states table is stored in memory. What's a reasonable 
> maximum for 384 megs? 
The general rule of thumb with pf is ~1 KB RAM per state.  You could 
probably use 300 MB RAM for states alone (depending on what other 
features you use), so you could have 300K+ states.  


> These states have to be processed, though, so it's 
> a processing power limitation too, no?
>   

That wouldn't really be directly related to the number of states, 
though.  Packets per second, and throughput, are very CPU-dependent.  
It's unlikely you could push enough pps or Mbps through a 500 MHz box to 
exhaust a 100K state table (which in a typical network with mostly web 
and mail traffic would probably mean somewhere around a 100 Mb Internet 
pipe, but it varies widely, so it's impossible to accurately guess). 

It's fair to say states are only limited by RAM, but your ability to 
fill those states is limited by CPU and the quality of your NIC's. 


> If I raise it very high, and then under heavy load it runs out of 
> memory, what happens?
>   

On a full install, it'll just start swapping to disk, and you'll end up 
with the performance problems stemming from that.  It's highly unlikely 
you'd run out of RAM with 384 MB, unless there was a problem. 


> Will pfSense do the smart thing and start dropping the oldest inactive 
> states? 

It won't drop anything prematurely.  If you're out of RAM and swap (or 
don't have swap), processes will start dying because they're out of 
memory, and the whole system will turn into a mess pretty quickly.  The 
system itself should continue to work, but userland things like the 
webGUI, caching DNS server, etc. will die.  The system will never "fail 
open" though. 

The point isn't what happens when you run out of memory, it's avoiding 
that happening in the first place.  :)  With 384 MB RAM, you'll never 
see that happen. 


> 2: Time to expire / Peremptory clean up of states:
> Can I change the amount of time states remain in the table, maybe based 
> on state type, protocol type or other factors? and what impact would 
> that have?
> Is there a way to selectively drop states based on priority as the state 
> table approaches capacity?
>   

You can only change the state lifetime globally, but there are several 
state-related advanced options on the rules pages.  So you can set it up 
so, say, outbound HTTP is allowed no more than 10K states of a 30K state 
table, and SMTP is allowed 5K, etc. etc.  It's very flexible and 
powerful with all the advanced options, there are plenty of commercial 
enterprise class firewalls that can't do that. 


> 3. Hardware
> I like that I can do more with less. But, I'm looking at my options 
> here. If I have a choice, and it's reasonable, I'd rather have more than 
> less.
> Soekris is cool. But their top of the line boxes are only half of what 
> this super craptacular box is that I'm working with here.
>   

Yep - your only considerations, if looking at PC vs. embedded, would be 
power cost and consumption, heat dissipation, noise, and reliability.  
that 500 MHz box probably takes around 75 wt, while a Soekris or WRAP 
board will run at around 3-4 wt.  It's nowhere near enough of a 
difference cost-wise, even if running 24/7/365, to make up for the cost 
of the box.  If heat and noise are a concern, or high reliability (no 
moving parts on embedded, vs. an old PC that could die at any time) then 
I'd suggest looking at embedded systems. 

You mention Soekris, lately I've preferred PC Engines WRAP systems due 
to lower cost for essentially the same thing.  A WRAP is the same as a 
4801, minus the SFF IDE and PCI slot, but around the same price as a 
4501 (if not cheaper).  Netgate (www.netgate.com) is my preferred source 
in the US. 


> What about other barebones embedded architectures? I'm thinking, like, 
> Soekris only with PowerPC procs and memory sockets (as opposed to 
> soldered memory).
>   

eh, I'd stick with x86 personally.  Hacom has several options for mid 
range to higher end equipment, I have some of their hardware that 
they've donated for m0n0wall and pfsense testing purposes and it's been 
great.  http://www.hacom.net/

I also have one of these:
http://linitx.com/product_info.php?products_id=909

I got it after a couple other project members messed with it, so I'm not 
sure if it actually came over to the US from the UK, or where it came 
from (it was donated by LinITX).  I know you can get them in the US 
though.  I use it for my core router at home, routing several VLAN's on 
my home network.  (just because I can...)  :) 


> And, why for godsakes do these things never come with gigabit or fe 
> ports? 

The vast majority of them don't have the processor power to push 100 Mb, 
much less 1 Gb.  Through a Soekris 4501, you can get ~17 Mb with 
m0n0wall, ~12 Mb with pfsense (the difference entirely due to 
performance differences between FreeBSD 4.x and 6.x, stock OS 
installations perform identically).  A Soekris 4801 or WRAP will get you 
in the mid 40 Mb range on FreeBSD 4.x, in the low 30 Mb on 6.x.  Your 
500 MHz will probably get 50-75 Mb, depending on what kind of NIC's you 
have in it.  To push gig at wire speed, you need a ~2+ GHz or so, plus 
good NIC's and a bus sufficient to holding up to such abuse (i.e. PCI-X 
or PCI-e, not 32 bit PCI). 

> But, I'd like to break the LAN down into subnets and I'd need to route 
> them, at 1 gig+ speeds to the application servers if I can.
>   

The only really good way to do this is to use a L3 switch.  No firewall 
or router will ever be able to match the kind of performance a L3 switch 
will give you.  But I know there are people out there running pfsense on 
Dell PowerEdge 2850 dual Xeon 3.6 systems, new HP dual Xeons, etc. that 
route gig speeds.  That's far from a box you can slap together from 
spare parts though.  Or if it is, can I scavenge through your spare 
parts?  :) 

If you need wire speed gigabit performance, look at a new(er) 1U or 2U 
standard server, with onboard gig NIC's. 

hope that cleared up more questions than it raises.  :) 

cheers,
-Chris