[nycbug-talk] Exploring pfSense (and an issue with States)

[Tim Allender]

After reading the topic for next months meeting, I looked into monowall 
and pfsense.
We had a little linksys router facilitating our DSL connection and I've 
been thinking of switching that out since I came to this company.
So, after rolling out a batch of computers I took one of the old ones 
and, last Friday installed pfSense on it and switched out the linksys.
(PIII 550 MHz, 384 Mgs)
I'm very impressed.
I re-setup the VPNs we are using.
Switched it over to the "Advanced Outgoing NAT" to get NAT working for 
our Jersey network as well.
(Not cool, I know, but they get their internet over our T1. Hey, that's 
the way I found it =)

So.... last night, from home, I figured I'd just take a peek at whats 
all up and open on the office network in the middle of the night.
I have a VPN connection to my house, so I just nmap the 10/24 network.
And, I'm watching the pfSense interface the whole time.
# of States starts going up.
And up.
And up.
Around 8000ish states I'm thinking, "hmm, I wonder what happens when we 
exceed the maximum."
(Oh yea, baby. Load test time!)
And I figure, "hey, worst case scenario, I reboot and restore the back 
up, no problem"

After 10005 states, it went to "Undefined", my shell froze (not 
disconnected) but froze up as if the machine was hung.
The http server stopped responding.

All new connection attempts failed. No ping, nothing.

I figured something like that'd happen. But, I wanted to see for myself 
at an off time.
I figured that either the states will expire and everything will be ok 
again. Or, I'll just go in a little early and reboot the box.
Everything was fine and back to normal in the morning after the states 
had expired.

So, my experience leaves me with some questions:

1. Max number of states:
I can change the max number of states. But why is 10000 the default? and 
what impact will raising it have?
I figure this states table is stored in memory. What's a reasonable 
maximum for 384 megs? These states have to be processed, though, so it's 
a processing power limitation too, no?
If I raise it very high, and then under heavy load it runs out of 
memory, what happens?
Will pfSense do the smart thing and start dropping the oldest inactive 
states? ~Is~ that the smart thing? And, I guess either it isn't or it 
won't otherwise it probably would have done it in this case.

2: Time to expire / Peremptory clean up of states:
Can I change the amount of time states remain in the table, maybe based 
on state type, protocol type or other factors? and what impact would 
that have?
Is there a way to selectively drop states based on priority as the state 
table approaches capacity?

3. Hardware
I like that I can do more with less. But, I'm looking at my options 
here. If I have a choice, and it's reasonable, I'd rather have more than 
less.
Soekris is cool. But their top of the line boxes are only half of what 
this super craptacular box is that I'm working with here.
What about other barebones embedded architectures? I'm thinking, like, 
Soekris only with PowerPC procs and memory sockets (as opposed to 
soldered memory).
And, why for godsakes do these things never come with gigabit or fe 
ports? Yes, I realize that for WAN routing 100mbps is 10x more than 
adequet.
But, I'd like to break the LAN down into subnets and I'd need to route 
them, at 1 gig+ speeds to the application servers if I can.
When we say "FreeBSD runs on ppc architecture," what boxes are we 
talking about? Who sells it? Are we talking about embedded boxes?
What other architectures (that fbsd runs on) are popular for embedded 
systems?