[nycbug-talk] pf/freebsd, route-to, reply-to and nat

[Max Gribov]

Hello all,
I am trying to figure this out for a couple of days..
I have a fbsd 6.1 router connected to local network, to DSL ISP and a
Cable ISP.

All user traffic goes out via the Cable line, the default route on the
box is the Cable.
There is a windows server behind the firewall, and firewall's DSL IP
address has a port forward for 3389/tcp (rdp) to the windows box.
Im able to pipe users' traffic via the cable, but no matter what i do, i
cannot get the windows server on the internal network to be accessible
from the DSL ip.
I can reach internet, I can see both cable and DSL routers, and if I
change my default gateway to the DSL, then it works fine.


[root at styx /home/max]# uname -a
FreeBSD styx.neuropunks.org 6.1-RELEASE FreeBSD 6.1-RELEASE #1: Mon Jun
12 19:44:57 EDT 2006    
max at styx.neuropunks.org:/usr/src/sys/sparc64/compile/STYX  sparc64

Here are the relevant rule parts
(the order of the rules below is actual order in the pf.conf)

int="hme0"
ext="hme1"
ext_cable="hme5"
gw_dsl="216.254.70.1"
gw_cable="207.38.217.1"
draco="192.168.0.4"

# nat
nat on $ext_cable  from $local_net to any -> ($ext_cable)
nat on $ext from $local_net to any -> ($ext)

# rdr
rdr inet proto tcp from any to $styx_ext/32 port 3389 -> $draco port 3389

# default deny
block log-all all
pass quick on lo0 all

# ensures that we can pass to draco's 192.168.x.x ip address
pass in log on $ext inet proto tcp from any to $draco/32 port 3389 flags
S/SA modulate state queue (prirdp, tcpack)

# pass tcp to DSL public IP to port 3389, reply through DSL interface/IP
pass in quick log on $ext reply-to ($ext $gw_dsl) inet proto tcp from
any to $styx_ext/32 port 3389 flags S/SA modulate state queue (prirdp,
tcpack)

# local interface filtering
pass out on $int from any to $local_net
pass in quick on $int from $local_net to $int

# pass into local interface with source of 192.168.x.x
pass in log on $int route-to ($ext $gw_dsl) proto tcp from $draco/32
port 3389 to any keep state queue (intprirdp, inttcpack)

# global allow all outgoing
pass out on $ext_cable inet proto tcp from any to any flags S/SA
modulate state
pass out on $ext_cable inet proto { udp, icmp } from any to any keep state
pass out on $ext inet proto tcp from any to any flags S/SA modulate state
pass out on $ext inet proto { udp, icmp } from any to any keep state

# keep track of the interfaces/sources
pass out on $ext route-to ($ext_cable $gw_cable) from $ext_cable to any
pass out on $ext_cable route-to ($ext $gw_dsl) from $ext to any
# EOF

Here is tcpdump from watching pflog0 for relevant log statements:
19:27:50.405748 rule 12/0(match): pass in on hme1:
finn.neuropunks.org.64868 > draco.rdp: S 2150035332:2150035332(0) win
65535 <mss 1460,nop,wscale 1,[|tcp]>
        0x0000:  4520 003c d29a 4000 3b06 3c2c 451f 2b0a
        0x0010:  c0a8 0004 fd64 0d3d 8026 ef84 0000 0000
        0x0020:  a002 ffff 5f15 0000 0204 05b4 0103 0301
19:27:50.405910 rule 67/0(match): pass out on hme0:
finn.neuropunks.org.64868 > draco.rdp: S 2150035332:2150035332(0) win
65535 <mss 1460,nop,wscale 1,[|tcp]>
        0x0000:  4520 003c d29a 4000 3a06 3d2c 451f 2b0a
        0x0010:  c0a8 0004 fd64 0d3d 8026 ef84 0000 0000
        0x0020:  a002 ffff 5f15 0000 0204 05b4 0103 0301


The packets are not being filtered, the global block policy logs denies.
I looked at plain interface tcpdump (hme0, hme1) and my router does
address packets to local DSL router MAC address,
and I am able to ssh into the firewall itself, which is handled by this
rule:

pass in quick log on $ext reply-to ($ext $gw_dsl) inet proto tcp from
any to $styx_ext/32 port 22 flags S/SA modulate state (max-src-conn-rate
8/60, overload <spammers> flush global) queue (prissh, tcpack)

so i know i can get packets back over the dsl interface even if the
static route is the cable.
There seems to be some issue with either nat'ing, or i am not using
reply-to/route-to rules, but ive tried everything, and i cant figure it out.

If anyone has any idea, or did something similar, please let me know

Thank you,

Max