FreeBSD 10Gbs Networking Working

[Isaac (.ike) Levy]

Hi All,

Just a quick excited post to share:

Using commodity hardware and 10Gbit Intel nics, and FreeBSD 10.0, I'm 
pushing an actual 9.1-9.3Gbps (netperf tests), across 4x nics, no 
tuning.

--
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.0.1 () 
port 0 AF_INET : histogram : interval : dirty data : demo
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

  65536  32768  32768    10.01    9388.17
--

Not a joke.  I've been running these in loops simultaneously on all 4 
ports.  Game changed, mind blown.

Best,
.ike



------------
more details

The OS is stock FreeBSD 10, dmesg posted here:
http://nycbug.org/?action=dmesgd&do=view&dmesgid=2527
That HW costs about 3200 bucks.  (Ponder that.)

The only tuning I did was in bios- to disable hyperthreading (pci 
interrupts), and disable cpu virtualization.

The two pci cards each have 2x SFP slots, with fiber SFP's.  The 1u 
chassis puts them both on a X16 pcie3 bus, (each becomming X8 pci).

The driver is ix,
http://downloadmirror.intel.com/14688/eng/README.txt
I think the man page for it is ixgbe(4), (but I may be wrong here?)
http://www.freebsd.org/cgi/man.cgi?query=ixgbe&apropos=0&sektion=0&manpath=FreeBSD+10.0-RELEASE&arch=default&format=html

General Notes:
Apparently I'm CPU bound here, but using somewhat "typical" CPU's here, 
could increase the number cores and see what happens next.  It's 
delightful to finally have a problem we can throw CPU at in the world 
again :)

--
Additionally, I'm running tests using pfSense, (2.1.1-prerelease has 
the driver fixes to support these cards.)
Since pfSense is based on FreeBSD 8.3, the performance is not obviously 
as stunning- but out the gate, with minimal fuss, I'm able to reliably 
push 4+Gbps across 2 ports (4Gbps each direction for total 8Gbps 
throughput), and 2+Gbps each direction when saturating all 4 ports 
(total 8Gbps throughput again).

CPU is pegged, so I'm pretty certain that dropping in more cores will 
help, as well as more tuning.
We've got some bigger boxes in the works, (8x total 10Gbe ports, 4x 
GigE ports on the motherboard, and a dedicated PCI slot for each card), 
so we'll be dropping more CPU cores in the box here too...  We'll see 
what improvements this brings.

While I'm personally looking forward to a bright shiny 10.x based 
pfSense in the future, I'm likewise impressed with the overall 
stability and responsiveness with this hardware.

--
Some urls on tuning:
I don't know much of this project, but these tuning urls have been 
fascinating,
   http://bsdrp.net/documentation/technical_docs/performance
Other interesting 10Gbps nics,
   
http://www.freebsd.org/cgi/man.cgi?query=cxgb&sektion=4&apropos=0&manpath=FreeBSD+10.0-RELEASE
   
http://www.freebsd.org/cgi/man.cgi?query=cxgbe&sektion=4&apropos=0&manpath=FreeBSD+10.0-RELEASE
Yeah yeah yeah Calomel, great notes on PCI calculations:
   https://calomel.org/network_performance.html