Re: scaling linux-based router hardware recommendations

[Eduardo Meyer <dudu.meyer () gmail com>]

> - 1x ServerU Netmap L800 box in Bridge Mode for Core Firewall protection
> - 2x ServerU Netmap L800 boxes as BGP router (redundant)
> - Several Netmap L800, L100 and iXSystems servers (iXS for everything else
> since ServerU are only networking-centric, not high storage high processing
> Xeon servers)
>
> In this setup I am running yet another not well known but very promising
> technology, called Netmap.
>
> A Netmap firewall (called netmap-ipfw) was supplied from ServerU vendor,
> it's a slightly modified version from what you can download from Luigi
> Rizzo's (netmap author) public repository with multithread capabilities
> based on the number of queues available in the ServerU igb(4) networking
> card.
>
> What it does is, IMHO, amazing for a x86 hardware: line rate firewall on
> 1GbE port (1.3-1.4Mpps) and line rate firewall for 10GbE port (12-14Mpps)
> in a system with 8 @2.4Ghz Intel Rangeley CPU.
>
> It's not Linux DNA. It's not PF_RING. It's not Intel DPDK.
>
> It's netmap, it's there, available, on FreeBSD base system with a number of
> utilities and code for reference on Rizzos' repositories. It's there, it's
> available and it's amazing.
>
> This firewall has saved my sleep several times since November, dropping up
> to 9Mpps amplified UDP/NTP traffic on peak DDoS attack rates.
>
> For the BGP box, I needed trunking, Q-in-Q and vlan. And sadly right now
> this is not available in a netmap implementation.
>
> It means I had to keep my BGP router in the kernel path. It's funny to say
> this, but Netmap usually skips kernel path completely and does its job
> direct on the NIC, reaching backplane and bus limits directly.
>
> ServerU people recommended me to use Chelsio Terminator 5 40G ports. OK I
> only needed 10G but they convinced me not to look at the bits per second
> numbers but the packets per seconds number.
>
> Honestly, I don't know how Chelsio T5 did it, even though ServerU 1GbE
> ports perform very good on interruption CPU usage (probably this is an
> Intel igb(4) / ix(4) credit) but everything I route from one 40GbE port to
> the other port on the same L-800 expansion card, I have very, very, very
> LOW interrupt rates. Sometimes I have no interrupt at all!!
>
> I peaked routing 6Mpps on ServerU L-800 and still had CPU there,
I am also a user for FreeBSD netmap-ipfw, running kipfw fwd to, say, "fwd"
http traffic to a peerapp appliance. My numbers are not line rate, I peak
on 900Kpps, but still have CPU idle.

I had a hard time figuring out how to use netmap-ipfw, due to lack of
updated documentation, but once I got it running and set up, ecerything was
very straightforward with default code, no modifications, just as available.

I agree FreeBSD-netmap seems more ready, with tools, toolchains and code
available wheh compared to DPDK or Linux DNA. Also in the hope for further
evolvings of Netmap in the base system.

Numbers are impressive indeed.




-- 
===========
Eduardo Meyer
pessoal: dudu.meyer () gmail com
profissional: ddm.farmaciap () saude gov br