Re: scaling linux-based router hardware recommendations

[Jim Shankland <nanog () shankland org>]

On 1/26/15 11:33 PM, Pavel Odintsov wrote:
> Hello!
>
> Looks like somebody want to build Linux soft router!) Nice idea for
> routing 10-30 GBps. I route about 5+ Gbps in Xeon E5-2620v2 with 4
> 10GE cards Intel 82599 and Debian Wheezy 3.2 (but it's really terrible
> kernel, everyone should use modern kernels since 3.16 because "buggy
> linux route cache"). My current processor load on server is about:
> 15%, thus I can route about 15 GE on my Linux server.
I looked into the promise and limits of this approach pretty intensively 
a few years back before abandoning the effort abruptly due to other 
constraints. Underscoring what others have said: it's all about pps, not 
aggregate throughput. Modern NICs can inject packets at line rate into 
the kernel, and distribute them across per-processor queues, etc. 
Payloads end up getting DMA-ed from NIC to RAM to NIC. There's really no 
reason you shouldn't be able to push 80 Gb/s of traffic, or more, 
through these boxes. As for routing protocol performance (BGP 
convergence time, ability to handle  multiple full tables, etc.): that's 
just CPU and RAM.

The part that's hard (as in "can't be fixed without rethinking this 
approach") is the per-packet routing overhead: the cost of reading the 
packet header, looking up the destination in the routing table, 
decrementing the TTL, and enqueueing the packet on the correct outbound 
interface. At the time, I was able to convince myself that being able to 
do this in 4 us, average, in the Linux kernel, was within reach. That's 
not really very much time: you start asking things like "will the entire 
routing table fit into the L2 cache?"

4 us to "think about" each packet comes out to 250Kpps per processor; 
with 24 processors, it's 6Mpps (assuming zero concurrency/locking 
overhead, which might be a little bit of an ... assumption). With 
1500-byte packets, 6Mpps is 72 Gb/s of throughput -- not too shabby. But 
with 40-byte packets, it's less than 2 Gb/s. Which means that your Xeon 
ES-2620v2 will not cope well with a DDoS of 40-byte packets. That's not 
necessarily a reason not to use this approach, depending on your 
situation; but it's something to be aware of.

I ended up convincing myself that OpenFlow was the right general idea: 
marry fast, dumb, and cheap switching hardware with fast, smart, and 
cheap generic CPU for the complicated stuff.

My expertise, such as it ever was, is a bit stale at this point, and my 
figures might be a little off. But I think the general principle 
applies: think about the minimum number of x86 instructions, and the 
minimum number of main memory accesses, to inspect a packet header, do a 
routing table lookup, and enqueue the packet on an outbound interface. I 
can't see that ever getting reduced to the point where a generic server 
can handle 40-byte packets at line rate (for that matter, "line rate" is 
increasing a lot faster than "speed of generic server" these days).

Jim