RE: Linux router network cards

["Keith Medcalf" <kmedcalf () dessus com>]

And do not use an Intel CPU.

Intel only has 4x PCIe lanes that are shared out into whatever configuration they claim to have and are totally 
unsuitable for use in a computer that actually has to be able to do high-speed I/O.

--
Be decisive.  Make a decision, right or wrong.  The road of life is paved with flat squirrels who could not make a 
decision.

> -----Original Message-----
> From: NANOG <nanog-bounces+kmedcalf=dessus.com () nanog org> On Behalf Of
> Eric Kuhnke
> Sent: Saturday, 24 October, 2020 06:22
> To: Jared Geiger <jared () compuwizz net>
> Cc: NANOG <nanog () nanog org>
> Subject: Re: Linux router network cards
>
> In addition to Jared's advice, I would recommend calculating PCI-Express
> bandwidth bus points for whatever platform one is using.
>
>
> For instance using the Intel X710-DA4, which could be capable in a
> maximal scenario of 80Gbps of traffic, ensure it's in at least a PCI-E
> 3.0 x4 slot. And calculate the total number of PCI-E 3.0 x1 (or PCI-E 4.0
> if a very new system) lanes that exist and are connected to the CPU. Big
> difference between some options for Ryzen and Threadripper vs Intel CPUs,
> towards the lower end of the cost range.
>
>
> With recent Linux kernels if you have an Intel 510 or 710 series two or
> four port card in a slot that can't support its full capability, you'll
> get a warning in dmesg at boot time.
>
>
>
> On Thu, Oct 22, 2020 at 9:30 PM Jared Geiger <jared () compuwizz net
> <mailto:jared () compuwizz net> > wrote:
>
>
>       I use DANOS with Intel XL710 10G NICs in DPDK mode for linux based
> routing.
>
>       If you're doing routing protocols, allocate 2 CPU cores to the
> control plane and then a CPU core per 10G/1G interface for the dataplane,
> plus an extra core for good measure. So for a 4 x 10G router taking in
> full routes, 2 cores for control plane, 5 cores for the dataplane. Those
> cores should be Intel Xeon E5-2600v3/4 or newer and faster the clocks,
> the better.
>
>       Similar CPU core allocations if you choose TNSR.
>
>       On Thu, Oct 22, 2020 at 3:21 PM Jean St-Laurent via NANOG
> <nanog () nanog org <mailto:nanog () nanog org> > wrote:
>
>
>               Chelsio cards are probably what you are looking for.
>
>               https://www.chelsio.com/terminator-6-asic/
>
>               It's closer to an asic than a traditional nic as the
> router/firewall rules
>               are pushed directly into the hardware.
>
>               I don't know how good they are with linux and they seem to be
> compatible.
>               https://www.chelsio.com/linux/
>
>               You will need to mess around a bit and fiddle here and there.
> If you don't
>               mind using FreeBSD instead of linux, you could achieve a
> smoother and more
>               integrated experience.
>
>               Jean
>
>               -----Original Message-----
>               From: NANOG <nanog-bounces+jean=ddostest.me () nanog org
> <mailto:ddostest.me () nanog org> > On Behalf Of micah
>               anderson
>               Sent: Thursday, October 22, 2020 5:31 PM
>               To: Philip Loenneker <Philip.Loenneker () tasmanet com au
> <mailto:Philip.Loenneker () tasmanet com au> >; NANOG
>               <nanog () nanog org <mailto:nanog () nanog org> >
>               Subject: RE: Linux router network cards
>
>
>               Thanks for the reply.
>
>               Philip Loenneker <Philip.Loenneker () tasmanet com au
> <mailto:Philip.Loenneker () tasmanet com au> > writes:
>               > Take a look at the Mellanox ConnectX 5 series of cards. They
> handle
>               > DPDK, PVRDMA (basically SR-IOV that allows live migration
> between
>               > hosts), and can even process packets within the NIC for some
>
>               From what I can tell, SR-IOV/PVRDMA aren't really useful for
> me in building
>               a router that wont be doing any virtualization.
>
>               If the card can do DPDK, can it do XDP?
>
>               > The slidedeck for the presentation is here:
>               > https://www.ausnog.net/sites/default/files/ausnog-
> 2019/presentations/1
>               > .9_Rhod_Brown_AusNOG2019.pdf
>               >
>               > It's heavily targeting virtualised workloads but some of the
> feature sets
>               apply to bare-metal uses too.
>
>               Yeah, this wont be a virtualized environment, just a router
> passing packets,
>               dropping them, handling bgp and collecting flows.
>
>               --
>                       micah