Re: Load balancing in routers

[Alex Rubenstein <alex () nac net>]

A few comments:


> I don't think flow-caching is necessarily due to CEF.

CEF, afaik, is unaware of flows.


> Even on dinky 2500 & 2600 series where you don't run CEF,

Many people run CEF on 2600's, it's about the only way to get to the
cisco-advertised PPS on the box.

> load balancing over multiple links uses a flow-hashed method. If you
> want per-packet load distribution you have to specifically enable it by
> saying "no ip route-cache" on each interface.

That is very deadly, please, don't anyone actually try that.

CEF load balancing, IIRC, had two options, specifyable on a per-interface
basis -- 'per-packet', and 'per-destination'. Both have obvious meanings.

Newer IOS's seem to have a defaulting mechanism available in global config
mode, but being a weirdo, I don't trust it. I still specify on the
per-interface.

We use this in several scenerios, specifically for load-balancing T1's,
and it amazingly works well, with the links often being in balance to the
tune of 1 to 3%. I've seen similar performance at DS3 rates.


> Paul's statement about CEF is interesting. It's probably the first public
> statement I've ever heard where someone was praising CEF. Usually
> discussions about CEF are accompanied by liberal amounts of swearing...

I dunno; except for some silliness in 12.1(8a)E[1-4] on a MSFC2, we've
seen general goodness from CEF from 2600, 3600, 4700, 5300, 7200, 7500.

Then again, we're not UU or Sprint, and don't have the traffic loading
they do.


> Joe
>
>
> On 4/8/02 9:03 AM, "Iljitsch van Beijnum" <iljitsch () muada com> wrote:
>
> > On 8 Apr 2002, Paul Vixie wrote:
> >
> > > > I seem to remember fast switching was per-destination, and CEF was
> > > > round robin. But it seems CEF is now per-destination as well in IOS 12.2.
> > > > Round robin is optional.
> >
> > > CEF is flow-hashed, and the hash seems to include both source and
> > > destination, and seems to include the port numbers.  This is by observing
> > > the behaviour of flows hitting various members of the F.ROOT-SERVERS.NET
> > > set, each of whom sends F's address to several upstream routers using OSPF.
> > > CEF works like a charm -- the load is never split by more than 45-55 and
> > > that's damn good for wire speed hashing in my view.
> >
> > > We used CEF in 11.x and it behaved the same way.  It was never round-robin
> > > in any way we could observe.
> >
> > You're right. I was thinking of process switching.
> >
> > According to:
> > http://www.ils.unc.edu/dempsey/186s00/reorderingpaper.pdf
> >
> > packet reordering at MAE East was extremely common a few years ago. Does
> > anyone have information whether this is still happening?

-- Alex Rubenstein, AR97, K2AHR, alex () nac net, latency, Al Reuben --
--    Net Access Corporation, 800-NET-ME-36, http://www.nac.net   --