nanog mailing list archives

best current practice: buffers


From: Baldur Norddahl <baldur.norddahl () gmail com>
Date: Sat, 19 Dec 2020 15:53:39 +0100

Hello

What is the best current practice for buffer size? For customer facing
ports, core network ports and transit links?

We have a buffer problem, discovered by a customer that moved their servers
to a cloud service some distance away. That resulted in a drastic reduced
transfer speed between their office and the cloud service. Nothing much
could be done since we, like so many others, have switches with extreme
fast port speeds (48x 10G, 4x 100G) with a tiny shared 12 MB buffer.

Now the time has come to upgrade that hardware to something that does have
plenty of buffer capacity, so I am planning out what the settings should be.

I have read this paper
http://web.stanford.edu/class/cs244/papers/sizing-router-buffers-redux.pdf
which claims not much buffer is needed at all. And I think they are
completely wrong. In the paper they assume we are trying to get as much
throughput on a congested core or transit port. But we always make sure
those are not congested, so that misses the mark completely. For core ports
we are concerned about microbursts and for customer ports we do care about
a single TCP session being able to get the max throughput. The paper
assumes there will be a lot of TCP sessions sharing the bandwidth, but that
is not always the case with customer ports. It might be a guy downloading
an ISO image with him being the only guy at the office.

The common wisdom is to set the buffer size to one bandwidth-delay product.
And also that bigger buffers than this is harmful. But that raises the
question of what distance do I tune for? Amsterdam is 10 ms away. East
coast USA is 100 ms.

Also the new hardware (Juniper ACX710) does support more than one queue per
port. Would it be possible to have that ISO download go into a queue for
heavy streams and allow smaller streams to skip the line, so we do not see
the downside of a heavy buffer (buffer bloat)? It is no longer just a
simple matter of buffer size.

What are others doing to deliver the best possible performance to customers
with regards to buffering?

Regards,

Baldur

Current thread: