Re: Headscratcher of the week

[Brett Frankenberger <rbf+nanog () panix com>]

On Fri, May 31, 2013 at 03:25:22PM -0700, Mike wrote:
> Gang,
>
>       In the interest of sharing 'the weird stuff' which makes the job of
> being an operator ... uh, fun? is that the right word?..., I would
> like to present the following two smokeping latency/packetloss
> plots, which are by far the weirdest I have ever seen.
>
>       These plots are from our smokeping host out to a customer location.
> The customer is connected via DSL and they run PPPoE over it to
> connect with our access concentrator. There is about 5 physical
> insfastructure hops between the host and customer; The switch, the
> BRAS, the Switch again, and then directly to the DSLAM and then
> customer on the end.
>
>
> The 10 day plot:
> http://picpaste.com/10_Day_graph-YV3IdvRV.png
>
> The 30 hour plot:
> http://picpaste.com/30_hour_graph-DrwzfhYJ.png
>
>       How can you possibly have consistent increase in latency like that?
> I'd love to hear theories (or offers of beer, your choice!).

Theory:

There's a stateful device (firewall, NAT, something else) in the path
that is creating state for every ICMP Echo Request it forwards and
(possibly) searching that state when forwarding the ICMP Echo Reply
responses, and never destroying that state, and either the create
operation or the search operation (or both) takes an amount of time
that is a linear function of the number of state entries.

     -- Brett