Re: Rapidly-variable routing on the time scale of seconds to minutes?

[Daniel Roesen <dr () cluenet de>]

On Mon, Jan 31, 2005 at 04:20:31AM -0500, Charles Shen wrote:
> We did a "traceroute" end-to-end routing measurement in 2004 and found about
> 5-10% of measuremnts exhibiting rapidly-variable routing on the time scale
> of a single traceroute (seconds to minutes). In other words, the packets
> belonging to a single traceroute took multiple paths. 
[...]
> Route change in such a short scale for packets in the same flow could be
> troublesome. But the occurrence of such behavior does not seem to have
> reduced over the past years at least from our measurements. Does anyone know
> how to explain this behavior? Thanks!

Yes, this is normal per-flow load balancing on parallel backbone lines.
Usually, "flows" are defined via a hash on L3 and possibly L4
addressing information (IP source/dest, and TCP/UDP port source/dest,
ICMP code, etc.). If the "flow hash" contains L4 information, every
traceroute probe packet is considered a different "flow" and you see
exactly that:

>  5  pos5-0-2488M.cr2.FRA2.gblx.net (67.17.65.53)  2.448 ms
> pos6-0-2488M.cr1.FRA2.gblx.net (67.17.65.77)  2.368 ms  2.281 ms
>  6  so3-0-0-2488M.ar2.FRA3.gblx.net (67.17.65.82)  2.676 ms  2.750 ms
> so2-0-0-2488M.ar2.FRA3.gblx.net (67.17.65.58)  2.569 ms

You would NOT see the same effect with packets of e.g. the same TCP
session, so this (multipath forwarding) is usually no problem (as for
TCP and UDP applications there is no reordering happening). So your
analysis results (traceroute) are misleading for most real-life
applications. I agree that it's irritating and I personally favor
using aggregated SONET/Ethernet devices (IEEE 801.3ad) to bundle
parallel lines if possible.


Best regards,
Daniel

-- 
CLUE-RIPE -- Jabber: dr () cluenet de -- dr@IRCnet -- PGP: 0xA85C8AA0