RE: Rapidly-variable routing on the time scale of seconds to minutes?

["Charles Shen" <charles () cs columbia edu>]

Please see inline.

> -----Original Message-----
> From: John Fraizer [mailto:nanog () enterzone net] 
> Sent: Monday, January 31, 2005 8:21 AM
> To: Charles Shen; nanog () merit edu
> Subject: Re: Rapidly-variable routing on the time scale of 
> seconds to minutes?
>
>
> Charles Shen wrote:
> > An example traceroute record containing the fluttering is 
> shown below 
> > (see the 5th hop)
> >
> >  Fri Apr 09 09:35:35 2004
> >
> >  1  cisfhfb.fh-friedberg.de (212.201.24.1)  1.095 ms  0.402 
> ms  0.321 
> > ms  2  ar-frankfurt2.g-win.dfn.de (188.1.42.9)  120.105 ms  
> 198.766 ms  
> > 200.040 ms  3  cr-frankfurt1-ge5-0.g-win.dfn.de 
> (188.1.80.1)  2.093 ms  
> > 2.142 ms  2.087 ms
> >  4  so-6-0-0.ar2.FRA2.gblx.net (208.48.23.141)  2.461 ms  
> 2.349 ms  2.333 ms
> >  5  pos5-0-2488M.cr2.FRA2.gblx.net (67.17.65.53)  2.448 ms
> > pos6-0-2488M.cr1.FRA2.gblx.net (67.17.65.77)  2.368 ms  2.281 ms
>
> That sure looks like ECM to me.  Equal Cost Multi-Path.  This is NOT 
> anything new.  What's the big deal?

> From the responses, the answer to "the rapidly-variable routing on the time
scale of seconds to minutes" seems to be: 

1. It could be link layer load balancing, with the two interfaces belonging
to the same router.
2. It could be per-flow load balancing where flows are defined via both L3
and L4 info, so traceroute probe could not reflect the truth. 

My question is then: would it be safe to argue that the above two causes
explain all (or most of?) the observed "fluttering" routers? (some examples
listed below)  What we are concerned about is per-packet load balancing
(packets in the same flow go through different paths), which will cause
trouble to protocols that install state information in routers along the
flow path.

Example pairs: 

144.223.27.146  sl-telia1-1-0.sprintlink.net
144.232.230.30  sl-telia1-4-0.sprintlink.net

216.140.0.66  s3-0-0.a1.hywr.broadwing.net
216.140.0.70  s4-0-0.a1.hywr.broadwing.net

67.17.65.53  pos5-0-2488M.cr2.FRA2.gblx.net
67.17.65.77  pos6-0-2488M.cr1.FRA2.gblx.net 

67.17.65.54 so5-0-0-2488M.ar2.FRA2.gblx.net     
67.17.65.78 so4-0-0-2488M.ar2.FRA2.gblx.net

67.17.65.57 pos11-0-2488M.cr2.FRA2.gblx.net
67.17.65.81 pos11-0-2488M.cr1.FRA2.gblx.net

67.17.65.58 so2-0-0-2488M.ar2.FRA3.gblx.net
67.17.65.82 so3-0-0-2488M.ar2.FRA3.gblx.net 

67.17.64.66 pos6-0-2488M.cr2.SFO1.gblx.net
67.17.74.157 pos8-0-2488M.cr1.SFO1.gblx.net

129.250.2.183 p16-3-0-0.r01.snjsca04.us.bb.verio.net
129.250.5.136 p16-7-0-0.r00.snjsca04.us.bb.verio.net