Re: impossible circuit

[Justin Shore <justin () justinshore com>]

This is just a WAG but what the hell.

Jon Lewis wrote:
> I've got this private line DS3.  It connects cisco 7206 routers in 
> Orlando (at our data center) and in Ocala (a colo rack in the Embarq CO).
>
> According to the DLR, it's a real circuit, various portions of it ride 
> varying sized OC circuits, and then it's handed off to us at each end 
> the usual way (copper/coax) and plugged into PA-2T3 cards.

Are you sure that they are not crossing some channels in the middle and 
accidentally handing them to a different customer?  You mention above 
that various portions of the DS3 ride different transport circuits in 
the middle.  That always creates the potential for someone to not put it 
back together correctly on either end.  I've seen DLCs get crossed 
before.  I could easily see a transport provider crossing portions of a 
circuit, especially if they break it into pieces in the middle and have 
to put it back together on the ends.

I think it makes sense too.  Somebody's getting traffic off a T1 that 
isn't destined for them.  Their router sees it, says WTF and sends a 
ICMP dest unreachable via their default route through Sprint.  Same 
thing goes for a traceroute; it simply follows its default route to 
reply to your packets with the expiring TTL.  Taking a path through a 
different provider would be expected since it doesn't have a connected 
route to the source of the traceroute (since it's not the far end of 
your T1 that you're expecting).  The site getting your crossed T1 could 
be using the T1 as a PtP to a branch office and has Internet through a 
different circuit that hasn't been hosed.

I would be curious to hear if Sprint is having any problems with a 
circuit connected to sl-bb20-dc-6-0-0.sprintlink.net, what the router is 
and if any directly connected customers are having T1 problems.  If 
nothing else Sprint should be able to track down the source of the 
traceroute return packets and contact the customer.  The T1 could be 
part of a bundle at their site and they may not even realize that the 
bundle dropped a path.

> Last Tuesday, at about 2:30PM, "something bad happened."  We saw a 
> serious jump in traffic to Ocala, and in particular we noticed one 
> customer's connection (a group of load sharing T1s) was just totally 
> full.  We quickly assumed it was a DDoS aimed at that customer, but 
> looking at the traffic, we couldn't pinpoint anything that wasn't 
> expected flows.

Are you sure that the traffic being received by each of the T1s is 
their's?  Do you have any way to getting flows or packets off of 
individual T1s and not the bundle as a whole?

Tracing through you to your upstream...

>  7  andc-br-3-f2-0.atlantic.net (209.208.9.138)  47.951 ms  56.096 ms  
> 56.154 ms
>  8  ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98)  56.199 ms  56.320 
> ms  56.196 ms
>  9  * * *

Circuit gets crossed onto the wrong customer.  Wrong site received a 
packet with an expiring TTL and goes to send a reply.  Destination IP 
isn't on a connected route so the site sends the reply via it's default 
route on Sprint.

> 10  sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174)  80.774 ms  81.030 
> ms  81.821 ms
> 11  sl-st20-ash-10-0.sprintlink.net (144.232.20.152)  75.731 ms  75.902 
> ms  77.128 ms

Reply traverses Sprint to L3 and on to you.

> 12  te-10-1-0.edge2.Washington4.level3.net (4.68.63.209)  46.548 ms  
> 53.200 ms  45.736 ms
> 13  vlan69.csw1.Washington1.Level3.net (4.68.17.62)  42.918 ms 
> vlan79.csw2.Washington1.Level3.net (4.68.17.126)  55.438 ms 
> vlan69.csw1.Washington1.Level3.net (4.68.17.62)  42.693 ms
> 14  ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137)  48.935 ms 
> ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129)  49.317 ms 
> ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141)  48.865 ms
> 15  ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85)  59.642 ms  56.278 ms  
> 56.671 ms
> 16  ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2)  47.401 ms  62.980 
> ms  62.640 ms
> 17  ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149)  40.300 ms  40.101 
> ms  42.690 ms
> 18  ae-6-6.car1.Orlando1.Level3.net (4.69.133.77)  40.959 ms  40.963 ms  
> 41.016 ms
> 19  unknown.Level3.net (63.209.98.66)  246.744 ms  240.826 ms  239.758 ms
> 20  andc-br-3-f2-0.atlantic.net (209.208.9.138)  39.725 ms  37.751 ms  
> 42.262 ms
> 21  ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98)  43.524 ms  45.844 
> ms  43.392 ms
> 22  * * *
> 23  sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174)  63.752 ms  61.648 
> ms  60.839 ms
> 24  sl-st20-ash-10-0.sprintlink.net (144.232.20.152)  66.923 ms  65.258 
> ms  70.609 ms
> 25  te-10-1-0.edge2.Washington4.level3.net (4.68.63.209)  67.106 ms  
> 93.415 ms  73.932 ms
> 26  vlan99.csw4.Washington1.Level3.net (4.68.17.254)  88.919 ms  75.306 
> ms vlan79.csw2.Washington1.Level3.net (4.68.17.126)  75.048 ms
> 27  ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129)  69.508 ms  
> 68.401 ms ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133)  79.128 ms
> 28  ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85)  64.048 ms  67.764 ms  
> 67.704 ms
> 29  ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18)  68.372 ms  67.025 
> ms  68.162 ms
> 30  ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149)  65.112 ms  65.584 
> ms  65.525 ms

I can't explain the continuous loop or the dupes.  I'm not sure if my 
theory fits those symptoms or not.

> Our circuit provider's support people have basically just maintained 
> that this behavior isn't possible and so there's nothing they can do 
> about it. i.e. that the problem has to be something other than the circuit.

Can you have them put the circuit into maintenance and have them test it 
end to end?  They can't deny it when their TDR says that there's a problem.

> I got tired of talking to their brick wall, so I contacted Sprint and 
> was able to confirm with them that the traffic in question really was 
> inexplicably appearing on their network...and not terribly close 
> geographically to the Orlando/Ocala areas.

Which supports with my theory of a crossed circuit.  Crossing a DS1 onto 
the wrong DS3 or OCx could easily make it pop up anywhere.  Somewhere is 
another site that's having T1 problems.

> So, I have a circuit that's bleeding duplicate packets onto an unrelated 
> IP network, a circuit provider who's got their head in the sand and 
> keeps telling me "this can't happen, we can't help you", and customers 
> who were getting tired of receiving all their packets in triplicate (or 
> more) saturating their connections and confusing their applications.  
> After a while, I had to give up on finding the problem and focus on just 
> making it stop.  After trying a couple of things, the solution I found 
> was to change the encapsulation we use at each end of the DS3.  I 
> haven't gotten confirmation of this from Sprint, but I assume they're 
> now seeing massive input errors one the one or more circuits where our 
> packets were/are appearing.  The important thing (for me) is that this 
> makes the packets invalid to Sprint's routers and so it keeps them from 
> forwarding the packets to us.  Cisco TAC finally got back to us the day 
> after I "fixed" the circuit...but since it was obviously not a problem 
> with our cisco gear, I haven't pursued it with them.

Right.  By changing the encap you've basically killed the circuit.  With 
that T1 effectively down on your end you won't be sending any packets 
down the problem path and aren't able to see that problem anymore with 
your traceroutes.  However your customer with the bundle of T1s is down 
a circuit.

It makes sense in my mind that it's simply a crossed circuit in the 
middle.  Your transport provider for whatever reason pulled out a DS1 
and sent it down a different path.  They accidentally crossed DS1s in 
the middle and are handing your DS1 to a Sprint customer and their DS1 
to your customer.  That's my theory at least.

Justin