RE: problems with uunet/worldcom/whoever-they-bought-this-week

[rdobbins () netmore net]

I just spent 80 hours over four days trying to get a DS3 to uu.net fixed -
turns out the circuit goes from their POP to WorldCom, from WorldCom to MFS,
and -then- to PacBell.

Only problem is, they didn't tell me that, and so we started troubleshooting
from the far side of the PacBell end, thinking that it was PacBell all the
way through, and only finding out after 2 days what the toplogy really
looked like.  Of course, they'd just assumed that there was no way that 
-their- portion of the circuit could have problems.

PacBell even replaced a module in a DACS 3, thinking -that- was the problem.

So it was a bad card or something in an MFS OC-48 mux . . . fine, MFS
replaced it.  Then they started doing intrusive testing on some OC-3 bundles
they were having problems with, of which bundles my DS3 was apparently
caught up in, so the circuit kept pogoing for another 8 hours.  Since uu.net
bought WorldCom and MFS, you'd think they'd have better coordination, no?

All that, combined with the fact that MFS and WorldCom apparently have zero
remote diags capability and will only dispatch techs when you threaten to
sue them (literally), made for a great weekend.

-----------------------------------------------------------
Roland Dobbins <rdobbins () netmore net> // 818.535.5024 voice

-----Original Message-----
From: Jim Mercer [mailto:jim () reptiles org]
Sent: Monday, September 18, 2000 10:53 PM
To: nanog () merit edu
Subject: problems with uunet/worldcom/whoever-they-bought-this-week




ok, this is getting ludicrous.
[ cut to the chase, if you are a non-US uunet/worldcom/mci customer,
  you can skip my diatribe and jump down to the bottom ]

this is the third monday night when we have had a major BGP/connectivity
outage with uunet.ca.

two mondays ago, (ie 3 weeks), we had an 11 hour combined outage of either
connectivity and/or BGP routing.  this was topped off by them restoring
the config from an old backup, including a slew of old static routes.

last monday we had a 4 hour outage, similar issues.

tonight, we got:

18:30 me -> uunet.ca: so, what's the problem
      uunet.ca:       not sure, US-NOC is looking into it

19:15 me -> uunet.ca: any ETA?
      uunet.ca:       problem solved, although the CPU on the router is 100%
                      due to BGP recalc.  should be OK soon.

21:30 me -> uunet.ca: so, what's the problem?
      uunet.ca:       bad ATM card, been replaced, back up, although the CPU
                      is 100% due to BGP recalc.  should be OK soon.

01:30 me -> uunet.ca: so, what's the problem?
      uunet.ca:       US-NOC is saving the configs customer-by-customer and
                      will be rebooting the router RSN.  no ETA.

i'm not sure, but a lot of major problems seem to have cropped up since
uunet.ca dumped their local technicians and went dependent on the US-NOC.

are other non-US uunet customers getting the same treatment?

-- 
[ Jim Mercer                 jim () reptiles org              +1 416 410-5633 ]
[          Reptilian Research -- Longer Life through Colder Blood          ]
[  Don't be fooled by cheap Finnish imitations; BSD is the One True Code.  ]