Re: Outage (planned and unplanned) notification

["Alex.Bligh" <amb () xara net>]

> The corporate establishment generally frowns at amateurish attempts to
> do "damage control" public relations.  That's the job for P.R. people,
> not engineers, you see?  Now, the only question is where to find P.R.
> people who can tell cidr from cider.

It's not only peers, it's customers as well. I take full BGP from 3 people.
I normally do my best to ensure when we have a routing related problem
we get some qualified personnel to ring the provider concerned.

Attached is an example phone call (provider = P, customer = C). I've
hidden the identities concerned as I'm sure it wasn't actually the fault
of the personnel concerned, and composited 2 calls. But this seems to
be becoming the norm.

Why "yes we have a major problem, we're looking at it, and we'll keep
you up to date by email and by phone if necessary" doesn't work is
beyond me. Then I can switch them off and switch them back on when
they've fixed it.

Alex Bligh
Xara Networks

--

P: Network Operations, X speaking, how can I help?

C: We seem to be having problems with backtraffic to us through your network.
   Can you check you see us right at (say) MAE-East?

P: Um, no, I'm not near a terminal at the moment.

C: Ur, could you move to a terminal, or transfer me to someone who is
   near a terminal please?

P: All the terminals are busy at the moment.

C: Oh well, could you tell me if you are running BGP Dampening on your
   MAE-East router?

P: Well I think we run BGP at MAE-East.

(so on for five minutes until transfered)

P: Y speaking, how can I help?

C: We seem to be having problems with backtraffic to us through your network.
   Can you check you see us right at (say) MAE-East?

P: Your BGP session has been up for a while.

C: OK, well [NSP] (for instance) can't reach us. Traceroutes star out
   when it hits an [NSP] router.

P: I can ping your router.

C: Yes, and I can get to your NOC fine. But I can't get to [NSP].

P: We can get to [NSP] fine. It must be a problem in your network.

C: Can you check you are sending [NSP] are routes? They claim they aren't
   hearing them from you.

P: There are no problems between us and [NSP].

< repeat for 10 minutes >

C: Are you sure you have no other network problems at the moment?

P: Not really.

C: None? [NSP] said you might have had some.

P: Well we do have a few problems at the moment. But everything is still
   working.

C: Can you tell me what the problems are?

P: They don't affect you.

< repeat until bored>

Eventually it transpires an East/West fibre cut prevented all East coast
customers of C reaching all West coast [NSP] destinations.

... sigh...

- - - - - - - - - - - - - - - - -