nanog mailing list archives

Re: qwest outage no notice


From: Mike <mike-nanog () tiedyenetworks com>
Date: Thu, 07 Jan 2010 09:04:13 -0800


Yeah, they refused to notify due to security concerns from what they told me last night. Notification was performed after maintenance was complete.


Ok, so the next question is, what harm would a simple advance notice of 'emergency maintenance' caused, vs the very real hassle and inconvenience that DS3-down in the middle of the night caused for operations staff? I personally was yanked out of bed by my network monitoring systems and had to spend at least 5 minutes looking at logs and perf monitors and so forth so I had enough information before making the call to qwest support, thinking we took a hit on our fiber. Some others reported getting 7 or more hours of advance notice - this would have been enough for me, even tho of course I don't like it - and I suspect it could have saved many many others from a similar fate of responding to the event so late.

Just saying "we need to do this", does not give the bad guys any ammo with which to attack, and would go a long ways twords preventing needless heroics such as middle of the night investigations by senior staff. The follow up email we subsequently received was fine and we appreciated learning it really was a necessary upgrade and seems justified now with that knowledge, I would simply have appreciated not having to engage my emergency processes for something that was planned.

Mike-



Current thread: