RE: Level3 worldwide emergency upgrade?

["Siegel, David" <David.Siegel () Level3 com>]

Hi Ray,

This topic reminds me of yesterday's discussion in the conference around getting some BCOP's drafted.  it would be 
useful to confirm my own view of the BCOP around communicating security issues.  My understanding for the best practice 
is to limit knowledge distribution of security related problems both before and after the patches are deployed.  You 
limit knowledge before the patch is deployed to prevent yourself from being exploited, but you also limit knowledge 
afterwards in order to limit potential damage to others (customers, competitors...the Internet at large).  You also do 
not want to announce that you will be deploying a security patch until you have a fix in hand and know when you will 
deploy it (typically, next available maintenance window unless the cat is out of the bag and danger is real and 
imminent).

As a service provider, you should stay on top of security alerts from your vendors so that you can make your own 
decision about what action is required.  I would not recommend relying on service provider maintenance bulletins or 
public operations mailing lists for obtaining this type of information.  There is some information that can cause more 
harm than good if it is distributed in the wrong way and information relating to security vulnerabilities definitely 
falls into that category.

Dave

-----Original Message-----
From: Ray Wong [mailto:rayw () rayw net] 
Sent: Wednesday, February 06, 2013 9:16 AM
To: nanog () nanog org
Subject: Re: Level3 worldwide emergency upgrade?

>

OK, having had that first cup of coffee, I can say perhaps the main reason I was wondering is I've gotten used to 
Level3 always being on top of things (and admittedly, rarely communicating). They've reached the top by often being a 
black box of reliability, so it's (perhaps
unrealistically) surprising to see them caught by surprise. Anything that pushes them into scramble mode causes me to 
lose a little sleep anyway. The alternative to what they did seems likely for at least a few providers who'll NOT 
manage to fix things in time, so I may well be looking at longer outages from other providers, and need to issue 
guidance to others on what to do if/when other links go down for periods long enough that all the cost-bounding 
monitoring alarms start to scream even louder.

I was also grumpy at myself for having not noticed advance communication, which I still don't seem to have, though 
since I outsourced my email to bigG, I've noticed I'm more likely to miss things. Perhaps giving up maintaining that 
massive set of procmail rules has cost me a bit more edge.

Related, of course, just because you design/run your network to tolerate some issues doesn't mean you can also budget 
to be in support contract as well. :) Knowing more about the exploit/fix might mean trying to find a way to get free 
upgrades to some kit to prevent more localized attacks to other types of gear, as well, though in this case it's all 
about Juniper PR839412 then, so vendor specific, it seems?

There are probably more reasons to wish for more info, too. There's still more of them (exploiters/attackers) than 
there are those of us trying to keep things running smoothly and transparently, so anything that smells of "OMG new 
exploit found!" also triggers my desire to share information. The network bad guys share information far more quickly 
and effectively than we do, it often seems.

-R>