Re: Famous operational issues

[Rogier van Eeten via NANOG <nanog () nanog org>]

Ahh, war stories. I like the one where I got a wake up call that our IRC 
server was on fire,  together with the rest of the DC.


Not that widespread, but we reached Slashdot. :)

November 2002, University of Twente, The Netherlands. Some idiot wanted 
to be a hero. He deflated peoples tires, to help inflate them. One 
morning he thought it would be a good idea to start a small fire and 
then extinguish it, so he would be the hero that stopped a fire. He 
failed and the building burned down. He got caught a few days later when 
he tried the same thing in a different building.

Almost all of the IT was in that building, including core network, 
uplinks to SURFNet (Dutch Educational Network) and to the 2000 students 
living on the campus. Ironically a new DC was already being built, so 
that was ready for use a few weeks later.

As we had quite a network for 2002 we hosted for instance 
security.debian.org. The students all had 100Mbit in their room, so some 
of them also hosted some popular websites. One I can remember was an 
image sharing site.

Some students immediately created a backup network; dhcp server, dns 
server with a catch all, website explaining what was going on, IRC 
server, etc..

A local ISP offered to sponsor 50Mbit for the residents, which was 
connected via a microwave relay and a temporary fiber was run through a 
ditch to connect two parts of the campus residencies. At the end of the 
day all 2000 students had their internet connection back, although all 
behind a single 50Mbit link.


Syslog message from the local SURFNet router:

lo0.ar5.enschede1.surf.net 3613: Nov 20 07:20:50.927 UTC: 
%ENV_MON-2-TEMP: Hotpoint temp sensor(slot 18) temperature has reached 
WARNING level at 61(C)


(Disclaimer: Where I say we, I mean we as University. I wasn't working 
for the university, but was part of the students working on the backup 
network. There are probably some other people on list with some more 
details and I've probably missed some details, but this is the summary.)


On 16-02-2021 23:08, Jared Mauch wrote:
> I was thinking about how we need a war stories nanog track. My favorite was being on call when the router was stolen.
>
> Sent from my TI-99/4a
>
> > On Feb 16, 2021, at 2:40 PM, John Kristoff <jtk () dataplane org> wrote:
> >
> > Friends,
> >
> > I'd like to start a thread about the most famous and widespread Internet
> > operational issues, outages or implementation incompatibilities you
> > have seen.
> >
> > Which examples would make up your top three?
> >
> > To get things started, I'd suggest the AS 7007 event is perhaps  the
> > most notorious and likely to top many lists including mine.  So if
> > that is one for you I'm asking for just two more.
> >
> > I'm particularly interested in this as the first step in developing a
> > future NANOG session.  I'd be particularly interested in any issues
> > that also identify key individuals that might still be around and
> > interested in participating in a retrospective.  I already have someone
> > that is willing to talk about AS 7007, which shouldn't be hard to guess
> > who.
> >
> > Thanks in advance for your suggestions,
> >
> > John