nanog mailing list archives
Re: Spain was offline
From: Peter Corlett <abuse () cabal org uk>
Date: Thu, 31 Aug 2006 17:30:37 +0100
On 31 Aug 2006, at 16:30, Joseph Jackson wrote:
I wish the article had more info since I have been wondering how a software upgrade downed the entire zone.
Oh, loads of ways.
Wasn't there any backup servers?
Well, a quick poke suggests, assuming a reasonably traditional setup, that ns1.nic.es is the master, and there are various slaves, not necessarily directly under their control. ns1.nic.es appears to be running BIND 9.3.2, and there's other versions running on the other nameservers. So if it *was* a software update of BIND, it's probably not global.
OTOH, I can believe that somebody broke a Perl script critical to it and it rolled out a valid, but empty, zonefile which the secondaries faithfully replicated. Not that I've watched cascading DNS failures at too many places with bits of crufty Perl, oh no...
Actually, it amazes me that this sort of thing doesn't happen more often.
Did they not test the upgrade before hand? I know I'd lose my job if I upgraded our dns servers all at once with out testing.
It's Europe, it's harder to fire people. There's probably a bit of scapegoating and shooting of messengers going on, but it's quite likely that the root cause is a general process failure that's not attributable to a single individual.
Current thread:
- Spain was offline Gunther Stammwitz (Aug 30)
- Re: Spain was offline billn (Aug 30)
- Re: Spain was offline Sean Donelan (Aug 30)
- Re: Spain was offline Michael . Dillon (Aug 31)
- Re: Spain was offline Joe Abley (Aug 31)
- Re: Spain was offline Michael . Dillon (Aug 31)
- Re: Spain was offline Florian Weimer (Aug 31)
- Re: Spain was offline Crist Clark (Aug 31)
- Re: Spain was offline Michael . Dillon (Aug 31)
- <Possible follow-ups>
- Re: Spain was offline Fergie (Aug 30)
- RE: Spain was offline Joseph Jackson (Aug 31)
- Re: Spain was offline Peter Corlett (Aug 31)
- Re: Spain was offline Valdis . Kletnieks (Aug 31)
- Re: Spain was offline Steven M. Bellovin (Aug 31)
- Re: Spain was offline Peter Corlett (Aug 31)