nanog mailing list archives

Re: Famous operational issues


From: sronan () ronan-online com
Date: Mon, 22 Feb 2021 19:58:29 -0500

Let me tell you about my personal favorite.

It’s 2002 and I am working as an engineer for an electronic stock trading platform (ECN), this platform happened to be 
the biggest platform for trading stocks electronically, on some days bigger than NASDAQ itself. This platform also 
happened to be run on DOS, FoxPro and a Novell file share, on a cluster of roughly 1,000 computers, two of which were 
the “engine” that matched all of the trades.

Well FoxPro has this “feature” where the ESC key halts the running program. We had the ability to remote control these 
DOS/FoxPro machines via some program we had written. Someone asked me to check the status of the process running on the 
primary matching engine, and when I was done, out of habit, I hit ESC. Trade processing grinds to a halt (phone calls 
have to be made to the SEC). I immediately called the NOC and told them it was me. Next thing I know, someone from the 
NOC is at my desk with a screwdriver putting the ESC key from my keyboard. I remained ESC keyless for the next several 
years until I left the company. I was hazed pretty good over it, but was essentially given a one time pass.



On Feb 22, 2021, at 7:30 PM, bzs () theworld com wrote:


At Boston Univ we discovered the hard way that a security guard's
walkie-talkie could cause a $5,000 (or $10K for the big machine room)
Halon dump.

Took a couple of times before we figured out the connection tho once
someone made it to the hold button before it actually dumped.

Speaking of halon one very hot day I'm goofing off drinking coffee at
a nearby sub shop when the owner tells me someone from the computing
center was on the phone, that never happened before.

Some poor operator was holding the halon shot, it's a deadman's switch
(well, button) and the building was doing its 110db thing could I come
help? The building is being evac'd.

So my boss who wasn't the sharpest knife in the drawer follows me down
as I enter and I'm sweating like a pig with a floor panel sucker
trying to figure out which zone tripped.

And he shouts at me over the alarms: WHY TF DOES IT DO THIS?! Angrily.

I answered: well, maybe THERE'S A FIRE!!!

At which point I notice the back of my shoulder is really bothering
me, which I say to him, and he says hmmm there's a big bee on your
back maybe he's stinging you?

Fun day.

-- 
       -Barry Shein

Software Tool & Die    | bzs () TheWorld com             | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD       | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*


Current thread: