nanog mailing list archives
Re: Famous operational issues
From: sronan () ronan-online com
Date: Mon, 22 Feb 2021 19:58:29 -0500
Let me tell you about my personal favorite. It’s 2002 and I am working as an engineer for an electronic stock trading platform (ECN), this platform happened to be the biggest platform for trading stocks electronically, on some days bigger than NASDAQ itself. This platform also happened to be run on DOS, FoxPro and a Novell file share, on a cluster of roughly 1,000 computers, two of which were the “engine” that matched all of the trades. Well FoxPro has this “feature” where the ESC key halts the running program. We had the ability to remote control these DOS/FoxPro machines via some program we had written. Someone asked me to check the status of the process running on the primary matching engine, and when I was done, out of habit, I hit ESC. Trade processing grinds to a halt (phone calls have to be made to the SEC). I immediately called the NOC and told them it was me. Next thing I know, someone from the NOC is at my desk with a screwdriver putting the ESC key from my keyboard. I remained ESC keyless for the next several years until I left the company. I was hazed pretty good over it, but was essentially given a one time pass.
On Feb 22, 2021, at 7:30 PM, bzs () theworld com wrote: At Boston Univ we discovered the hard way that a security guard's walkie-talkie could cause a $5,000 (or $10K for the big machine room) Halon dump. Took a couple of times before we figured out the connection tho once someone made it to the hold button before it actually dumped. Speaking of halon one very hot day I'm goofing off drinking coffee at a nearby sub shop when the owner tells me someone from the computing center was on the phone, that never happened before. Some poor operator was holding the halon shot, it's a deadman's switch (well, button) and the building was doing its 110db thing could I come help? The building is being evac'd. So my boss who wasn't the sharpest knife in the drawer follows me down as I enter and I'm sweating like a pig with a floor panel sucker trying to figure out which zone tripped. And he shouts at me over the alarms: WHY TF DOES IT DO THIS?! Angrily. I answered: well, maybe THERE'S A FIRE!!! At which point I notice the back of my shoulder is really bothering me, which I say to him, and he says hmmm there's a big bee on your back maybe he's stinging you? Fun day. -- -Barry Shein Software Tool & Die | bzs () TheWorld com | http://www.TheWorld.com Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD The World: Since 1989 | A Public Information Utility | *oo*
Current thread:
- Re: Famous operational issues, (continued)
- Re: Famous operational issues Jared Mauch (Feb 18)
- Re: Famous operational issues Warren Kumari (Feb 18)
- Re: Famous operational issues Karl Auer (Feb 18)
- Re: Famous operational issues Patrick W. Gilmore (Feb 18)
- Re: Famous operational issues George Herbert (Feb 18)
- Re: Famous operational issues Henry Yen (Feb 20)
- Re: Famous operational issues Alain Hebert (Feb 22)
- Re: Famous operational issues Bruce H McIntosh (Feb 22)
- RE: Famous operational issues Tony Wicks (Feb 22)
- Re: Famous operational issues bzs (Feb 22)
- Re: Famous operational issues sronan (Feb 22)
- Re: Famous operational issues Warren Kumari (Feb 23)
- Re: Famous operational issues Tony Finch (Feb 22)
- Re: Famous operational issues Christopher Morrow (Feb 22)
- Re: Famous operational issues Paul Ebersman (Feb 18)
- Re: Famous operational issues Randy Bush (Feb 18)
- Re: Famous operational issues Andy Ringsmuth (Feb 18)
- Re: Famous operational issues Jethro R Binks (Feb 22)
- Re: Famous operational issues Mark Tinka (Feb 18)
- Re: Famous operational issues Suresh Ramasubramanian (Feb 18)
- Re: Famous operational issues Sabri Berisha (Feb 19)