Re: Famous operational issues

[Erik Sundberg <ESundberg () nitelusa com>]

Worked a cronic support call where their internet would bounce at noon every workday. The Cisco 1601 or 1700 Router 
that had there T1 in, ended up being on top a microwave. Weeks of troubleshooting and shipping new routers on this one.

Also had another one where the router was plugged in to an outlet that was controlled by a light switch, discovered 
this after shipping them two new routers.

Customer had there building remodeled and the techs counldn't find the T1 Smartjack for the building. The contract who 
did the remodel job, decided it would be a good idea to cut out the section of wall where the telco equipment was and 
mounted it to the ceiling. It's new location was in the ladys bathroom, above the drop ceiling mounted to the 
building's rafters 10' in the air.

Customer needed a new router, because the first one died. It was a machine shop and they mounted the router to the wall 
next to a lathe or drill press that used oil to cool the bit while it was cutting. It looked like some dumped the 
router in a bucket of oil when we got it back.

Arriving at another large colo for a buildout. Only to find that our ASR9K that arrived 2 weeks ago was stored outside 
on the load dock which has no roof or locked gate. I guess that why Cisco put the plastic bag over the chassis when 
there shipped.

Colo techs at another larger colo decided to unpack our router which was a fully loaded 1/2 rack chassis. Since they 
couldn't lift it, they tipped the router on the side and walked it back by shifting the weight from one corner of the 
chassis to another. Bending the chassis. I could see the scrap marks in the floor from it.

We had colo space in top floor of an ATT CO where we put a Cisco 7513 to terminate about a dozen CHDS3's. The roof was 
leaking and instead of fixing the roof. The fix was to put a sheet of plastic over our cabinet. It was more like a tent 
over the cabinet.  A pool of water formed in a diviot at the top and it was 120+ degrees under the plastic tarp.

Our office was in a work loft off an older building and they had the AC unit mounted to the ceiling with a drip pan 
underneath them. Well, AC on the 2nd floor had the pump for the drip pan died. Who every installed the drip pan didn't 
secure it or center it under the AC unit. It filled up with water and since it was not secured and was off centered. 
The drip pan came crashing down with a few gallons of water. The water worked it's way over to the wall and traveled 
down one story in the building. The floor below had all the telco equipment mounted to that same wall and the water 
flowed down right through a couple of ATT's Ciena mounted to the wall shorting them out. I was at the Chicago Nanog 
Hackathon on Sunday and was called out to work that one 😕

Was working in the back of a cabinet that had -48 VDC power for a Cisco Router, a screw fell and shorted out the power. 
My co worker who was standing in front of the rack wasn't happy because the ADC PowerWorx Fuse panel was about 6" from 
his face where he was working. It had those little black alarm fuses, that had the spring-loaded arm. When it tripped a 
nice shower of sparks had flew right at his face Luckly he wore glasses.

I was 18 at my first IT job and it was a brand-new building. I was plugging in a 208VAC 30A APC UPS in the server room 
the electrican had just energized and check the circuit. I plugged in the APC UPS and gave it a good turn for the twist 
lock plug to catch and KA BAMB!!! Sparks came shooting out of the outlet at me. I think I pooped myself that day. Turns 
out the electricians deiced that a single Gange electrical box was good enough for a 208 VAC 30A outlet, that barely 
fit in the box. Didn't put any tape around the wire terminals. When they energized the circuit there was enough of an 
air gap that the hot screw didn't ground out. When I gave it that good old twist while plugging in the APC, I grounded 
the hot screw to the side of the electrical box.






________________________________
From: NANOG <nanog-bounces+esundberg=nitelusa.com () nanog org> on behalf of Seth Mattinen <sethm () rollernet us>
Sent: Thursday, February 18, 2021 10:23 AM
To: nanog () nanog org <nanog () nanog org>
Subject: Re: Famous operational issues

On 2/18/21 1:07 AM, Eric Kuhnke wrote:
> On that note, I'd be very interested in hearing stories of actual
> incidents that are the cause of why cardboard boxes are banned in many
> facilities, due to loose particulate matter getting into the air and
> setting off very sensitive fire detection systems.


I had a customer that tried to stack their servers - no rails except the
bottom most one - using 2x4's between each server. Up until then I
hadn't imagined anyone would want to fill their cabinet with wood, so I
made a rule to ban wood and anything tangentially related (cardboard,
paper, plastic, etc.). Easier to just ban all things. Fire reasons too
but mainly I thought a cabinet full of wood was too stupid to allow.

The "no wood" rule has become a fun story to tell everyone who asks how
that ended up being a rule. The wood customer turned out to be a
complete a-hole anyway, wood was just the tip of the iceberg.

________________________________

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it 
may contain confidential information that is legally privileged. If you are not the intended recipient, or a person 
responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, 
distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If 
you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must 
destroy the original transmission and its attachments without reading or saving in any manner.
Thank you.