Re: POE switches and lightning

[Daniel Senie <dts () senie com>]

While the equipment may well be affected by an EM pulse, if the gear returns to normal after a power cycle, then the 
equipment vendor didn't do their job fully developing the product. A product should be tested to take such pulses and 
should recover provided it has not suffered a catastrophic failure (and in fact it should contain sufficient protection 
to avoid such in most cases).

In working on one particular router in the lab some years ago, I was verifying some software functionality and the 
hardware engineer I was working with reached over my shoulder and used a device that delivered a high voltage spike 
(simulated lightning) to a 10BaseT network port. After I peeled myself off the ceiling (and he stopped laughing), we 
set to work figuring out how to get the device to self-reset after such a strike. One component, an Ethernet hub chip, 
got into a confused state. I was able to detect this in software, so we adjusted the product design so that the 
software could yank the hub chip's reset line.

It's unfortunate that products, both hardware and software, receive minimal quality testing these days. Guess it's not 
a surprise, since buyers seemed to prefer products that were quick to market, with lots of bugs, rather than 
reliability and resilience.


On May 13, 2010, at 12:39 PM, Pete Carah wrote:

> On 05/13/2010 12:19 PM, Larry Sheldon wrote:
> > On 5/13/2010 10:36, Caleb Tennis wrote:
> >
> > > We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that 
> > > goes outdoors underground to our facility's gate.  
> > >
> > > What's interesting is that various POE switches throughout the entire building seemed to be affected in that some 
> > > of their ports they just shut down/off.  Rebooting these switches brought everything back to life.  It didn't 
> > > impact anything non-POE, and even then, only impacted some devices.  But it was spread across the whole building, 
> > > across multiple switches.
> > >
> > > I was just curious if anyone had seen anything similar to this before?  Our incoming electrical power has surge 
> > > suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of 
> > > them would have been impacted at all.  I'm guessing that the strike had some impact on the electrical ground, but I 
> > > don't know what we can do to prevent future strikes from causing the same issues.  Thoughts?
> >
> > I don't know how to account for this in a PoE world, but when I last
> > managed a campus network, we had major issues (particularly in an
> > active-thunder-storm environment) of severe difference in
> > ground-potential between buildings.
> Cat 5 has isolation transformers in or just behind each jack.  However,
> in most equipment the grounds aren't really isolated, and in the case of
> POE they (mostly) aren't at all.
>
> Lightning likes to do "interesting" things.  It can induce a 20kv per
> few feet gradient (or more) across the ground mesh of a power substation
> (like 4/0 wire in a mesh of 4 foot squares or so; normally more
> complicated than that since it has to clear equipment etc...).  It likes
> to eat power supplies in well-grounded equipment and leave cheaper stuff
> alone.  It can hit an antenna, leave the receiver completely intact, and
> fry the power supply of the next box over.  We tended to lose either
> fluorescent ballasts or the thermostat transformer in our furnace when I
> lived in an active ham's house in Alabama, the radios tended to live. 
> (you should have seen his coax entry panel (1/4 inch copper sheet,
> grounded outside)), and stuff got manually disconnected from both
> antennas and power when a storm was expected (every afternoon :-).
>
> It wouldn't surprise me if the first answer was right and either the
> ground pulse or EMP reset the safety switches in the POE feeders.
>
> -- Pete