FC: Internal glitches shut down Boston hospital for four days

[Declan McCullagh <declan () well com>]

---

From: "Richard M. Smith" <rms () computerbytesman com>
To: "'Declan McCullagh'" <declan () well com>
Subject: Cyberattack shuts down Boston's Beth Israel Deaconess Hospital for 
4 days
Date: Mon, 2 Dec 2002 16:08:56 -0500

Hi,

About 6 months ago, I did an informal survey asking if people knew on
any examples of cyberterror attacks.  At the time, no one could point me
to any real attacks.  A few weeks ago however, Boston's Beth Israel
Deaconess Hospital did suffer a massive denial of service attack which
shutdown the internal computer networks for almost 4 days and forced the
hospital to cut back on its operations.  The attached Boston Globe
article gives the details.

As you can see from this article, the wounds however were
self-inflicted.  The shutdown of the computer network at Beth Israel
Deaconess Hospital does illustrate that critical infrastructure can be
disrupted via software.  However it is still very unclear how likely
this type of disruption can be caused by an outside hacker.

After I did my survey, I did find three examples of cyberattacks on
critical infrastructure:

  Juvenile computer hacker cuts off FAA tower
  http://www.usdoj.gov/criminal/cybercrime/juvenilepld.htm

  Net saboteur faces 41 months
  http://www.nwfusion.com/news/2002/0304lloyd.html

  Aussie hacker jailed for sewage attacks
  http://cooltech.iafrica.com/technews/837110.htm

As two of these cases illustrate, the insider threat is generally
greater than the outside hacker threat.

Richard M. Smith
http://www.ComputerBytesMan.com

========================================================

http://www.boston.com/dailyglobe2/330/science/Got_paper_P.shtml

Got paper? Beth Israel Deaconess copes with a massive computer crash
By Anne Barnard, Globe Staff, 11/26/2002

Thirteen days ago, as his computer crunched the mountain of data he
hoped would be his humble contribution to medical progress, the
researcher - he shall remain nameless - got a phone call he'd never
forget.

It was Dr. John Halamka, the former emergency-room physician who runs
Beth Israel Deaconess Medical Center's gigantic computer network. He
told the professor that his flood of numbers was overwhelming the
system, threatening to freeze thousands of electronic medical records
and grind the hospital's network to a halt.

''He said, `Oh, my God!' and pulled the plug out of the wall,'' Halamka
said last week.

It was too late. Somewhere in the web of copper wires and glass fibers
that connects the hospital's two campuses and satellite offices, the
data was stuck in an endless loop. Halamka's technicians shut down part
of the network to contain it, but that created a cascade of new
problems.

The entire system crashed, freezing the massive stream of information -
prescriptions, lab tests, patient histories, Medicare bills - that
shoots through the hospital's electronic arteries every day, touching
every aspect of care for hundreds of patients.

Within a few hours, Cisco Systems, the hospital's network provider, was
loading thousands of pounds of network equipment onto an airplane in
California, bound for a 2 a.m. arrival at Logan International Airport.
In North Carolina's Research Triangle area, computer experts were being
rousted out of bed to join a batallion of electronic shock troops who
would troubleshoot the situation. Closer to home, Cisco technicians were
converging on Boston from across Massachusetts.

The crisis began on a Wednesday afternoon, Nov. 13, and lasted nearly
four days. Before it was over, the hospital would revert to the paper
systems that governed patient care in the 1970s, in some cases reverting
to forms printed ''Beth Israel Hospital,'' from before its 1996 merger.
Hundreds of employees, from lab technicians to chief executive officer
Paul Levy, would work overtime running a quarter-million sheets of paper
from one end of the campus to the other.

And hospitals across the country - not to mention investment banks,
insurance companies and every other business that relies on a constantly
accessible stream of quickly-changing information - would get a scary
reminder of how dependent they are on their networks, and what would
happen if they disappeared.

''It's like the Y2K that never happened,'' said Dianne Anderson, vice
president for patient care services at Beth Israel Deaconess.

Now, Halamka - the hospital's chief information officer and a networking
addict who answers e-mails on his Blackberry device whether he's at a
meeting or a family dinner - is hustling to answer questions from all
over the country, from community hospitals in Western Massachusetts and
major medical centers such as Johns Hopkins University, and
financial-services companies that could lose millions in a crash.

''The message,'' he said, ''is make sure you're ready for a massive
disruption of your network - whether it's 9/11 or a natural disaster or
whatever.''

As a result of the crash, Beth Israel Deaconess plans to spend $3
million to replace its entire network - creating an entire parallel set
of wires and switches, double the capacity the medical center thought it
needed.

No other Massachusetts hospital has ever reported such a long-lasting or
disruptive network crash, said Elliot Stone, executive director of the
Massachusetts Health Data Consortium, a group that brings together chief
information officers from hospitals and health plans around the state.
He praised Beth Israel Deaconess for being open about the problem and
sharing lessons learned, both about technology itself and about policy -
such as the need to enforce rules against unauthorized additions of new
software onto the network. Not least, Stone said, Halamka's counterparts
see the incident as ammunition in their constant quest to convince
management to pay for network upgrades.

The crash surprised experts in the field because most disaster planners
mainly worry about backing up hard drives and building redundant
servers. But in this case, it wasn't those repositories of information
that were in trouble. It was the network itself - the ''pipes'' that
carry the information from one place to the other. It was like when at
busy times at the office, your e-mail slows down - only so bad that
everything ceased to function.

''Usually, when you think about backup, you're talking about backing up
hard drives. You don't think about the network itself,'' said Mark
Tuomenoksa, founder and chairman of Woburn-based OpenReach, a
network-security consulting company.
Halamka said that was the case at Beth Israel Deaconess: ''We don't just
have a backup generator, we have a backup-backup generator, and then we
have batteries. Servers are clustered; data writes on five different
hard drives.'' There is even a double ''pipeline'' between the computer
center on Tremont Street and Beth Israel Deaconess's main campuses - but
during the crash, both were clogged.

The crisis had nothing to do with the particular software the researcher
was using. The problem had to do with a system called ''spanning tree
protocol,'' which finds the most efficient way to move information
through the network and blocks alternate routes to prevent data from
getting stuck in a loop. The large volume of data the researcher was
uploading happened to be the last drop that made the network overflow.

Halamka said Beth Israel Deaconess's recent economic troubles were not
behind the problem. In fact, on Oct. 1, hospital officials had approved
a consultant's plan to overhaul the network - just not quite in time.
''Now,'' he said, ''we're going to do it faster.''

The crisis also tapped into medicine's ambivalence about computers.
Yesterday, doctors at Brigham and Women's Hospital reported in the
Archives of Internal Medicine that 73 percent of medication-related
mistakes involved in malpractice claims are preventable and probably
could be averted through computerized prescription ordering - the latest
in a growing pile of evidence that computerization can cut medical
errors.

At the same time, clinicians have sometimes been wary of turning over
control to a computer, Tuomenesko said: ''When I enter something into a
computer, how do I know it got there?''

That was part of the problem Beth Israel Deaconess had: New information
could sometimes be entered, but since network function was fading in and
out, clinicians weren't sure whether that information was being
delivered. So, the hospital decided to shut down the computers - taping
handwritten ''Do Not Use'' notes to monitors - creating an instant
generation gap, said Anderson, the hospital's top nurse executive.

''Nurses and doctors over the age of 35 were very much at ease,'' she
said. ''The younger nurses and doctors were very uncertain. We were
teaching residents how to write orders; we were showing nurses how to do
flow sheets.''

Meanwhile, the hospital was figuring out how to run at its usual pace
without the 100,000 e-mails it usually sends a day. The lab was dumping
3,000 results a day on paper into plastic bins, to be delivered by
runners who came by every 10 to 15 minutes. Microbiologists were
ferrying lab results. Cardiac fellows were digging through paper records
to find old cardiograms to compare to new ones. People at all levels of
the hospital hierarchy had to deal with each other face to face.

''The lab is usually anonymous until something goes wrong,'' said Gina
McCormack, technical director of the West Campus lab. ''A lot of people
realized we're here. People got to understand each other's jobs.''
 




-------------------------------------------------------------------------
POLITECH -- Declan McCullagh's politics and technology mailing list
You may redistribute this message freely if you include this notice.
To subscribe to Politech: http://www.politechbot.com/info/subscribe.html
This message is archived at http://www.politechbot.com/
Declan McCullagh's photographs are at http://www.mccullagh.org/
-------------------------------------------------------------------------
Like Politech? Make a donation here: http://www.politechbot.com/donate/
Recent CNET News.com articles: http://news.search.com/search?q=declan
-------------------------------------------------------------------------