Disruptions Put Bank of New York to the Test

[InfoSec News <isn () c4i org>]

http://www.nytimes.com/2001/10/06/business/06BONY.html

By SAUL HANSELL with RIVA D. ATLAS
October 6, 2001

> From the roof of the Bank of New York's Art Deco headquarters, just
down the block from the New York Stock Exchange, one can easily see
why the bank has been at the uncomfortable center of Wall Street's
recovery.

Before Sept. 11, nearly half of the bank's 19,000 employees, including
those responsible for its main computer center, worked within view of
the headquarters.

Now the vista is dominated by the still- smoldering rubble of the
World Trade Center; just beyond are two dust-covered buildings that
had housed 5,300 employees and the bank's main computer center.

The Bank of New York is woven into the fabric of Wall Street by more
than just its location. It has played down the lending and
deposit-taking functions of most banks and devoted itself to the
plumbing of finance, accounting for and moving money and securities
for traders, corporations and investors. Half of the trading in
government bonds, for example, moves through its settlement system. As
such, many Wall Street firms depend on the bank to ensure that their
trading runs smoothly.

Despite a reluctance to criticize publicly as the financial world is
striving to show unity, some traders at those firms say the Bank of
New York has had a harder time recovering from the disaster than
others, with system failures and delayed processing leaving some firms
in the dark about their financial positions or waiting extra days for
money or securities to arrive.

The Bank of New York, some say, was just too concentrated in
Manhattan. The bank's penchant for lean operations, others surmise,
may have kept it from having quite the systems or the manpower to
recover quickly.

Now, stock analysts wonder whether the bank may lose business from
clients who want to be less vulnerable in the future.

Thomas A. Renyi, the Bank of New York's chairman, acknowledges that
there have been flaws in the bank's service. But he says the bank has
bounced back from a nearly unthinkable disruption largely intact. "We
are proud of what we have done," Mr. Renyi said, from his customary
spot at the end of a vast wooden conference table next to his office
in the headquarters at 1 Wall Street.

"Clearly there were ups and downs," he said. "But our longstanding
disaster recovery plans worked, and they worked in the extreme." Until
Sept. 11, no company had ever had to cope with a sudden dislocation of
so many people and computer systems, the bank's executives point out.
Indeed, it is a testament to contingency planning across Wall Street
that so many companies were back in business only days after being
displaced.

But the Bank of New York, because its systems are so central to the
business of all the other firms, was under extraordinary pressure to
resume running at full speed. The places where it lagged became
especially noticeable.

Mr. Renyi also gingerly suggests that the Bank of New York may have
become a lightning rod for a wide range of complaints. Broken
communication links were the bulk of the problems, he said, and it is
not clear whether the fault, if any, lies with the sender, the
receiver or the phone company in the middle. But those problems showed
up when the bank fell behind on processing trades. "We are a highly
visible provider and have a highly vocal customer base," Mr. Renyi
said.  Michael Mayo, an analyst at Prudential Securities, said that
many of the bank's customers were understanding of its problems, for
now.

"The Bank of New York had some bad luck and was given a grace period
by their customers, but eventually the grace period will expire," Mr.
Mayo said. "This is a big test for the Bank of New York's management.
The spotlight is on the firm to get their problems fixed."  Bank
executives argue that some of the criticism has taken on some aspects
of urban legend, especially the notion that the bank was in disarray
because the main backup for its computer center in Lower Manhattan was
at another location in Lower Manhattan.

The bank says that all of its several computer centers in Manhattan
were always set to revert to centers outside the city in case of
emergency, and they did on Sept. 11. The bank also insists that its
spending for disaster recovery has increased substantially in recent
years.

In a few cases, though, the bank simply threw out its plan. The
department that processes paper checks at its computer center, at 101
Barclay Street, was supposed to move to Cherry Hill, N.J. But with
communication so disrupted, the bank did not want that operation so
far away.

So the checks were sent to another check processing center in Lodi,
N.J. But that center did not have machines needed for the bank's
lockbox business, in which it opens envelopes that contain bill
payments, both depositing checks and reading payment stubs to credit
the right accounts. So the bank is only now catching up with
processing some of those small payments.

Mr. Renyi said the bank had quite deliberately planned to have
different levels of backup for different functions. The crucial
government bond processing, for example, had a system in which a
second computer was receiving and processing all the data going into
the main computer, making it ready to pick up at a moment's notice, in
theory.

The bank had no such system for its 350 automated teller machines. The
rationale was that its customers would be able to use other banks'
machines in case of a problem. That is what customers were forced to
do for more than a week after the attack. (The bank absorbed fees
imposed by other banks.)

As it turned out, though, even the expensive backup system was unable
to get the government bond business up and running smoothly. That is
largely because of problems maintaining the communications links that
receive information on trades from its customers and report their
positions back to them.

"In many cases our backup sites were dealing with our customers'
backup sites," Mr. Renyi said. And though the bank had established
communications lines in advance connecting these various backup
centers, they often were of low capacity and typically had not been
fully tested and debugged.

Even a week after the attack, the Bank of New York was having trouble
with some crucial communications links, like its connection to the
Government Securities Clearing Corporation, a central part of the
government bond market. On several days that week, the bank had to
drive computer tapes with its trades to G.S.C.C. offices. Though the
trades were ultimately posted properly, clients were deprived of
instantaneous reports on their positions.

"Everyone had redundant telecommunications facilities, but a lot of
them turned out to be routed through the same phone company offices,"
said Thomas F. Costa, chief operating officer of the Government
Securities Clearing Corporation. "We've all learned that when we have
backup lines, we should know a lot more about where they run."

The bond market, including the Bank of New York, is getting back up to
speed and working through its backlog of work. Still, it could be
months before the Bank of New York can return to its computer center
to 101 Barclay Street. The bank is planning to build a new data center
to house its operations in the intervening months. The bank hired
I.B.M. to help it figure out where to put the center and to look at
other ways to bolster its backup systems. But Mr. Renyi said the bank
still plans to move back to Barclay Street so it can stay close to its
clients.

Last week, the bank said the disaster would knock $125 million off its
profits, although insurance will reimburse much of that. Less clear,
however, is the long-term effect. Few analysts expect the weeklong
disruption in A.T.M. service to cause many individual customers to
leave. But the big Wall Street firms are a more demanding group. The
only other player in government securities clearing is J. P. Morgan
Chase, but State Street and Mellon Financial compete, as well, in
other processing areas.

Mr. Renyi says companies will certainly talk about whether they want
to rely on a single provider of processing services, just as the Bank
of New York, he said, is looking at whether it was too dependent on
some vendors.

But he remains optimistic that the bank will pick up some business
that had gone to other banks. "We can say that we are the only one
that has been tested in an extreme situation," he said. "If there is
someone that has a blue-chip recovery plan that has been proven to
work, it should be the Bank of New York."



-
ISN is currently hosted by Attrition.org

To unsubscribe email majordomo () attrition org with 'unsubscribe isn' in the BODY
of the mail.