RE: Building a NOC

[Morgan Sarges <blip () morgan iw net>]

On 24-Mar-98 Howard C. Berkowitz wrote:
> At 09:22 +0000 3/24/98, Bill Unsworth wrote:
> > At 08:52 AM 3/23/98 -0500, you wrote:
[snip]


The notes below will be a keeper for our organization for some time.

We are in the process of building a NOC.  Anyone interested in hearing
the progress (or lack of) thus far, let me know and I can email some
of the information direct.

Thanks to those who contributed to the list below, I will be using it
much from now on.

Morgan


> The anecdotes in this discussion thread, I think, are definitely worth
> keeping somewhere, although I'm not sure of the venue.  Some of us had been
> vaguely chatting in Albuquerque about a documentation part of the NANOG web
> page.
>
> But some non-obvious threads are surfacing in this discussion, at least
> non-obvious if you haven't been there.  Restating a few of the less obvious
> ones,
>
>   -- Define the NOC's function
>
>   -- Define the NOC's audience, including people that will have no useful
>      function but do have political influence that MUST be satisfied
>
>   -- When considering a physical site, do a careful and paranoid threat
>      of the planned space, the building, and its immediate environs.  You
>      will want to identify possible hazards including fire, flood, etc.
>
>   -- Consider how you will get large equipment in and out of the site,
>      especially those that might need emergency replacement.  What if the
>      building is on limited power and the elevator is down?  Some buildings
>      have to have large equipment lifted in with a crane or helicopter.
>      Air conditioning equipment or major power supplies are examples of
>      very hard to handle components.
>
>   -- As part of the site survey, understand how electrical power and
>      communications feeders come into the space and building.  Are there
>      alternatives for redundancy?
>
>   -- Will there be a backup NOC, even consisting of a node into which staff
>      can dial?
>
>   -- Do a thorough electrical requirmements plan, and be sure the grounding
>      system is up to current practices.  When devices have multiple power
>      supplies (e.g., Cisco 7000/7500), be sure they are plugged into circuits
>      on different breakers.
>
>   -- Think through who will be planning, installing, and inspecting signal
>      cabling.  If it's a union shop, or especially if non-union personnel
>      will be doing any work in a union area, see if you can get the shop
>      steward on your side.
>
>   -- Be sure cellular/cordless phones will work in all your wire closets,
>      or be sure there are voice jacks where a telephone set can be plugged
>      in to coordinate testing.
>
>   -- When planning cable runs, be sure you can access critical components
>      that otherwise might be buried under a heavy mass of cables.  Plan the
>      runs so they will not interfere with cards sliding in and out of
> chassis.
>
>   -- Beware of snakes, floor drains, shock hazards, etc., under raised
> floors.
>
>   -- Consider backup facilities for critical people.  If there was a major
>      disaster that isolated the site, but still let some of its function
> work,
>      where will staff sleep?  Are there emergency food supplies (not a bad
>      idea even beyond major disasters -- if people are working around the
>      clock, it's good to have alternatives besides pizza, if that is
> available)?
>
>
> Especially if you are in earthquake or amorous rat country, think about
> catastrophes that can knock out the NOC but leave parts of your network
> working.

-----------------------------------------
Morgan Sarges                   Voice Phone:  605-338-8334
blip () morgan iw net              Fax:          605-335-3942

System Administrator
Dakota Telecommunications Group, Internet Division
Network Operations Center

No man is an island, but some of us are long peninsulas.


PGP Public Key block available upon request.
------------------------------------------