nanog mailing list archives

Re: Are the days of the showpiece NOC office display gone forever?


From: Tom Beecher <beecher () beecher cc>
Date: Thu, 17 Dec 2020 12:58:43 -0500


I'm sure when the automation is perfect and widespread to the point that
it catches and alerts on every network event, the monitoring rooms will
disappear.


The chances of this happening are exactly 0%.


But unless you have an entire organization dedicated to automation
development or pay an incredibly large sum of money for pre-built packages,
the business decision may still be made to actively monitor the network
with eyeballs.


Contrary to what salespeople will say, the answer is not 100% automation,
or 100% humans. The proper answer is an often changing combination of the
two.

 Every failure mode is known until a new one pops up. Automation without
any kind of ML secret sauce relies on known failure-modes.


ML is not the magical unicorn solution that solves everything, contrary to
what many papers and salespeople tell you. Let's take a network interface
that is randomly shitting on packets. What is more important operationally,
identifying that the packets are being shat on, or having ML predict when
the next shatting will occur? Clearly the first right? You want to find it
and fix it as fast as possible. There could be a place for ML when it comes
to diagnosing the reason for the shitting , but that's different.

There may be some interesting applications for ML in tracking down really
complicated operational anomalies, but it will never be a primary mode of
detection for the same reason you said ; Every failure mode is known until
a new one pops up. You can't train an ML model for a failure condition that
you don't know exists.

On Wed, Dec 16, 2020 at 5:51 PM Matt Erculiani <merculiani () gmail com> wrote:

 That is not to say that large monitoring rooms are a better choice over
automation (which they are not).

I'm sure when the automation is perfect and widespread to the point that
it catches and alerts on every network event, the monitoring rooms will
disappear.

But unless you have an entire organization dedicated to automation
development or pay an incredibly large sum of money for pre-built packages,
the business decision may still be made to actively monitor the network
with eyeballs.

Every failure mode is known until a new one pops up. Automation without
any kind of ML secret sauce relies on known failure-modes.

Not advocating one or the other, just playing Devil's advocate for the
Devil's advocate.

-Matt

On Wed, Dec 16, 2020 at 2:28 PM Töma Gavrichenkov <ximaera () gmail com>
wrote:

Peace,

On Thu, Dec 17, 2020, 12:21 AM Lady Benjamin PD Cannon <ben () 6by7 net>
wrote:

We are still operating ours - 27 1080P projectors - but with a skeleton
crew of just 3.  Given the air volume, it’s almost like outside.


A devil advocate here,

First of all, COVID-19 is really serious.

With that in mind, with all the necessary precautions office space *may*
be managed safely to prevent the spread.

Production plants had security measures preventing workforce injuries for
a century already.  Just a bit of that, with constant trainings, would get
your monitoring room safe, especially with all the bars closed and
everything operating on delivery.

That is not to say that large monitoring rooms are a better choice over
automation (which they are not).

--
Töma



--
Matt Erculiani
ERCUL-ARIN


Current thread: