nanog mailing list archives

Re: Operations task management software?


From: Matt Ryanczak <ryanczak () gmail com>
Date: Tue, 02 Aug 2016 13:59:01 +0000

Jira works well as a task tracking system for ops. Customizable work flows,
decent integration with ldap, etc. Also good for tracking software
projects. Having both software and ops tasks in one place has many benefits.

On Wed, Jul 27, 2016, 16:28 David Hubbard <dhubbard () dino hostasaurus com>
wrote:

Full automation is planned but does not eliminate the need for the
software.  Zero human auditing of fully automated processes and data
collection are not acceptable to various certifying entities, the relevant
auditors, the inevitably involved lawyers, and won’t pick up on bad data,
like a bad thermometer or snmp counter that says a CRAC is 65 degrees when
it’s really 90.  So I’m still going to need a management solution to the
issue whether it’s to tell someone to do the work or to tell someone to
check the automated work.

David

On 7/27/16, 7:19 PM, "Lee" <ler762 () gmail com> wrote:

    On 7/27/16, David Hubbard <dhubbard () dino hostasaurus com> wrote:
    > Hi all, curious if anyone has recommendations on software that helps
manage
    > routine duties assigned to operations staff?

    Have computers do the routine scut work - not people.

    > For example, let’s say we have a P&P that says someone from the
netops group
    > must check that Rancid is successfully backing up all router configs
    > bi-weekly.

    You've got the source code for rancid, so change rancid-run to do
something like
      LOGFILE=$LOGDIR/$GROUP.`date +%Y%m%d.%H%M%S`; export LOGFILE
    change the
      ) >$LOGDIR/$GROUP.`date +%Y%m%d.%H%M%S` 2>&1
    to
      ) >$LOGFILE 2>&1

    and then in control_rancid do something like
      grep "clogin error:" $LOGFILE | sort | uniq -c >$TMP.fail
      if [ -s $TMP.fail ]; then
         # got some output, mail the report
         ...

    Do the same type thing for checking on
    > backup failures, backup internet circuit status, out of band
interfaces, etc.

    Automate the checks, put the scripts in crontab & mail out an
    "OhNoes!" or "all clear" msg at the end.   At which point you're left
    with the problem of making sure the managers are looking at the emails
    & making sure whatever problems are found actually get fixed :)

    Regards,
    Lee





Current thread: