Re: Disaster Recovery Process

[jim deleskie <deleskie () gmail com>]

World broke.  Crazy $$ per hour down time.  Doors open with a fire axe.
Glass breaks super easy too and much less expensive then adding 15 min to
failure.

-jim

On Tue., Oct. 5, 2021, 7:05 p.m. Jeff Shultz, <jeffshultz () sctcweb com>
wrote:

> 7. Make sure any access controlled rooms have physical keys that are
> available at need - and aren't secured by the same access control that they
> are to circumvent. .
> 8. Don't make your access control dependent on internet access - always
> have something on the local network  it can fall back to.
>
> That last thing, that apparently their access control failed, locking
> people out when either their outward facing DNS and/or BGP routes went
> goodbye, is perhaps the most astounding thing to me - making your access
> control into an IoT device without (apparently) a quick workaround for a
> failure in the "I" part.
>
> On Tue, Oct 5, 2021 at 6:01 AM Jared Mauch <jared () puck nether net> wrote:
>
> > > On Oct 4, 2021, at 4:53 PM, Jorge Amodio <jmamodio () gmail com> wrote:
> > >
> > > How come such a large operation does not have an out of bound access in
> > case of emergencies ???
> > >
> >
> > I mentioned to someone yesterday that most OOB systems _are_ the
> > internet.  It doesn’t always seem like you need things like modems or
> > dial-backup, or access to these services, except when you do it’s
> > critical/essential.
> >
> > A few reminders for people:
> >
> > 1) Program your co-workers into your cell phone
> > 2) Print out an emergency contact sheet
> > 3) Have a backup conference bridge/system that you test
> >   - if zoom/webex/ms are down, where do you go?  Slack?  Google meet?
> > Audio bridge?
> >   - No judgement, but do test the system!
> > 4) Know how to access the office and who is closest.
> >   - What happens if they are in the hospital, sick or on vacation?
> > 5) Complacency is dangerous
> >   - When the tools “just work” you never imagine the tools won’t work.
> > I’m sure the lessons learned will be long internally.
> >   - I hope they share them externally so others can learn.
> > 6) No really, test the backup process.
> >
> >
> >
> > * interlude *
> >
> > Back at my time at 2914 - one reason we all had T1’s at home was largely
> > so we could get in to the network should something bad happen.  My home IP
> > space was in the router ACLs.  Much changed since those early days as this
> > network became more reliable.  We’ve seen large outages in the past 2 years
> > of platforms, carriers, etc.. (the Aug 30th 2020 issue is still firmly in
> > my memory).
> >
> > Plan for the outages and make sure you understand your playbook.  It may
> > be from snow day to all hands on deck.  Test it at least once, and ideally
> > with someone who will challenge a few assumptions (eg: that the cell
> > network will be up)
> >
> > - Jared
>
>
>
> --
> Jeff Shultz
>
>
> Like us on Social Media for News, Promotions, and other information!!
>
>    <https://www.facebook.com/SCTCWEB/>     [image:
> https://www.instagram.com/sctc_sctc/]
> <https://www.instagram.com/sctc_sctc/>
> <https://www.yelp.com/biz/sctc-stayton-3>
> <https://www.youtube.com/c/sctcvideos>
>
>
>
>
>
>
>
> **** This message contains confidential information and is intended only
> for the individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. E-mail transmission cannot be
> guaranteed to be secure or error-free as information could be intercepted,
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
> The sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> transmission. ****