Re: Rogers Outage Canada

[Jon Sands <fohdeesha () gmail com>]

Given the outage was so bad it was disrupting select E911 services 
nationwide for something like 24+ hours, it's great to see such an in 
depth analysis and plan of action to prevent such things in the future. 
bravo rogers

On 7/10/2022 2:55 PM, L F wrote:
> fyi - see BOLD.
>
> A Message from Rogers President and CEO
>
>
>
> Dear Valued Customer,
>
> As you know, we experienced a service outage across the Rogers, Fido, 
> Chatr and Cityfone wireless networks on Friday.
>
> I am reaching out to share that our services have been restored, and 
> our networks and systems are close to fully operational. Our technical 
> teams are continuing to monitor for any remaining intermittent issues. 
> I also want to outline an action plan we are putting in place to 
> address what happened.
>
> I want to share what we know about what happened on Friday. We now 
> believe *we’ve narrowed the cause to a network system failure 
> following a maintenance update in our core network, which caused some 
> of our routers to malfunction. We disconnected the specific equipment 
> and redirected traffic,*which allowed our network and services to come 
> back online over time as we managed traffic volumes returning to 
> normal levels.
>
>
>
>
> On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas () gmail com> wrote:
>
>
>     Lest we ALL MOVE ON….
>
>
>     Yes he said
>
>     “RETARD” =
>
>     The delay in processing the current status of a Situation…
>
>
>     Lets move on to bigger n scarier Post mortems:
>
>
>     Was This or Was this NOT
>
>     A CYBERATTACK ???
>
>     Done.
>
>     Lets deal with REAL threats not PC references.
>
>     Time to evaluate n re eval Our NTS!!
>
>     Veni vidi vici 2022
>
>
>
>
>     On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke () gmail com>
>     wrote:
>
>         Can we have a discussion with the list admins about a list
>         member appending a threat of violence to their outbound
>         emails?  Whether serious or not.
>
>         Does this person need directions to some local mental
>         healthcare resources?
>
>
>         On Sat, 9 Jul 2022 at 08:48, Keith Medcalf
>         <kmedcalf () dessus com> wrote:
>
>
>             >I can't either, but the reality right now seems to be
>             that 911 calls are
>             >failing for anyone on a Rogers cellphone.
>
>             This is par for the course.  These people chose to deal
>             with Rogers despite knowing the consequences.  It is like
>             if you bought a Rogers Snowblower and it did not work. 
>             That would mean that people who bought the Rogers
>             Snowblower will not be using it to get rid of the snow
>             that is preventing them from leaving their house.
>
>             Mutatis mutandis when Rogers is down things that are
>             Rogers dependent will not work.
>
>             Some people are so retarded it is astonishing!
>
>             -- 
>             (CAUTION) You are advised that if you attack my person or
>             property, you will be put down in accordance with the
>             provisions of section 34 & 35 of the Criminal Code
>             respectively.  If you are brandishing (or in possession)
>             of a weapon then lethal force will be applied to your
>             person in accordance with the law.  This means that your
>             misadventures may end in your death. Consider yourself
>             cautioned and govern your actions appropriately.
>
>             >-----Original Message-----
>             >From: NANOG <nanog-bounces+kmedcalf=dessus.com () nanog org>
>             On Behalf Of
>             >Eric Kuhnke
>             >Sent: Friday, 8 July, 2022 13:34
>             >To: jim deleskie <deleskie () gmail com>
>             >Cc: NANOG list <nanog () nanog org>
>             >Subject: Re: Rogers Outage Canada
>             >
>             >
>             >I have seen anecdotal reports that the mobile network is
>             in a half broken
>             >state that phones remain registered to, so a 911 call
>             will attempt and
>             >then fail.
>             >
>             >
>             >This is unlike what would happen if you had a US/Canada
>             cellphone with
>             >battery power but no SIM card in it that would search for
>             any available
>             >network in RF range for a 911 call if needed.
>             >
>             >
>             >On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie () gmail com
>             ><mailto:deleskie () gmail com> > wrote:
>             >
>             >
>             >       i cant see BGP taking out SS7.
>             >
>             >       -jim
>             >
>             >       On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004
>             ><greenjosh6499 () gmail com <mailto:greenjosh6499 () gmail com>
>             > wrote:
>             >
>             >
>             >               According to Cloudflare Radar
>             ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours>
>             , Rogers
>             >BGP announcements spiked massively to levels 536,777%
>             higher than normal
>             >(343,601 vs 64 normally) just minutes before the outage.
>             I would not be
>             >surprised if this happened to be the culprit.
>             >
>             >               Regards,
>             >               Josh Green
>             >
>             >               On Fri, Jul 8, 2022 at 2:19 PM Andrew
>             Paolucci via NANOG
>             ><nanog () nanog org <mailto:nanog () nanog org> > wrote:
>             >
>             >
>             >                       In the early hours of the morning
>             around 2-3am my modem
>             >got hit with a configuration update that caused a DHCP
>             release that
>             >wasn't renewed for about two hours, after rollback the
>             connection was
>             >fine for 3 hours before this network wide outage.
>             >
>             >
>             >                       Maybe a failed night time update
>             was attempted again
>             >during office hours, I've heard daytime guys are still
>             WFH and night
>             >shift is in building.
>             >
>             >
>             >                       I expect we'll never get a real
>             explanation. Rogers is
>             >notorious for withholding any type of helpful or
>             technical information.
>             >
>             >
>             >                       Sent from my inoperable Rogers
>             Mobile via emergency eSIM.
>             >
>             >
>             >                       Regards,
>             >
>             >                       Andrew Paolucci
>             >                       -------- Original Message --------
>             >                       On Jul. 8, 2022, 1:48 p.m., Jay
>             Hennigan < jay () west net
>             ><mailto:jay () west net> > wrote:
>             >
>             >
>             >                               On 7/8/22 07:44, Robert
>             DeVita wrote: > Does anyone
>             >have information on a widespread Rogers outage in Canada.
>             I > have
>             >customers with multiple sites down. There's discussion on
>             the Outages
>             >mailing list. Seems widespread, affecting all services,
>             mobile, voice,
>             >Internet. No cause or ETR posted yet. -- Jay Hennigan -
>             jay () west net
>             ><mailto:jay () west net> Network Engineering - CCIE #7880
>             503 897-8550 -
>             >WB6RDV
>             >
>             >
>             >
>             >               --
>             >
>             >               Josh Green.

--
Jon Sands
MFI Labs
https://fohdeesha.com/