Re: Rogers Outage Canada

[Victor Kuarsingh <victor () jvknet com>]

This is the most they can and will say.  For liabilities reasons, specifics
are likely not in the cards.  As most services ride over common service
networks, its quite possible that a network substrate failure can have a
number of upstream service impacts.  The point here is that the CEO is
directly addressing the customer base, which is needed here.

regards,

Victor K

On Mon, Jul 11, 2022 at 10:11 AM Shane Ronan <shane () ronan-online com> wrote:

> What in depth analysis have you seen? Seems to me, this was a failure in a
> known maintenance activity, and they simply disconnected the devices under
> maintenance from the network.
>
> Shane
>
> On Mon, Jul 11, 2022 at 5:41 AM Jon Sands <fohdeesha () gmail com> wrote:
>
> > Given the outage was so bad it was disrupting select E911 services
> > nationwide for something like 24+ hours, it's great to see such an in depth
> > analysis and plan of action to prevent such things in the future. bravo
> > rogers
> >
> > On 7/10/2022 2:55 PM, L F wrote:
> >
> > fyi - see BOLD.
> >
> > A Message from Rogers President and CEO
> >
> >
> > Dear Valued Customer,
> >
> > As you know, we experienced a service outage across the Rogers, Fido,
> > Chatr and Cityfone wireless networks on Friday.
> >
> > I am reaching out to share that our services have been restored, and our
> > networks and systems are close to fully operational. Our technical teams
> > are continuing to monitor for any remaining intermittent issues. I also
> > want to outline an action plan we are putting in place to address what
> > happened.
> >
> > I want to share what we know about what happened on Friday. We now
> > believe *we’ve narrowed the cause to a network system failure following
> > a maintenance update in our core network, which caused some of our routers
> > to malfunction. We disconnected the specific equipment and redirected
> > traffic,* which allowed our network and services to come back online
> > over time as we managed traffic volumes returning to normal levels.
> >
> >
> >
> > On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas () gmail com> wrote:
> >
> > > Lest we ALL MOVE ON….
> > >
> > >
> > > Yes he said
> > >
> > > “RETARD” =
> > >
> > > The delay in processing the current status of a Situation…
> > >
> > >
> > > Lets move on to bigger n scarier Post mortems:
> > >
> > >
> > > Was This or Was this NOT
> > >
> > > A CYBERATTACK ???
> > >
> > > Done.
> > >
> > > Lets deal with REAL threats not PC references.
> > >
> > > Time to evaluate n re eval Our NTS!!
> > >
> > > Veni vidi vici 2022
> > >
> > >
> > >
> > >
> > > On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke () gmail com>
> > > wrote:
> > >
> > > > Can we have a discussion with the list admins about a list member
> > > > appending a threat of violence to their outbound emails?  Whether serious
> > > > or not.
> > > >
> > > > Does this person need directions to some local mental healthcare
> > > > resources?
> > > >
> > > >
> > > > On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf () dessus com> wrote:
> > > >
> > > > > > I can't either, but the reality right now seems to be that 911 calls
> > > > > are
> > > > > > failing for anyone on a Rogers cellphone.
> > > > >
> > > > > This is par for the course.  These people chose to deal with Rogers
> > > > > despite knowing the consequences.  It is like if you bought a Rogers
> > > > > Snowblower and it did not work.  That would mean that people who bought the
> > > > > Rogers Snowblower will not be using it to get rid of the snow that is
> > > > > preventing them from leaving their house.
> > > > >
> > > > > Mutatis mutandis when Rogers is down things that are Rogers dependent
> > > > > will not work.
> > > > >
> > > > > Some people are so retarded it is astonishing!
> > > > >
> > > > > --
> > > > > (CAUTION) You are advised that if you attack my person or property,
> > > > > you will be put down in accordance with the provisions of section 34 & 35
> > > > > of the Criminal Code respectively.  If you are brandishing (or in
> > > > > possession) of a weapon then lethal force will be applied to your person in
> > > > > accordance with the law.  This means that your misadventures may end in
> > > > > your death.  Consider yourself cautioned and govern your actions
> > > > > appropriately.
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: NANOG <nanog-bounces+kmedcalf=dessus.com () nanog org> On Behalf
> > > > > Of
> > > > > > Eric Kuhnke
> > > > > > Sent: Friday, 8 July, 2022 13:34
> > > > > > To: jim deleskie <deleskie () gmail com>
> > > > > > Cc: NANOG list <nanog () nanog org>
> > > > > > Subject: Re: Rogers Outage Canada
> > > > > >
> > > > > >
> > > > > > I have seen anecdotal reports that the mobile network is in a half
> > > > > broken
> > > > > > state that phones remain registered to, so a 911 call will attempt and
> > > > > > then fail.
> > > > > >
> > > > > >
> > > > > > This is unlike what would happen if you had a US/Canada cellphone with
> > > > > > battery power but no SIM card in it that would search for any
> > > > > available
> > > > > > network in RF range for a 911 call if needed.
> > > > > >
> > > > > >
> > > > > > On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie () gmail com
> > > > > > <mailto:deleskie () gmail com> > wrote:
> > > > > >
> > > > > >
> > > > > >       i cant see BGP taking out SS7.
> > > > > >
> > > > > >       -jim
> > > > > >
> > > > > >       On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004
> > > > > > <greenjosh6499 () gmail com <mailto:greenjosh6499 () gmail com> > wrote:
> > > > > >
> > > > > >
> > > > > >               According to Cloudflare Radar
> > > > > > <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> ,
> > > > > Rogers
> > > > > > BGP announcements spiked massively to levels 536,777% higher than
> > > > > normal
> > > > > > (343,601 vs 64 normally) just minutes before the outage. I would not
> > > > > be
> > > > > > surprised if this happened to be the culprit.
> > > > > >
> > > > > >               Regards,
> > > > > >               Josh Green
> > > > > >
> > > > > >               On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via
> > > > > NANOG
> > > > > > <nanog () nanog org <mailto:nanog () nanog org> > wrote:
> > > > > >
> > > > > >
> > > > > >                       In the early hours of the morning around 2-3am
> > > > > my modem
> > > > > > got hit with a configuration update that caused a DHCP release that
> > > > > > wasn't renewed for about two hours, after rollback the connection was
> > > > > > fine for 3 hours before this network wide outage.
> > > > > >
> > > > > >
> > > > > >                       Maybe a failed night time update was attempted
> > > > > again
> > > > > > during office hours, I've heard daytime guys are still WFH and night
> > > > > > shift is in building.
> > > > > >
> > > > > >
> > > > > >                       I expect we'll never get a real explanation.
> > > > > Rogers is
> > > > > > notorious for withholding any type of helpful or technical
> > > > > information.
> > > > > >                       Sent from my inoperable Rogers Mobile via
> > > > > emergency eSIM.
> > > > > >                       Regards,
> > > > > >
> > > > > >                       Andrew Paolucci
> > > > > >                       -------- Original Message --------
> > > > > >                       On Jul. 8, 2022, 1:48 p.m., Jay Hennigan <
> > > > > jay () west net
> > > > > > <mailto:jay () west net> > wrote:
> > > > > >
> > > > > >
> > > > > >                               On 7/8/22 07:44, Robert DeVita wrote:
> > > > > > Does anyone
> > > > > > have information on a widespread Rogers outage in Canada. I > have
> > > > > > customers with multiple sites down. There's discussion on the Outages
> > > > > > mailing list. Seems widespread, affecting all services, mobile, voice,
> > > > > > Internet. No cause or ETR posted yet. -- Jay Hennigan - jay () west net
> > > > > > <mailto:jay () west net>  Network Engineering - CCIE #7880 503 897-8550
> > > > > -
> > > > > > WB6RDV
> > > > > >
> > > > > >
> > > > > >
> > > > > >               --
> > > > > >
> > > > > >               Josh Green.
> >
> >
> > --
> > Jon Sands
> > MFI Labshttps://fohdeesha.com/