Re: AS37468 (Angola Cables) - Route Leak?

[Frederico Oliveira <clayberf.oliveira () gmail com>]

Hi everyone,

I'm Frederico Oliveira, Head of Engineering at Angola Cables.

Let me explain the root cause:

We had an issue into our routing automatic tool, which daily performs some changes into our VRFs. During this activity, 
we had an anomalous situation and the routes were leaked between the VRFs(Peering and Transit). 

Our sincere apologies for the incovenience, please make sure we are working to avoid this kind of situation.

Att,
Frederico Oliveira

> On 25 May 2023, at 13:23, Phil Lavin via NANOG <nanog () nanog org> wrote:
>
> Thanks for the efforts, Mark.
>
> Tricky for me to check as the one known/testable traffic path was fixed by AWS around 12:27 UTC - they rejected 
> prefixes from Angola Cables in LAX and Singapore.
>
> That said, I see from the Kentik folks that it appears to have been resolved at 12:42 UTC: 
> https://twitter.com/DougMadory/status/1661723901519380481
>
>
> > On 25 May 2023, at 17:13, Mark Tinka <mark@tinka.africa <mailto:mark@tinka.africa>> wrote:
> >
> > Angola Cable say they have fixed the issue.
> >
> > Can you please check again and verify? Thanks.
> >
> > Mark.
> >
> > On 5/25/23 14:36, Phil Lavin wrote:
> > > Thanks, Mark
> > >
> > >
> > >
> > > > On 25 May 2023, at 13:34, Mark Tinka <mark@tinka.africa <mailto:mark@tinka.africa>> wrote:
> > > >
> > > > I've reached out to some known folk at Angola Cables. Will let you know if I hear back.
> > > >
> > > > Mark.
> > > >
> > > > On 5/25/23 14:01, Phil Lavin via NANOG wrote:
> > > > > Cross-posting from outages list:
> > > > >
> > > > > Hey Folks,
> > > > >
> > > > > Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69).
> > > > >
> > > > > Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.
> > > > >
> > > > > Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX.
> > > > >
> > > > > I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.
> > > > >
> > > > >                            My traceroute  [v0.94]
> > > > > jcasc-rtc01 (10.36.21.252) -> phil.lavin.me.uk <http://lavin.me.uk/>      2023-05-25T11:50:13+0000
> > > > > Keys:  Help   Display mode   Restart statistics   Order of fields   quit
> > > > >                                     Packets               Pings
> > > > >  Host                             Loss%   Snt   Last   Avg  Best  Wrst StDev
> > > > >  1. ip-10-36-17-199.ap-southeast-  0.0%    18    0.1   0.6   0.1   7.7   1.8
> > > > >  2. ec2-18-141-171-15.ap-southeas  0.0%    18    4.3  17.5   0.9  95.7  24.0
> > > > >     ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com 
> > > > > <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com/>
> > > > >  3. (waiting for reply)
> > > > >  4. (waiting for reply)
> > > > >  5. (waiting for reply)
> > > > >  6. (waiting for reply)
> > > > >  7. (waiting for reply)
> > > > >  8. 100.65.10.161                  0.0%    17    0.6   1.4   0.3   9.6   2.4
> > > > >     100.65.11.65
> > > > >  9. 150.222.108.75                 0.0%    17    2.1   3.0   1.0  13.1   3.3
> > > > >     52.93.10.74
> > > > > 10. 150.222.108.82                 0.0%    17    2.2   2.4   1.3  10.2   2.1
> > > > > 11. 52.93.11.115                   0.0%    17    5.8   2.4   1.2   9.7   2.5
> > > > >     52.93.10.185
> > > > > 12. 37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com/>      68.8%    17  268.6 279.0 262.4 312.8  19.8
> > > > > 13. 102.219.127.3                 93.8%    17  420.0 420.0 420.0 420.0   0.0
> > > > > 14. (waiting for reply)
> > > > > 15. 195.66.227.209                62.5%    17  467.9 456.9 440.3 488.2  18.5
> > > > > 16. core6.par.hetzner.com <http://core6.par.hetzner.com/>         75.0%    17  455.6 464.1 455.6 486.3  14.8
> > > > > 17. core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com/>       75.0%    17  459.5 467.5 445.9 497.3  21.8
> > > > > 18. core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com/>       68.8%    17  478.4 441.0 390.0 478.4  32.0
> > > > > 19. ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com/>    87.5%    17  460.4 454.2 448.1 460.4   8.7
> > > > > 20. vmh02.lavin.me.uk <http://vmh02.lavin.me.uk/>             75.0%    17  433.3 424.2 382.7 441.9  27.9
> > > > > 21. (waiting for reply)
> > > > > 22. http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk/>        75.0%    17  436.8 438.6 385.0 493.3  44.2
> > > > >
> > > > > I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks?
> > > > >
> > > > >
> > > > > Phil