Re: afrinic rpki issue

[Carlos Friaças via NANOG <nanog () nanog org>]

On Wed, 14 Jun 2023, Alex Band wrote:

> Hi Carlos,

Hi Alex, All,


> Because of the issues that AfriNIC is facing, they are forcing all 
> traffic from HTTPS to rsync, so you should check if rsync can properly 
> set up outbound connections from your machine. What?s the output you get 
> when you rsync rsync://rpki.afrinic.net/repository/ ?


drwxr-xr-x          4,096 2023/06/14 12:04:28 .
-rw-r--r--            496 2020/04/08 19:58:03 AfriNIC-simple.tal
-rw-r--r--          1,216 2020/03/30 13:00:32 AfriNIC.cer
drwxr-xr-x          4,096 2023/06/09 13:50:13 
04E8B0D80F4D11E0B657D8931367AE7D
drwxr-xr-x         32,768 2023/06/14 12:04:28 afrinic
drwxr-xr-x          4,096 2023/06/14 01:05:30 apnic
drwxr-xr-x          8,192 2023/06/14 11:42:38 arin
drwxr-xr-x            120 2023/06/14 01:15:32 lacnic
drwxr-xr-x         16,384 2023/06/14 12:04:01 member_repository
drwxr-xr-x          4,096 2023/06/14 01:20:30 ripe


Seems to be working...



> I do an interactive Routinator validation run with debug logging enabled, like so:
>
> $ routinator -vv vrps -f summary
>
> Then I see the following in the logs:
>
> [WARN] RRDP https://rrdp.afrinic.net/notification.xml: Getting notification file failed with status 204 No Content
> [INFO] RRDP https://rrdp.afrinic.net/notification.xml: Update failed and current copy is expired since 2023-05-30 
> 10:43:44 UTC.
> [INFO] RRDP https://rrdp.afrinic.net/notification.xml: Falling back to rsync.
> [INFO] rsyncing from rsync://rpki.afrinic.net/repository/.


Found valid trust anchor https://rpki.afrinic.net/repository/AfriNIC.cer. 
Processing.
RRDP https://rrdp.afrinic.net/notification.xml: Updating server
RRDP https://rrdp.afrinic.net/notification.xml: malformed XML
rsync://rpki.afrinic.net/repository/04E8B0D80F4D11E0B657D8931367AE7D/62gPOPXWxxu0sQa4vQZYUBLaMbY.mft: 
failed to validate
CA for 
rsync://rpki.afrinic.net/repository/04E8B0D80F4D11E0B657D8931367AE7D/ 
rejected, resources marked as unsafe:
   0.0.0.0/0
   ::/0
   AS0-AS4294967295





> Then, rsyncing the contents works just fine; objects are fetched and validated. Some objects fail validation with 
> "certificate is not yet valid.?, "certificate has been revoked.? and ?Object not found.? but that appears unrelated to 
> the connectivity issues they?re facing.
>
> I end up with the following totals:
>
> Summary at 2023-06-14 13:43:24.366013 UTC
> afrinic:              ROAs:    5756 verified;
>            VRPs:    7121 verified,    6820 final;
>    router certs:       0 verified;
>     router keys:       0 verified,       0 final.
>           ASPAs:       0 verified,       0 final.

Where do you see this?
Command output?


Summary at 2023-06-14 14:11:34.413948850 UTC
ripe: 39230 verified ROAs, 212122 verified VRPs, 6 unsafe VRPs, 212117 
final VRPs.
apnic: 24878 verified ROAs, 111967 verified VRPs, 0 unsafe VRPs, 111699 
final VRPs.
arin: 64077 verified ROAs, 79176 verified VRPs, 0 unsafe VRPs, 78064 final 
VRPs.
lacnic: 17966 verified ROAs, 32624 verified VRPs, 5 unsafe VRPs, 31033 
final VRPs.
afrinic: 0 verified ROAs, 0 verified VRPs, 0 unsafe VRPs, 0 final VRPs.
total: 146151 verified ROAs, 435889 verified VRPs, 11 unsafe VRPs, 432913 
final VRPs.



> If you want some logs to compare, you can have a look here:
> https://routinator.do.nlnetlabs.nl/log

Thanks.


> It all still works without any extra configuration in Routinator.

Well, for me it's still not really working yet.... :-)

Thanks anyway.


Cheers,
Carlos


> Cheers,
>
> Alex
>
>
>
> > On 14 Jun 2023, at 15:15, Carlos Friaças via NANOG <nanog () nanog org> wrote:
> >
> >
> > Hi All,
> >
> > Did this issue resurface some days ago...?
> > I had nearly 6000 ROAs on June 1st.
> > That went to ZERO on June 2nd.
> >
> > I'm using routinator. Should i have changed something in my config to accomodate for some change?
> >
> > Best Regards,
> > Carlos
> >
> >
> >
> > On Sun, 20 Nov 2022, Cedrick Adrien Mbeyet wrote:
> >
> > > Hi Job,
> > > Thank you for this good analysis and for sharing your findings.
> > > The issue has since been fixed and the team will publish a post-mortem accordingly once we are done with making sure 
> > > the issue will not
> > > reappear.
> > > Your recommendation is well noted and I cc my colleague so that they can take that into consideration in our 
> > > improvement roadmap.
> > > Best regards,
> > > ==============================
> > > Cedrick Adrien MBEYET
> > > Ebene Cybercity, Mauritius
> > > +230 5851 7674
> > > +++ Never give up, Keep moving forward +++
> > > On Sun, Nov 20, 2022 at 3:49 PM Job Snijders via NANOG <nanog () nanog org> wrote:
> > >      Hi all,
> > >
> > >      It appears PacketVis correctly identified an issue.
> > >
> > >      AFRINIC's self-signed root AfriNIC.cer [1] points via its SIA to
> > >      'afrinic-ca.cer' [2] which in turn references a RPKI Manifest named
> > >      'K1eJenypZMPIt_e92qek2jSpj4A.mft'.
> > >
> > >      The K1eJenypZMPIt_e92qek2jSpj4A Manifest lists 499 Certificate
> > >      Authorities. This Manifest represents the demarcation point between
> > >      "Afrinic as root CA operator" and "Afrinic hosting rpki on behalf of its
> > >      members". In other words; this is an important top-level Manifest in the
> > >      critical path towards the ROAs of the Afrinic members.
> > >
> > >      There was a ~ 7 hour gap in the validity window of this Manifest and its
> > >      companion CRL (from 20221120T000311Z until 20221120T071514Z). The
> > >      serials 1E19 and 1E1A (respectively 12B2 and 12B3) are successive.
> > >
> > >      rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.crl
> > >          CRL Serial Number:        1E19
> > >          CRL valid since:          Nov 18 00:03:11 2022 GMT
> > >          CRL valid until:          Nov 20 00:03:11 2022 GMT
> > >
> > >          CRL Serial Number:        1E1A
> > >          CRL valid since:          Nov 20 07:15:12 2022 GMT
> > >          CRL valid until:          Nov 22 07:15:12 2022 GMT
> > >
> > >      rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.mft
> > >          Manifest Number:          12B2
> > >          Manifest valid since:     Nov 18 00:03:13 2022 GMT
> > >          Manifest valid until:     Nov 20 00:03:13 2022 GMT
> > >
> > >          Manifest Number:          12B3
> > >          Manifest valid since:     Nov 20 07:15:14 2022 GMT
> > >          Manifest valid until:     Nov 22 07:15:14 2022 GMT
> > >
> > >      (The above can be reconstructed using archives from http://www.rpkiviews.org)
> > >
> > >      The rcynic validator hosted at Afrinic also noticed a gap in objects:
> > >      https://validator.afrinic.net/rpki/rcynic/rpki.afrinic.net_week_svg.html
> > >
> > >      A possible recommendation might be to increase the validity window of
> > >      these two objects from a sliding 48-hour window to a 1 or 2 week window.
> > >      This way any stalling in the issuance process wouldn't case operational
> > >      issues on the weekend.
> > >
> > >      Kind regards,
> > >
> > >      Job
> > >
> > >      [1]: SKI EB:68:0F:38:F5:D6:C7:1B:B4:B1:06:B8:BD:06:58:50:12:DA:31:B6
> > >      [2]: SKI 2B:57:89:7A:7C:A9:64:C3:C8:B7:F7:BD:DA:A7:A4:DA:34:A9:8F:80
> > >
> > >      On Sat, Nov 19, 2022 at 08:36:23PM -0800, Randy Bush wrote:
> > >     > From: PacketVis <notifications () packetvis com>
> > >     > Date: Sun, 20 Nov 2022 04:30:44 +0000
> > >     >
> > >     > Possible TA malfunction or incomplete VRP file: 73.95% of the ROAs disappeared from afrinic
> > >     >
> > >     > See more details about the event:
> > >     
> > > >https://packetvis.com/#/bgp/event/905ec8b7d37e89a2d7b547bca99fd57e-372b0bf3-9056-407e-9e8d-e986567155fc/4f309cb51ba9314fafa64da53d007e342fac
> > >      a613