[DBWG] AFRINIC RPKI VRP graph for November 2023 - heavy fluctuations affecting 2 members

[Job Snijders]

Dear AfriNIC,

It appears there frequently are publication point issues throughout
November 2023:

    https://sobornost.net/~job/afrinic-vrps_htm_bd0e1e1af38645c.png

The above chart shows the number of Validated ROA Payloads (VRPs)
plotted against the number of 'Digest Errors' observed over the same
period of time (starting at November 1st 2023 until now).

A 'Digest Error' happens when the computed hash value of a file listed
on a RPKI Manifest does not match the hash value contained in the
Manifest. See https://www.rfc-editor.org/rfc/rfc9286.html#section-6.5

An increase in Digest Errors correlates to a reduction of VRPs (meaning,
most if not all prefixes belonging to creator of the ROA will change
validation state from valid/invalid to 'not-found')

I suspect that this problem originates in non-atomic/inconsistent
publications within the RSYNC, these inconsistencies in turn propagated
into the the RRDP sessions. For example, inconsistencies can arise when
a new Manifest is published before the associated changed/added ROAs
have also been moved into place in an atomic fashion.

I associated the errors to manifests, and counted the errors, and
counted the number of files listed on those Manifests:

 Errors  File count  Manifest path
6801063        2945  rpki.afrinic.net/repository/member_repository/F368F2D0/7F4A98EA6E0511E89C0D6E4BF8AEA228/JdY-COq-fPpnhdTB1tNBFt4Vs9w.mft
2787492        2252  rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/eX2I2BPiD_-YLMdBnpabrqa_1ps.mft
  40090         298  rpki.afrinic.net/repository/member_repository/F3682B65/4E851E34DB5511E885B29951F8AEA228/CUDmOCgzNt5cjFpApMI7NPP5ylM.mft
      0           *  all other manifests

So despite heavy fluctuations when looking at the overall VRP numbers,
this problem really only seems to affect 3 CARepositories, belonging to
2 AFRINIC members: IPXO and Cloud Innovation.

To me it appears that the larger the Manifest is, the higher the chances
of a mispublication, and I theorize that the larger the CA, the higher
the chances that this CA frequently adds or updates ROAs.

Kind regards,

Job