[Community-Discuss] 06 April 2019 RPKI incident - Postmortem report

[Sunday Folayan]

Hi Cedrick and the team,

Can the certificate generation and update be automated and handled by a
script? I guess alerts when such an update fails will be taken
more seriously.

Can the AfriNIC RPKI-WG be more involved in assuring stability rather than
leave the community to discover and complain?

Just musing.

Good luck with the automation.

Sunday.

On Mon, Apr 8, 2019, 16:46 Cedrick Adrien Mbeyet <cedrick.mbeyet at afrinic.net>
wrote:

> Dear AFRINIC community,
>
>
> Find below postmortem report on the incident that happen on 06 April 2019.
>
>
>
> The AFRINIC RPKI engine has an offline part that has to be renewed on a
> monthly bases. The process is known, documented and automated reminders
> set. The system is set to send 2 reminders each month, one 15 days prior to
> the expiry date and the second one 7 days before expiry. On the 2nd half of
> March, the monitoring system sent a reminder to perform the offline refresh
> but this was not acted upon.
>
>
>
>
>
> On Saturday 06 April 2019,  Certificate revocation List (CRL) and the
> manifest file of AFRINIC RPKI repository expired (around 07:24AM UTC). Our
> monitoring system picked this up. The immediate action was to generate new
> certificates and manifest file and upload them onto RPKI engine system.
>
>
>
> The failure was as a result of human error, no changes were made on the
> system but we have taken additional steps to the existing process to ensure
> that this does not happen again. We do acknowledge that it is unacceptable
> to have such a failure with critical infrastructure and necessary done in
> this regard.
>
>
>
>
> We do apologize for the inconvenience caused and thank you for your
> patience in this regard.
>
>
> --
> _______________________________________________________________
> Cedrick Adrien Mbeyet
> Infrastructure Unit Manager, AFRINIC Ltd.
> t:  +230 403 5100 / 403 5115 | f: +230 466 6758 | tt: @afrinic | w: www.afrinic.netfacebook.com/afrinic | flickr.com/afrinic | youtube.com/afrinicmedia
> ______________________________________________________
>
>
> _______________________________________________
> Community-Discuss mailing list
> Community-Discuss at afrinic.net
> https://lists.afrinic.net/mailman/listinfo/community-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.afrinic.net/pipermail/community-discuss/attachments/20190410/2912a952/attachment.html>