[Community-Discuss] 06 April 2019 RPKI incident - Postmortem report

[Owen DeLong]

If you automate the process, you have to store the private key in a manner in which it can be accessed automatically.

This compromises the integrity of the key as it must be stored online (or be usable through an on-line process)  rather than being kept offline and utilized via an HSM or other secure process.

Owen


> On Apr 10, 2019, at 3:34 AM, Sunday Folayan <sfolayan at gmail.com> wrote:
> 
> Hi Cedrick and the team,
> 
> Can the certificate generation and update be automated and handled by a script? I guess alerts when such an update fails will be taken more seriously.
> 
> Can the AfriNIC RPKI-WG be more involved in assuring stability rather than leave the community to discover and complain?
> 
> Just musing.
> 
> Good luck with the automation.
> 
> Sunday.
> 
> On Mon, Apr 8, 2019, 16:46 Cedrick Adrien Mbeyet <cedrick.mbeyet at afrinic.net <mailto:cedrick.mbeyet at afrinic.net>> wrote:
> 
> Dear AFRINIC community,
> 
> 
> 
> Find below postmortem report on the incident that happen on 06 April 2019. 
> 
>  
> The AFRINIC RPKI engine has an offline part that has to be renewed on a monthly bases. The process is known, documented and automated reminders set. The system is set to send 2 reminders each month, one 15 days prior to the expiry date and the second one 7 days before expiry. On the 2nd half of March, the monitoring system sent a reminder to perform the offline refresh but this was not acted upon.
> 
>  
>  
> On Saturday 06 April 2019,  Certificate revocation List (CRL) and the manifest file of AFRINIC RPKI repository expired (around 07:24AM UTC). Our monitoring system picked this up. The immediate action was to generate new certificates and manifest file and upload them onto RPKI engine system.
> 
>  
> The failure was as a result of human error, no changes were made on the system but we have taken additional steps to the existing process to ensure that this does not happen again. We do acknowledge that it is unacceptable to have such a failure with critical infrastructure and necessary done in this regard.
> 
>  
>  
> We do apologize for the inconvenience caused and thank you for your patience in this regard.
> 
> -- 
> _______________________________________________________________
> Cedrick Adrien Mbeyet                                           
> Infrastructure Unit Manager, AFRINIC Ltd.
> t:  +230 403 5100 / 403 5115 | f: +230 466 6758 | tt: @afrinic | w: www.afrinic.net <http://www.afrinic.net/>
> facebook.com/afrinic <http://facebook.com/afrinic> | flickr.com/afrinic <http://flickr.com/afrinic> | youtube.com/afrinicmedia <http://youtube.com/afrinicmedia>
> ______________________________________________________
> 
> _______________________________________________
> Community-Discuss mailing list
> Community-Discuss at afrinic.net <mailto:Community-Discuss at afrinic.net>
> https://lists.afrinic.net/mailman/listinfo/community-discuss <https://lists.afrinic.net/mailman/listinfo/community-discuss>
> _______________________________________________
> Community-Discuss mailing list
> Community-Discuss at afrinic.net
> https://lists.afrinic.net/mailman/listinfo/community-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.afrinic.net/pipermail/community-discuss/attachments/20190410/fff511bd/attachment.html>