<div dir="auto">Hi Cedrick and the team,<div dir="auto"><br></div><div dir="auto">Can the certificate generation and update be automated and handled by a script? I guess alerts when such an update fails will be taken more seriously.</div><div dir="auto"><br></div><div dir="auto">Can the AfriNIC RPKI-WG be more involved in assuring stability rather than leave the community to discover and complain?</div><div dir="auto"><br></div><div dir="auto">Just musing.</div><div dir="auto"><br></div><div dir="auto">Good luck with the automation.</div><div dir="auto"><br></div><div dir="auto">Sunday.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 8, 2019, 16:46 Cedrick Adrien Mbeyet <<a href="mailto:cedrick.mbeyet@afrinic.net">cedrick.mbeyet@afrinic.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>
</p>
<p class="MsoNormal"><span>Dear
AFRINIC community,</span></p>
<p class="MsoNormal"><span><br>
</span></p>
<p class="MsoNormal"><span>Find
below postmortem report on the incident that happen on 06 April
2019. <br>
</span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal"><span>The
AFRINIC RPKI
engine has an offline part that has to be renewed on a monthly
bases. The
process is known, documented and automated reminders set. The
system is set to
send 2 reminders each month, one 15 days prior to the expiry
date and the
second one 7 days before expiry. On the 2nd half of March, the
monitoring
system sent a reminder to perform the offline refresh but this
was not acted
upon. </span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal"><span>On
Saturday 06 April
2019,<span> </span>Certificate
revocation List (CRL)
and the manifest file of AFRINIC RPKI repository expired (around
07:24AM UTC).
Our monitoring system picked this up. The immediate action was
to generate new
certificates and manifest file and upload them onto RPKI engine
system.</span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal"><span>The
failure was as a
result of human error, no changes were made on the system but we
have taken
additional steps to the existing process to ensure that this
does not happen
again. We do acknowledge that it is unacceptable to have such a
failure with
critical infrastructure and necessary done in this regard. </span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal"><span> </span></p>
<span>We do apologize for the
inconvenience caused and
thank you for your patience in this regard.</span>
<p>
</p>
<pre class="m_9116748212214592339moz-signature" cols="72">
--
_______________________________________________________________
Cedrick Adrien Mbeyet
Infrastructure Unit Manager, AFRINIC Ltd.
t: +230 403 5100 / 403 5115 | f: +230 466 6758 | tt: @afrinic | w: <a class="m_9116748212214592339moz-txt-link-abbreviated" href="http://www.afrinic.net" target="_blank" rel="noreferrer">www.afrinic.net</a>
<a href="http://facebook.com/afrinic" target="_blank" rel="noreferrer">facebook.com/afrinic</a> | <a href="http://flickr.com/afrinic" target="_blank" rel="noreferrer">flickr.com/afrinic</a> | <a href="http://youtube.com/afrinicmedia" target="_blank" rel="noreferrer">youtube.com/afrinicmedia</a>
______________________________________________________
</pre>
</div>
_______________________________________________<br>
Community-Discuss mailing list<br>
<a href="mailto:Community-Discuss@afrinic.net" target="_blank" rel="noreferrer">Community-Discuss@afrinic.net</a><br>
<a href="https://lists.afrinic.net/mailman/listinfo/community-discuss" rel="noreferrer noreferrer" target="_blank">https://lists.afrinic.net/mailman/listinfo/community-discuss</a><br>
</blockquote></div>