<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<font face="Tahoma">Thanks, Cedrick.<br>
<br>
A question that is, perhaps, obvious... are you able to take the
human component out of this? If 2 reminders were not enough to get
the humans to act, I'm not sure the current methodology is
sustainable.<br>
<br>
Mark.<br>
</font><br>
<div class="moz-cite-prefix">On 8/Apr/19 17:46, Cedrick Adrien
Mbeyet wrote:<br>
</div>
<blockquote type="cite"
cite="mid:536476d1-635c-6c84-eb0c-22e8730afe51@afrinic.net">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p> </p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US">Dear
AFRINIC community,</span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US"><br>
</span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US">Find
below postmortem report on the incident that happen on 06
April 2019. <br>
</span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US"> </span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US">The
AFRINIC RPKI engine has an offline part that has to be renewed
on a monthly bases. The process is known, documented and
automated reminders set. The system is set to send 2 reminders
each month, one 15 days prior to the expiry date and the
second one 7 days before expiry. On the 2nd half of March, the
monitoring system sent a reminder to perform the offline
refresh but this was not acted upon. </span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US"> </span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US"> </span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US">On
Saturday 06 April 2019,<span style="mso-spacerun:yes"> </span>Certificate
revocation List (CRL) and the manifest file of AFRINIC RPKI
repository expired (around 07:24AM UTC). Our monitoring system
picked this up. The immediate action was to generate new
certificates and manifest file and upload them onto RPKI
engine system.</span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US"> </span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US">The
failure was as a result of human error, no changes were made
on the system but we have taken additional steps to the
existing process to ensure that this does not happen again. We
do acknowledge that it is unacceptable to have such a failure
with critical infrastructure and necessary done in this
regard. </span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US"> </span></p>
<p class="MsoNormal"><span style="mso-ansi-language:EN-US"> </span></p>
<span
style="font-size:12.0pt;font-family:"Calibri",sans-serif;mso-ascii-theme-font:minor-latin;mso-fareast-font-family:Calibri;mso-fareast-theme-font:minor-latin;
mso-hansi-theme-font:minor-latin;mso-bidi-font-family:"Times New
Roman";
mso-bidi-theme-font:minor-bidi;mso-ansi-language:EN-US;mso-fareast-language:
EN-US;mso-bidi-language:AR-SA">We do apologize for the
inconvenience caused and thank you for your patience in this
regard.</span>
<p>
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:variable;
mso-font-signature:-536870145 1107305727 0 0 415 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;
mso-font-charset:0;
mso-generic-font-family:swiss;
mso-font-pitch:variable;
mso-font-signature:-536859905 -1073732485 9 0 511 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-ansi-language:EN-GB;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-ansi-language:EN-GB;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;
mso-header-margin:36.0pt;
mso-footer-margin:36.0pt;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
-->
</style></p>
<pre class="moz-signature" cols="72">--
_______________________________________________________________
Cedrick Adrien Mbeyet
Infrastructure Unit Manager, AFRINIC Ltd.
t: +230 403 5100 / 403 5115 | f: +230 466 6758 | tt: @afrinic | w: <a class="moz-txt-link-abbreviated" href="http://www.afrinic.net" moz-do-not-send="true">www.afrinic.net</a>
facebook.com/afrinic | flickr.com/afrinic | youtube.com/afrinicmedia
______________________________________________________
</pre>
</blockquote>
<br>
</body>
</html>