<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="font-family: 'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif; font-size: 14px;">
<div class="hiri-body-wrapper" contenteditable="true">
<div>Hi Owen,</div>
</div>
<div class="hiri-extra-edited" contenteditable="true">
<p>On 2019-04-10 15:00:28+02:00 Owen DeLong wrote:</p>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div>
<blockquote type="cite">
<div>On Apr 10, 2019, at 3:57 AM, Ben Maddison via Community-Discuss <<a href="mailto:community-discuss@afrinic.net">community-discuss@afrinic.net</a>> wrote:</div>
<div>
<div style="font-family: 'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif; font-size: 14px;">
<div class="hiri-body-wrapper" applecontenteditable="true">
<div>Hi all,</div>
</div>
<div class="hiri-extra-edited" applecontenteditable="true" contenteditable="true">
<p>On 2019-04-10 12:10:22+02:00 Noah wrote:</p>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div dir="auto">+1 and Ack @saul</div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 10 Apr 2019, 12:57 Saul Stein, <<a href="mailto:saul@enetworks.co.za">saul@enetworks.co.za</a>> wrote:</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-ZA" link="blue" vlink="purple">
<div class="m_-4592790368224664121WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1f497d">Agreed.</span></p>
<div><span style="font-size:11.0pt;color:#1f497d"> </span></div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1f497d">There is a bigger issue at stake here: I have yet to see any evidence that AFRINIC takes RPKI seriously.</span></p>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
Until relatively recently, this attitude may have been understandable, since the RPKI was largely a curiosity with almost no impact on operations.<br>
This is no longer the case, and all of the RIRs have serious work to do to improve operations in this area. This is clearly the case in this region.</div>
</div>
</div>
</blockquote>
<div></div>
Absent a badly flawed implementation, there’s no serious consequence to an RPKI outage… It merely reverts routing back to it’s previous unauthenticated state.</div>
<div></div>
</div>
</blockquote>
<br>
That's not entirely true. A partial outage (where for example a single TAL becomes unverifiable, as in this case) may lead to a missing ROA for a prefix that remains covered by other ROAs issued under other TALs.<br>
<br>
Consider ROAs:<br>
{prefix: 2001:db8::/32, maxLength: 48, asn: 65000, tal: AFRINIC}<br>
{prefix: 2001:db8:f00::/48, maxLength: 48, asn: 65001, tal: RIPE}<br>
<br>
With the above, a route 2001:db8:f00::/48 via 65000_65001 will have a status Valid.<br>
If the RIPE TAL fails verification, it will become Invalid.<br>
<br>
This is most certainly a corner case, but is at least theoretically possible given that all RIRs claim 0/0 in their root certs.<br>
<br>
A more likely scenario is that an existing mis-origination that is being dropped as Invalid suddenly becomes Not Found, and wins path selection, thereby misdirecting traffic.<br>
<br>
I'm not aware of any such cases on our network from this last outage, but it's possible that they went undetected. The likelihood of this case increases substantially as more operators begin to drop Invalids.<br>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div></div>
<div>I’m not convinced that all of the RIRs have serious work to do here. I think some of the RIRs have stable, reliable operations in this regard. I’m not yet convinced that the level of stability in AfriNIC operations here is impactful, let alone seriously
impactful to operations.</div>
</div>
</blockquote>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div><span style="font-family: "Segoe UI", Frutiger, "Frutiger Linotype", "Dejavu Sans", "Helvetica Neue", Arial, sans-serif; font-size: 14px;"> </span>
<blockquote type="cite">
<div style="font-family: 'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif; font-size: 14px;">
<div class="hiri-extra-edited" applecontenteditable="true" contenteditable="true">
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-ZA" link="blue" vlink="purple">
<div class="m_-4592790368224664121WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1f497d">The last issue I had, when no ROAs could be added, deleted etc, it was admitted that the issue was known about for over two weeks without anything on the announce list or being fixed! After escalation
to the CEO and others it was fixed in a couple of hours!</span></p>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
As an operator community, we need to have a serious conversation about what we expect from afrinic (and the other RIRs). 24x7 availability comes with a price tag, as everyone on this list should be all too aware.</div>
</div>
</blockquote>
<div></div>
Any availability comes with a price tag. The higher the level of availability, the higher the price tag.</div>
<div>
<blockquote type="cite">
<div style="font-family: 'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif; font-size: 14px;">
<div class="hiri-extra-edited" applecontenteditable="true" contenteditable="true">
It is quite clear however, both from recent experience and from the postmortem below, that the current system is unfit for purpose.</div>
</div>
</blockquote>
<div></div>
Is it? I’m unconvinced at this time…</div>
</div>
</blockquote>
<br>
A system, whether human or computerised, that fails to mitigate an impending and predictable failure, and then takes several hours to correct it is, in my book the very definition of unfit for purpose.<br>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div></div>
<div>
<blockquote type="cite">
<div style="font-family: 'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif; font-size: 14px;">
<div class="hiri-extra-edited" applecontenteditable="true" contenteditable="true">
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-ZA" link="blue" vlink="purple">
<div class="m_-4592790368224664121WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1f497d">RPKI is serious and needs to be taken seriously. We can’t continuously be having issues with it. It is like customs at immigration being offline!</span></p>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
RPKI is operational. I’m not sure how serious it is, as I have trouble taking seriously a system which, at best, tells you what you need to prepend. It’s a nice protection from fat fingers, but, in its current state, it provides little to no protection beyond
that for anyone but the largest operators.</div>
</div>
</blockquote>
<br>
Becoming bestpath in a densely-interconnected network using a forged-origin hijack in the face of a ROA that has all it's covered prefixes in the DFZ is actually not trivial, and often not possible, because you loose on path-length.<br>
<br>
This is even more true in networks that filter peers and customers by prefix, as you have less chance of exploiting a higher local-pref to overcome path-length.<br>
As a result, the protection that OV offers is not meaningless.<br>
<br>
OV is also a prerequisite for validating the entire path against stated policy, various mechanisms for which are currently WIP in SIDROPS and GROW.<br>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div></div>
<div></div>
<div>Nonetheless, even if one wants to take RPKI seriously, a quick review of the RFCs and IETF guidance on the matter shows that the worst case scenario for an RIR outage on ROA publication should be that routing reverts to its pre-RPKI unauthenticated state.
It should not cause any sort of outage (except to the extent you might start accepting routes you previously rejected).</div>
<div></div>
<div>If you’re rejecting routes for RPKI validation failure, you should be tracking down the advertisers and getting those situations corrected. If you’re doing that, then any such outages should be somewhere between minimal and non-existent.</div>
<div></div>
</div>
</blockquote>
If issuing party tools are unavailable to resource holders, they will be unable to effect the correction.<br>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div></div>
<div>Did any packets go the wrong way due to the AfriNIC outage? Was there any actual operational impact?</div>
<div></div>
<div>I suspect not. I suspect that this is a lot of handwaving about a non-issue.</div>
<div></div>
</div>
</blockquote>
<br>
I suspect some, though certainly few. The same incident at a time when more networks are performing OV could look very different, and I'm simply suggesting that we look closely at the options before that happens.<br>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div></div>
<div>Don’t get me wrong, I’m all for making AfriNIC’s systems more resilient and more available, but, I think we also need to consider the actual impact of failures and not over-react to failures without impact.</div>
<div></div>
<div>Based on the information in the post mortem, it does not look like a systems failure, but purely human error. Taking the humans out of the loop on that monthly maintenance would involve compromising the integrity of the private key and thus reduce the
validity of the RPKI data. As such, I’m not convinced that there is a problem here to solve beyond the procedural changes that AfriNIC says they have already implemented.</div>
<div></div>
</div>
</blockquote>
<br>
I'm not arguing for any specific change. I would point out that when there are humans in the system, then a human failure *is* a system failure, and we should be clear on what failure modes can and cannot be tolerated.<br>
<br>
Adding rpki-discuss, so that we can continue there...<br>
<br>
Cheers,<br>
<br>
Ben<br>
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div></div>
<div>Owen</div>
<div></div>
<div></div>
<div>
<blockquote type="cite">
<div style="font-family: 'Segoe UI',Frutiger,'Frutiger Linotype','Dejavu Sans','Helvetica Neue',Arial,sans-serif; font-size: 14px;">
<div class="hiri-extra-edited" applecontenteditable="true" contenteditable="true">
<blockquote style="padding-left:10px; border-left:1px solid #ccc; margin:0">
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-ZA" link="blue" vlink="purple">
<div class="m_-4592790368224664121WordSection1">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;color:windowtext">From:</span></b><span lang="EN-US" style="font-size:11.0pt;color:windowtext"> Mark Tinka [mailto:<a target="_blank" rel="noreferrer" href="mailto:mark.tinka@seacom.mu">mark.tinka@seacom.mu</a>]<br>
<b>Sent:</b> 10 April 2019 08:32 AM<br>
<b>To:</b> <a target="_blank" rel="noreferrer" href="mailto:community-discuss@afrinic.net">
community-discuss@afrinic.net</a><br>
<b>Subject:</b> Re: [Community-Discuss] 06 April 2019 RPKI incident - Postmortem report</span></p>
</div>
</div>
<div><span lang="EN-GB"> </span></div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-GB" style="font-family:"Tahoma",sans-serif">Thanks, Cedrick.<br>
<br>
A question that is, perhaps, obvious... are you able to take the human component out of this? If 2 reminders were not enough to get the humans to act, I'm not sure the current methodology is sustainable.<br>
<br>
Mark.</span></p>
<div>
<p class="MsoNormal"><span lang="EN-GB">On 8/Apr/19 17:46, Cedrick Adrien Mbeyet wrote:</span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US">Dear AFRINIC community,</span></p>
<div><span lang="EN-GB"> </span></div>
<p class="MsoNormal"><span lang="EN-US">Find below postmortem report on the incident that happen on 06 April 2019.
</span></p>
<div><span lang="EN-US"> </span></div>
<p class="MsoNormal"><span lang="EN-US">The AFRINIC RPKI engine has an offline part that has to be renewed on a monthly bases. The process is known, documented and automated reminders set. The system is set to send 2 reminders each month, one 15 days prior
to the expiry date and the second one 7 days before expiry. On the 2nd half of March, the monitoring system sent a reminder to perform the offline refresh but this was not acted upon.
</span></p>
<div><span lang="EN-US"> </span></div>
<div><span lang="EN-US"> </span></div>
<p class="MsoNormal"><span lang="EN-US">On Saturday 06 April 2019, Certificate revocation List (CRL) and the manifest file of AFRINIC RPKI repository expired (around 07:24AM UTC). Our monitoring system picked this up. The immediate action was to generate new
certificates and manifest file and upload them onto RPKI engine system.</span></p>
<div><span lang="EN-US"> </span></div>
<p class="MsoNormal"><span lang="EN-US">The failure was as a result of human error, no changes were made on the system but we have taken additional steps to the existing process to ensure that this does not happen again. We do acknowledge that it is unacceptable
to have such a failure with critical infrastructure and necessary done in this regard.
</span></p>
<div><span lang="EN-US"> </span></div>
<div><span lang="EN-US"> </span></div>
<p class="MsoNormal"><span lang="EN-US">We do apologize for the inconvenience caused and thank you for your patience in this regard.</span><span lang="EN-US">
</span></p>
<pre>-- </pre>
<pre>_______________________________________________________________</pre>
<pre>Cedrick Adrien Mbeyet </pre>
<pre>Infrastructure Unit Manager, AFRINIC Ltd.</pre>
<pre>t: +230 403 5100 / 403 5115 | f: +230 466 6758 | tt: @afrinic | w: <a target="_blank" rel="noreferrer" href="http://www.afrinic.net/">www.afrinic.net</a></pre>
<pre><a target="_blank" rel="noreferrer" href="http://facebook.com/afrinic">facebook.com/afrinic</a> | <a target="_blank" rel="noreferrer" href="http://flickr.com/afrinic">flickr.com/afrinic</a> | <a target="_blank" rel="noreferrer" href="http://youtube.com/afrinicmedia">youtube.com/afrinicmedia</a></pre>
<pre>______________________________________________________</pre>
<pre>
</pre>
</blockquote>
<div><span style="font-family:"Times New Roman",serif"> </span></div>
</div>
</div>
_______________________________________________<br>
Community-Discuss mailing list<br>
<a target="_blank" rel="noreferrer" href="mailto:Community-Discuss@afrinic.net">Community-Discuss@afrinic.net</a><br>
<a rel="noreferrer noreferrer" target="_blank" href="https://lists.afrinic.net/mailman/listinfo/community-discuss">https://lists.afrinic.net/mailman/listinfo/community-discuss</a></blockquote>
</div>
</div>
</blockquote>
</div>
</div>
_______________________________________________<br>
Community-Discuss mailing list<br>
<a href="mailto:Community-Discuss@afrinic.net">Community-Discuss@afrinic.net</a><br>
https://lists.afrinic.net/mailman/listinfo/community-discuss</blockquote>
</div>
</div>
</blockquote>
</div>
</body>
</html>