[rpd] Cloud Innovation Displays Very Poor, If Not Criminal, Netizenship

[Paschal Ochang]

This sounds like a personal attack in my opinion instead of a justified
accusation. Why is this topic allowed to be discussed here when it’s rather
irrelevant?. The parties involved have tendered formal apologies as
reflected in their letters which is a sign of admittance and promotes the
intent of good netizenship. It’s just irrelevant.

On Monday, May 25, 2020, Arnaud AMELINA <amelnaud at gmail.com> wrote:


> Hello, community

>

> +1 @Gregoire and @Mark Tinka

>

> *cloud innovation*  were allocated  *big bunch of IPv4* space as a *LIR*

> with *no ASN*. Interesting, and no *v6*

>

> While the bylaws defines LIR as followed:

> ++++++

> Local Internet Registry (LIR):

> any Network Operator that provides Internet services to distinct end-users

> and end-sites

> ++++++

>

> I wonder  which network does cloud innovation operate and which internet

> services it provides to end-users and end-sites in *Africa*.

>

> How does this network *managing 3 x /11 of IPv4* *operate*?

>

> There is something here for the community to learn about.

>

> Regards

>

> --

> Arnaud

>

> Le sam. 23 mai 2020 à 14:20, Gregoire EHOUMI via RPD <rpd at afrinic.net> a

> écrit :

>

>> Hello,

>>

>> Thanks Mark for exposing the details of the SEACOM AS37353 hijacking.

>>

>> I carefully read your report and also the Cloud Innovation Limited quick

>> response including their attachments as justifications.

>>

>> I note that;

>>

>> ⁃ the service contract with Cloud Innovation covering the announcement of

>> their prefixes by SEACOM AS37353 was terminated  by SEACOM.

>>

>> ⁃ some stale IRR route objects existed after termination of the contract.

>>

>> ⁃ through some multiple layer distribution an organisation in Manila

>> Philippines was “delegated“ an IP block from Cloud Innovation address space.

>>

>> ⁃ both upstream ISP and the customer in Manila set up a BGP session using

>> SEACOM's AS37353 as the ASN of the Manila customer.

>>

>> ⁃ there was a prompt reaction from the involved parties that included

>> apologies to SEACOM and the wider internet community.

>>

>> ⁃ there were promises from said parties to be a better netizen which

>> would mean, them not hijacking other networks ASN's.

>>

>> ⁃ there was clear refusal to disclose the details of the customer in

>> Manila Philippines who hijacked the affected SEACOM ASN.

>>

>> All put together, demonstrates that what happened was an impersonation

>> and not a BGP configuration error, nor an oversight in checking the right

>> to use of the SEACOM ASN.

>>

>> 1. Why is it that the real customer did not bother presenting its

>> apologies to the community

>>

>> 2. Why is there refusal to reveal customer’s details?

>>

>> 3. Why is it that the said prefix is no longer seen in the routing table

>> originated by the genius ASN or any other ASN?

>>

>> 4. Which networks were involved and what happened to the end users?

>>

>> Can someone from AFRINIC explain what “delegation of IP block” mean?

>>

>> Note: The self organised Internet knows how to deal with bad net

>> citizens.!

>>

>> Best regards

>> Gregoire Ehoumi

>>

>>

>> -------- Original message --------

>> From: Lu Heng <h.lu at anytimechinese.com>

>> Date: 2020-05-09 5:43 a.m. (GMT-05:00)

>> To: Mark Tinka <mark.tinka at seacom.mu>

>> Cc: "rpd at afrinic.net >> AfriNIC Resource Policy Discussion List" <

>> rpd at afrinic.net>

>> Subject: Re: [rpd] Cloud Innovation Displays Very Poor, If Not Criminal,

>> Netizenship

>>

>> To whom it may concern,

>>

>> On May 8, Mark Think posted a claim to multiple lists that Cloud

>> Innovation was abusing an ASN (37353) that didn’t belong to them (Cloud

>> Innovation) but rather belonged to Seacom through their acquisition of

>> MacroLAN.

>>

>> While we regret this unfortunate incident, Mark’s claims that it was

>> criminal or bad netizenship on the part of Cloud Innovation is without

>> foundation and utterly incorrect.

>>

>> As shown below in the attached document from Paul Wollner(Ex-CTO of

>> Macrolan who created IRR routes to allow Macrolan to announce Cloud

>> Innovation's prefix); letter from Link Infinity International Ltd. (Link

>> Infinity), A customer of Cloud Innovation; and attached LOA from LARUS

>> authorizing IPDC Solutions to announce the prefix with origin AS134190.

>> And a Letter from IPDC. This was an innocent mistake committed by third

>> parties and had nothing to do with any action by Cloud Innovation or LARUS.

>>

>> Here’s what happened:

>>

>> Cloud Innovation delegated a /24 to Link Infinity, an ISP in December

>> 2019.

>>

>>

>> Link Infinity further delegated that same /24 to IPDC and asked Cloud

>> innovation to issue an LOA, which we did. The LOA specifically required

>> IPDC to use its own ASN to announce the space (AS134190).

>>

>> IPDC subsequently authorized one of its customers to use the said prefix.

>>

>>

>> For reasons still unknown to Cloud Innovation, IPDC and their customer

>> set up a BGP session wherein their customer used AS37353 as the origin to

>> advertise prefix 156.241.3.0/24.

>>

>>

>> Upon discovering the announcement, rather than contact Cloud Innovation,

>> Mark contacted IPDC who provided him with an incomplete explanation blaming

>> their customer and Mark, not realizing that Cloud Innovation was not the

>> customer in question posted far and wide about the event. It is unclear to

>> us why he chose to do this rather than contact us to try and resolve the

>> issue.

>>

>>

>> A contributing factor to the erroneous BGP configuration by IPDC's

>> customer may have been data contained in some outdated IRR route objects

>> for 156.241.0.0/16 which have subsequently been deleted.

>>

>> As soon as we became aware of the problem (via Mark’s email), we began to

>> investigate the situation. As soon as it was clear that this was the result

>> of third-party actions, we reached out to Mark privately to let him know

>> what we knew and that we were still investigating. We delayed making a

>> public statement in order to try and ascertain all of the facts of the

>> situation. We prefer not to make public statements based on incomplete

>> information.

>>

>> We apologize to the community for our small part in this unfortunate

>> incident and assure you that we work very hard to remain good netizens and

>> will address any concerns promptly when they come to our attention.

>>

>>

>> Sincerely,

>>

>> Lu Heng

>> CEO

>> Cloud Innovations

>>

>> Attached:

>> 1. Letter from Paul Wollner

>> 2. Letter from Link Infinity

>> 3. LOA Issued to IPDC Solutions for announcing 156.241.3.0/24 from

>> AS134190

>>         4.     Letter from IPDC

>>

>> FYI: LARUS is proving IP management service for Cloud Innovation, while

>> LARUS is also providing IP management service to other third parties in all

>> regions, for full disclosure, LARUS and Cloud Innovation are headed by same

>> CEO.

>>

>> Content of those letters have been posted here for your convince:

>>

>> *IPDC:*

>>

>> FOR IMMEDIATE RELEASE [Perusal of Cloud Innovation Ltd]

>>

>>

>> IPDC Solutions Pte Ltd (IPDC) is a customer of Cloud Innovation Ltd and

>> over the course of our corporate relationship we were given the authority

>> to use address block 156.241.3.0/24 since 9th December 2019.

>>

>>

>> On 12th December 2019, we have delegated that address block to our

>> client. Following which our client further instructed us to announce the

>> prefix with AS37353. In good will after minimal verification through WHOIS’

>> IP Prefix we have proceeded to execute our client’s request.

>>

>>

>> On 7th May 2020 IPDC was then contacted by SEACOM, the legitimate holder

>> of record for that ASN and have since learned that the customer’s use of

>> that ASN was erroneous and not permitted by SEACOM and immediate action has

>> been taken to rectify this situation.

>>

>>

>> IPDC would like to apologize for our lack of attention in conducting

>> thorough verification and wish to highlight that the involvement of Cloud

>> Innovation Ltd in the entire process was providing the addresses to us and

>> that we truly apologize as we understand that this incident may have

>> indirectly implicated Cloud Innovation Ltd.

>>

>>

>> IPDC however, does not wish to disclose our client information and our

>> client information shall remain confidential under present circumstances.

>> Once again, we apologize for our shortcomings.

>>

>>

>> *Link Infinity:*

>>

>>

>> To whom it may concern,

>>

>> We at HK Infinity International Ltd are a customer of Cloud Innovation

>> and in the course received rights to use 156.241.3.0/24 from them.

>> Beginning December, 2019, we delegated the right to announce this prefix to

>> IPDC Solutions Pte Ltd. (IPDC). We asked Cloud Innovation to provide an LOA

>> authorizing them to announce the space which was subsequently received.

>> IPDC subsequently and without our knowledge delegated this space to one of

>> their customers and allowed them to originate it from AS37353.

>>

>> This announcement was not authorized by us, nor is it permitted by the

>> LOA provided by Cloud Innovation.

>>

>> Unfortunately, we were not aware of the situation until after it had

>> already been resolved.

>>

>> It was never our intent to violate the LOA or to allow the prefix to be

>> announced from a hijacked ASN. We do not condone this and apologize

>> sincerely to the community for what has happened here.

>>

>> Sincere Apologies,

>>

>>

>> *Paul Wollner:*

>>

>>

>> 8 May 2020

>>

>> TO WHOM IT MAY CONCERN

>>

>> In the light of the recent email on NAPAfrica mailing list, I would just

>> like to clear the air.

>>

>>

>> The IP route objects were created by myself for Cloud Innovation when

>> they signed up as a client of Macrolan ( now SEACOM) as they didn't have

>> their own AS.

>>

>> At the time I was working for Macrolan (now SEACOM). I left the

>> employment of SEACOM in October 2019.

>>

>> It appears that when Cloud Innovation's contract with SEACOM came to an

>> end, the route objects were never cleaned up.

>>

>> This occurred after I left SEACOM's employment. Since leaving, I have no

>> access to these objects.

>>

>> Regards

>>

>> Paul Wollner

>> 082-786-9776

>>

>>

>>

>> On Fri, 8 May 2020 at 22:10, Mark Tinka <mark.tinka at seacom.mu> wrote:

>>

>>> Hi all.

>>>

>>> I'm not one to b**ch & moan in public, but per subject, I could not let

>>> this one slide.

>>>

>>> And if you get this from multiple mailing lists, apologies for that -

>>> I'm just trying to make sure that this reaches as wide an audience as

>>> possible, on the continent.

>>>

>>> SEACOM (AS37100) acquired MacroLan (AS37353) a couple of years ago.

>>> MacroLan is now part of the SEACOM family, and while we are in the process

>>> of integrating that network into AS37100, some existing services continue

>>> to be delivered on AS37353 until that exercise is completed.

>>>

>>> One of the customers that AS37353 was providing services to was Cloud

>>> Innovation, in Cape Town. From a routing perspective, because Cloud

>>> Innovation had no AS number for this service, all of their IP address space

>>> was being originated by AS37353, on their behalf.

>>>

>>> In December of 2019, AS37353 ceased providing services to Cloud

>>> Innovation. That is 6 months ago.

>>>

>>> In recent days, it came to SEACOM's attention that some of Cloud

>>> Innovation's IP address space was being originated by AS37353 -

>>> specifically, 156.241.3.0/24 - even though we were sure that they were

>>> no longer a customer of AS37353 since December of 2019. At first, we

>>> thought this was a cached entry in a number of popular looking glasses, but

>>> then every looking glass had the same entry, which made this an unlikely

>>> bug.

>>>

>>> As of yesterday afternoon, see below what Telia's looking glass was

>>> saying about this prefix:

>>>

>>> *****

>>>

>>> Path #1: Received by speaker 0

>>>   4809 134190 37353

>>>     2.255.249.42 (metric 2134) from 2.255.253.101 (80.91.242.40)

>>>       Origin incomplete, localpref 200, valid, internal, best,

>>> group-best, import-candidate

>>> Communities:

>>>

>>> 1299:431

>>>     (RPKI state Unknown)

>>>

>>> 1299:1000 1299:30000 1299:30600 23456:20413 45352:4500 45352:9204

>>>

>>> *****

>>>

>>> But when I run a traceroute from my house to that prefix, it clearly was

>>> not ending up in Cape Town, where AS37353's main operation resides:

>>>

>>> *****

>>>

>>> MacBook-Pro-7:~ tinka$ traceroute -I 156.241.3.1

>>> traceroute to 156.241.3.1 (156.241.3.1), 64 hops max, 72 byte packets

>>>  1  172.16.0.254 (172.16.0.254)  14.824 ms  11.522 ms  3.525 ms

>>>  2  xe-1-3-0-1064.er-01-jnb.za.seacomnet.com (105.22.37.13)  5.620 ms

>>> 9.714 ms  9.887 ms

>>>  3  ce-0-2-0-0.cr-02-jnb.za.seacomnet.com (105.16.28.2)  175.232 ms

>>> 172.699 ms  175.896 ms

>>>  4  xe-0-0-0-8.cr-02-cpt.za.seacomnet.com (105.16.9.182)  164.496 ms

>>> 163.578 ms  163.546 ms

>>>  5  105.16.14.153 (105.16.14.153)  169.812 ms  171.272 ms  177.115 ms

>>>  6  xe-0-0-0-0.br-02-lhr.uk.seacomnet.com (105.16.34.253)  168.911 ms

>>> 172.958 ms  165.165 ms

>>>  7  82.112.115.169 (82.112.115.169)  172.700 ms  176.482 ms  174.375 ms

>>>  8  ae-17.r05.londen12.uk.bb.gin.ntt.net (129.250.2.147)  672.099 ms

>>> 613.617 ms  615.109 ms

>>>  9  ae-2.r24.londen12.uk.bb.gin.ntt.net (129.250.4.244)  181.952 ms

>>> 183.087 ms  187.302 ms

>>> 10  ae-16.r20.frnkge13.de.bb.gin.ntt.net (129.250.3.13)  190.511 ms

>>> 185.579 ms  187.058 ms

>>> 11  ae-3.r20.sngpsi07.sg.bb.gin.ntt.net (129.250.4.17)  520.882 ms

>>> 613.982 ms  615.275 ms

>>> 12  ae-9.r24.tkokhk01.hk.bb.gin.ntt.net (129.250.7.67)  612.301 ms

>>> 586.886 ms  436.711 ms

>>> 13  ae-1.r03.tkokhk01.hk.bb.gin.ntt.net (129.250.6.98)  614.470 ms

>>> 613.416 ms  614.281 ms

>>> 14  ce-0-3-0-3.r03.tkokhk01.hk.ce.gin.ntt.net (203.131.241.126)

>>> 614.128 ms  613.661 ms  615.416 ms

>>> 15  * *     *

>>> 16  * * *

>>> 17  156.241.3.1 (156.241.3.1)  494.400 ms  410.180 ms *

>>> MacBook-Pro-7:~ tinka$

>>>

>>> *****

>>>

>>> So we, then, realized that this was a fraudulent use of MacroLan's

>>> AS37353, to which we had given no express permission.

>>>

>>> As luck would have it, due to my days living and working in Malaysia, I

>>> know the good folk that operate AS134190 (IPDC Solutions), who was the

>>> upstream providing transit for this prefix. So I rang them up yesterday

>>> afternoon, told them what was happening, and within the hour, they got that

>>> eBGP session shutdown. I am most grateful to them for their quick response

>>> and immediate understanding of what was going on. Even with all the

>>> technology we have today, it, many times, comes down to having good friends

>>> in good places.

>>>

>>> Anyway, it turns out the ISP that had acquired this prefix from Cloud

>>> Innovation is based in Manila, Philippines. When IPDC delivered their

>>> transit service to them in Manila, that ISP informed them that they should

>>> use AS37353 for the eBGP session. Yes, one could argue that IPDC should

>>> have done their checks to ensure that the AS number being provided by their

>>> customer belongs to them, but to be honest, I'm not too bothered about that

>>> compared to Cloud Innovation's thinking that it was okay to use another

>>> network's AS number in order to deliver their services to their customers.

>>>

>>> IPDC confirm that this service was activated for the Manila ISP in

>>> December of 2019, right around the time Cloud Innovation's service with

>>> SEACOM, in Cape Town, ended.

>>>

>>> As it currently stands, the ISP in Manila is now off the Internet, with

>>> some 12 paying customers currently without service. Their excuse - they do

>>> not have an AS number of their own.

>>>

>>> IPDC tried to find out why the ISP in Manila thought that it was okay to

>>> use another network's AS number for their service, and as it turns out, the

>>> network engineer at the Manila ISP that set this up has since left the

>>> company. All the ones currently there do not have any history about all of

>>> this.

>>>

>>> Currently, 156.241.3.0/24 is not in the global BGP table:

>>>

>>> *****

>>>

>>> lg-01-ams.nl>sh ip bgp 156.241.3.0/24

>>> % Network not in table

>>> lg-01-ams.nl>

>>>

>>> lg-01-nbo.ke>sh ip bgp 156.241.3.0/24

>>> % Network not in table

>>> lg-01-nbo.ke>

>>>

>>> lg-01-cpt.za>sh ip bgp 156.241.3.0/24

>>> % Network not in table

>>> lg-01-cpt.za>

>>>

>>> *****

>>>

>>> That Cloud Innovation thought it was okay for them to use MacroLan's AS

>>> number to originate their own prefixes into the global BGP is as morally

>>> reprehensible as it is technologically distasteful.

>>>

>>> SEACOM have been working very closely with AFRINIC to delete previous

>>> route objects from their IRR that certify a relationship between Cloud

>>> Innovation's parent /16 aggregates that cover this prefix, and AS37353, but

>>> this is one of those objects that cannot be removed via the MyAFRINIC

>>> portal, and requires manual intervention from AFRINIC.

>>>

>>> I have not, personally, spoken to the proprietors of Cloud Innovation

>>> and/or Outside Heaven, as I don't see how anything could explain this with

>>> any degree of justification.

>>>

>>> For now, I will find some beer to wipe the foul taste from my mouth,

>>> while we (SEACOM) consider what to do about this.

>>>

>>> And yes, for those who are wondering, RPKI's ROV would not have

>>> prevented this, in its current form. This is AS hijacking, and ROV, today,

>>> tries to solve the prefix-hijacking problem, first.

>>>

>>> Thank you for your attention.

>>>

>>> Mark.

>>> _______________________________________________

>>> RPD mailing list

>>> RPD at afrinic.net

>>> https://lists.afrinic.net/mailman/listinfo/rpd

>>>

>>

>>

>> --

>> --

>> Kind regards.

>> Lu

>>

>> _______________________________________________

>> RPD mailing list

>> RPD at afrinic.net

>> https://lists.afrinic.net/mailman/listinfo/rpd

>>

>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.afrinic.net/pipermail/rpd/attachments/20200525/2a5a72a4/attachment-0001.html>