[Community-Discuss] Cloud Innovation Displays Very Poor, If Not Criminal, Netizenship

Gregoire EHOUMI gregoire.ehoumi at yahoo.fr
Sat May 23 14:21:26 UTC 2020

Hello,Thanks Mark for exposing the details of the SEACOM AS37353 hijacking.I carefully read your report and also the Cloud Innovation Limited quick response including their attachments as justifications.I note that; ⁃ the service contract with Cloud Innovation covering the announcement of their prefixes by SEACOM AS37353 was terminated  by SEACOM. ⁃ some stale IRR route objects existed after termination of the contract. ⁃ through some multiple layer distribution an organisation in Manila Philippines was “delegated“ an IP block from Cloud Innovation address space. ⁃ both upstream ISP and the customer in Manila set up a BGP session using SEACOM's AS37353 as the ASN of the Manila customer. ⁃ there was a prompt reaction from the involved parties that included apologies to SEACOM and the wider internet community. ⁃ there were promises from said parties to be a better netizen which would mean, them not hijacking other networks ASN's. ⁃ there was clear refusal to disclose the details of the customer in Manila Philippines who hijacked the affected SEACOM ASN.All put together, demonstrates that what happened was an impersonation and not a BGP configuration error, nor an oversight in checking the right to use of the SEACOM ASN.1. Why is it that the real customer did not bother presenting its apologies to the community2. Why is there refusal to reveal customer’s details?3. Why is it that the said prefix is no longer seen in the routing table originated by the genius ASN or any other ASN?4. Which networks were involved and what happened to the end users?Can someone from AFRINIC explain what “delegation of IP block” mean?Note: The self organised Internet knows how to deal with bad net citizens.!Best regards Gregoire Ehoumi 
-------- Original message --------From: Lu Heng <h.lu at anytimechinese.com> Date: 2020-05-09 5:41 a.m. (GMT-05:00) To: General Discussions of AFRINIC <community-discuss at afrinic.net> Subject: Re: [Community-Discuss] Cloud Innovation Displays Very Poor, If Not Criminal, Netizenship To whom it may concern,On May 8, Mark Think posted a claim to multiple lists that Cloud Innovation was abusing an ASN (37353) that didn’t belong to them (Cloud Innovation) but rather belonged to Seacom through their acquisition of MacroLAN.While we regret this unfortunate incident, Mark’s claims that it was criminal or bad netizenship on the part of Cloud Innovation is without foundation and utterly incorrect.As shown below in the attached document from Paul Wollner(Ex-CTO of Macrolan who created IRR routes to allow Macrolan to announce Cloud Innovation's prefix); letter from Link Infinity International Ltd. (Link Infinity), A customer of Cloud Innovation; and attached LOA from LARUS authorizing IPDC Solutions to announce the prefix with origin AS134190.  And a Letter from IPDC. This was an innocent mistake committed by third parties and had nothing to do with any action by Cloud Innovation or LARUS.Here’s what happened:Cloud Innovation delegated a /24 to Link Infinity, an ISP in December 2019.Link Infinity further delegated that same /24 to IPDC and asked Cloud innovation to issue an LOA, which we did. The LOA specifically required IPDC to use its own ASN to announce the space (AS134190).IPDC subsequently authorized one of its customers to use the said prefix.For reasons still unknown to Cloud Innovation, IPDC and their customer set up a BGP session wherein their customer used AS37353 as the origin to advertise prefix 156.241.3.0/24.Upon discovering the announcement, rather than contact Cloud Innovation, Mark contacted IPDC who provided him with an incomplete explanation blaming their customer and Mark, not realizing that Cloud Innovation was not the customer in question posted far and wide about the event. It is unclear to us why he chose to do this rather than contact us to try and resolve the issue.A contributing factor to the erroneous BGP configuration by IPDC's customer may have been data contained in some outdated IRR route objects for 156.241.0.0/16 which have subsequently been deleted.As soon as we became aware of the problem (via Mark’s email), we began to investigate the situation. As soon as it was clear that this was the result of third-party actions, we reached out to Mark privately to let him know what we knew and that we were still investigating. We delayed making a public statement in order to try and ascertain all of the facts of the situation. We prefer not to make public statements based on incomplete information.We apologize to the community for our small part in this unfortunate incident and assure you that we work very hard to remain good netizens and will address any concerns promptly when they come to our attention.Sincerely,Lu HengCEOCloud InnovationsAttached: 1. Letter from Paul Wollner 2. Letter from Link Infinity 3. LOA Issued to IPDC Solutions for announcing 156.241.3.0/24 from AS134190        4.     Letter from IPDCFYI: LARUS is proving IP management service for Cloud Innovation, while LARUS is also providing IP management service to other third parties in all regions, for full disclosure, LARUS and Cloud Innovation are headed by same CEO.Content of those letters have been posted here for your convince:IPDC:FOR IMMEDIATE RELEASE [Perusal of Cloud Innovation Ltd]IPDC Solutions Pte Ltd (IPDC) is a customer of Cloud Innovation Ltd and over the course of our corporate relationship we were given the authority to use address block 156.241.3.0/24 since 9th December 2019. On 12th December 2019, we have delegated that address block to our client. Following which our client further instructed us to announce the prefix with AS37353. In good will after minimal verification through WHOIS’ IP Prefix we have proceeded to execute our client’s request. On 7th May 2020 IPDC was then contacted by SEACOM, the legitimate holder of record for that ASN and have since learned that the customer’s use of that ASN was erroneous and not permitted by SEACOM and immediate action has been taken to rectify this situation. IPDC would like to apologize for our lack of attention in conducting thorough verification and wish to highlight that the involvement of Cloud Innovation Ltd in the entire process was providing the addresses to us and that we truly apologize as we understand that this incident may have indirectly implicated Cloud Innovation Ltd. IPDC however, does not wish to disclose our client information and our client information shall remain confidential under present circumstances. Once again, we apologize for our shortcomings. Link Infinity:To whom it may concern,We at HK Infinity International Ltd are a customer of Cloud Innovation and in the course received rights to use 156.241.3.0/24 from them. Beginning December, 2019, we delegated the right to announce this prefix to IPDC Solutions Pte Ltd. (IPDC). We asked Cloud Innovation to provide an LOA authorizing them to announce the space which was subsequently received. IPDC subsequently and without our knowledge delegated this space to one of their customers and allowed them to originate it from AS37353.This announcement was not authorized by us, nor is it permitted by the LOA provided by Cloud Innovation.Unfortunately, we were not aware of the situation until after it had already been resolved.It was never our intent to violate the LOA or to allow the prefix to be announced from a hijacked ASN. We do not condone this and apologize sincerely to the community for what has happened here.Sincere Apologies,Paul Wollner:8 May 2020TO WHOM IT MAY CONCERN                                        In the light of the recent email on NAPAfrica mailing list, I would just like to clear the air. The IP route objects were created by myself for Cloud Innovation when they signed up as a client of Macrolan ( now SEACOM) as they didn't have their own AS.At the time I was working for Macrolan (now SEACOM). I left the employment of SEACOM in October 2019.It appears that when Cloud Innovation's contract with SEACOM came to an end, the route objects were never cleaned up.This occurred after I left SEACOM's employment. Since leaving, I have no access to these objects.RegardsPaul Wollner082-786-9776To whom it may concern,On May 8, Mark Think posted a claim to multiple lists that Cloud Innovation was abusing an ASN (37353) that didn’t belong to them (Cloud Innovation) but rather belonged to Seacom through their acquisition of MacroLAN.While we regret this unfortunate incident, Mark’s claims that it was criminal or bad netizenship on the part of Cloud Innovation is without foundation and utterly incorrect.As shown below in the attached document from Paul Wollner(Ex-CTO of Macrolan who created IRR routes to allow Macrolan to announce Cloud Innovation's prefix); letter from Link Infinity International Ltd. (Link Infinity), A customer of Cloud Innovation; and attached LOA from LARUS authorizing IPDC Solutions to announce the prefix with origin AS134190.  And a Letter from IPDC. This was an innocent mistake committed by third parties and had nothing to do with any action by Cloud Innovation or LARUS.Here’s what happened:Cloud Innovation delegated a /24 to Link Infinity, an ISP in December 2019.Link Infinity further delegated that same /24 to IPDC and asked Cloud innovation to issue an LOA, which we did. The LOA specifically required IPDC to use its own ASN to announce the space (AS134190).IPDC subsequently authorized one of its customers to use the said prefix.For reasons still unknown to Cloud Innovation, IPDC and their customer set up a BGP session wherein their customer used AS37353 as the origin to advertise prefix 156.241.3.0/24.Upon discovering the announcement, rather than contact Cloud Innovation, Mark contacted IPDC who provided him with an incomplete explanation blaming their customer and Mark, not realizing that Cloud Innovation was not the customer in question posted far and wide about the event. It is unclear to us why he chose to do this rather than contact us to try and resolve the issue.A contributing factor to the erroneous BGP configuration by IPDC's customer may have been data contained in some outdated IRR route objects for 156.241.0.0/16 which have subsequently been deleted.As soon as we became aware of the problem (via Mark’s email), we began to investigate the situation. As soon as it was clear that this was the result of third-party actions, we reached out to Mark privately to let him know what we knew and that we were still investigating. We delayed making a public statement in order to try and ascertain all of the facts of the situation. We prefer not to make public statements based on incomplete information.We apologize to the community for our small part in this unfortunate incident and assure you that we work very hard to remain good netizens and will address any concerns promptly when they come to our attention.Sincerely,Lu HengCEOCloud InnovationsAttached: 1. Letter from Paul Wollner 2. Letter from Link Infinity 3. LOA Issued to IPDC Solutions for announcing 156.241.3.0/24 from AS134190        4.     Letter from IPDCFYI: LARUS is proving IP management service for Cloud Innovation, while LARUS is also providing IP management service to other third parties in all regions, for full disclosure, LARUS and Cloud Innovation are headed by same CEO.Content of those letters have been posted here for your convince:IPDC:FOR IMMEDIATE RELEASE [Perusal of Cloud Innovation Ltd]IPDC Solutions Pte Ltd (IPDC) is a customer of Cloud Innovation Ltd and over the course of our corporate relationship we were given the authority to use address block 156.241.3.0/24 since 9th December 2019. On 12th December 2019, we have delegated that address block to our client. Following which our client further instructed us to announce the prefix with AS37353. In good will after minimal verification through WHOIS’ IP Prefix we have proceeded to execute our client’s request. On 7th May 2020 IPDC was then contacted by SEACOM, the legitimate holder of record for that ASN and have since learned that the customer’s use of that ASN was erroneous and not permitted by SEACOM and immediate action has been taken to rectify this situation. IPDC would like to apologize for our lack of attention in conducting thorough verification and wish to highlight that the involvement of Cloud Innovation Ltd in the entire process was providing the addresses to us and that we truly apologize as we understand that this incident may have indirectly implicated Cloud Innovation Ltd. IPDC however, does not wish to disclose our client information and our client information shall remain confidential under present circumstances. Once again, we apologize for our shortcomings. Link Infinity:To whom it may concern,We at HK Infinity International Ltd are a customer of Cloud Innovation and in the course received rights to use 156.241.3.0/24 from them. Beginning December, 2019, we delegated the right to announce this prefix to IPDC Solutions Pte Ltd. (IPDC). We asked Cloud Innovation to provide an LOA authorizing them to announce the space which was subsequently received. IPDC subsequently and without our knowledge delegated this space to one of their customers and allowed them to originate it from AS37353.This announcement was not authorized by us, nor is it permitted by the LOA provided by Cloud Innovation.Unfortunately, we were not aware of the situation until after it had already been resolved.It was never our intent to violate the LOA or to allow the prefix to be announced from a hijacked ASN. We do not condone this and apologize sincerely to the community for what has happened here.Sincere Apologies,Paul Wollner:8 May 2020TO WHOM IT MAY CONCERN                                        In the light of the recent email on NAPAfrica mailing list, I would just like to clear the air. The IP route objects were created by myself for Cloud Innovation when they signed up as a client of Macrolan ( now SEACOM) as they didn't have their own AS.At the time I was working for Macrolan (now SEACOM). I left the employment of SEACOM in October 2019.It appears that when Cloud Innovation's contract with SEACOM came to an end, the route objects were never cleaned up.This occurred after I left SEACOM's employment. Since leaving, I have no access to these objects.RegardsPaul Wollner082-786-9776On Fri, 8 May 2020 at 22:08, Mark Tinka <mark.tinka at seacom.mu> wrote:




Hi all.

I'm not one to b**ch & moan in public, but per subject, I
could not let this one slide.

And if you get this from multiple mailing lists, apologies for
that - I'm just trying to make sure that this reaches as wide an
audience as possible, on the continent.

SEACOM (AS37100) acquired MacroLan (AS37353) a couple of years
ago. MacroLan is now part of the SEACOM family, and while we are
in the process of integrating that network into AS37100, some
existing services continue to be delivered on AS37353 until that
exercise is completed.

One of the customers that AS37353 was providing services to was
Cloud Innovation, in Cape Town. From a routing perspective,
because Cloud Innovation had no AS number for this service, all of
their IP address space was being originated by AS37353, on their
behalf.

In December of 2019, AS37353 ceased providing services to Cloud
Innovation. That is 6 months ago.

In recent days, it came to SEACOM's attention that some of Cloud
Innovation's IP address space was being originated by AS37353 -
specifically, 156.241.3.0/24 - even though we were sure that they
were no longer a customer of AS37353 since December of 2019. At
first, we thought this was a cached entry in a number of popular
looking glasses, but then every looking glass had the same entry,
which made this an unlikely bug.

As of yesterday afternoon, see below what Telia's looking glass
was saying about this prefix:

*****

Path #1: Received by speaker 0
  4809 134190 37353
    2.255.249.42 (metric 2134) from 2.255.253.101 (80.91.242.40)
      Origin incomplete, localpref 200, valid, internal, best,
group-best, import-candidate
Communities:

1299:431
    (RPKI state Unknown)

1299:1000 1299:30000 1299:30600 23456:20413 45352:4500 45352:9204

*****

But when I run a traceroute from my house to that prefix, it
clearly was not ending up in Cape Town, where AS37353's main
operation resides:

*****

MacBook-Pro-7:~ tinka$ traceroute -I 156.241.3.1
traceroute to 156.241.3.1 (156.241.3.1), 64 hops max, 72 byte
packets
 1  172.16.0.254 (172.16.0.254)  14.824 ms  11.522 ms  3.525 ms
 2  xe-1-3-0-1064.er-01-jnb.za.seacomnet.com (105.22.37.13)  5.620
ms  9.714 ms  9.887 ms
 3  ce-0-2-0-0.cr-02-jnb.za.seacomnet.com (105.16.28.2)  175.232
ms  172.699 ms  175.896 ms
 4  xe-0-0-0-8.cr-02-cpt.za.seacomnet.com (105.16.9.182)  164.496
ms  163.578 ms  163.546 ms
 5  105.16.14.153 (105.16.14.153)  169.812 ms  171.272 ms  177.115
ms
 6  xe-0-0-0-0.br-02-lhr.uk.seacomnet.com (105.16.34.253)  168.911
ms  172.958 ms  165.165 ms
 7  82.112.115.169 (82.112.115.169)  172.700 ms  176.482 ms 
174.375 ms
 8  ae-17.r05.londen12.uk.bb.gin.ntt.net (129.250.2.147)  672.099
ms  613.617 ms  615.109 ms
 9  ae-2.r24.londen12.uk.bb.gin.ntt.net (129.250.4.244)  181.952
ms  183.087 ms  187.302 ms
10  ae-16.r20.frnkge13.de.bb.gin.ntt.net (129.250.3.13)  190.511
ms  185.579 ms  187.058 ms
11  ae-3.r20.sngpsi07.sg.bb.gin.ntt.net (129.250.4.17)  520.882
ms  613.982 ms  615.275 ms
12  ae-9.r24.tkokhk01.hk.bb.gin.ntt.net (129.250.7.67)  612.301
ms  586.886 ms  436.711 ms
13  ae-1.r03.tkokhk01.hk.bb.gin.ntt.net (129.250.6.98)  614.470
ms  613.416 ms  614.281 ms
14  ce-0-3-0-3.r03.tkokhk01.hk.ce.gin.ntt.net (203.131.241.126) 
614.128 ms  613.661 ms  615.416 ms
15  * *     *
16  * * *
17  156.241.3.1 (156.241.3.1)  494.400 ms  410.180 ms *
MacBook-Pro-7:~ tinka$

*****

So we, then, realized that this was a fraudulent use of MacroLan's
AS37353, to which we had given no express permission.

As luck would have it, due to my days living and working in
Malaysia, I know the good folk that operate AS134190 (IPDC
Solutions), who was the upstream providing transit for this
prefix. So I rang them up yesterday afternoon, told them what was
happening, and within the hour, they got that eBGP session
shutdown. I am most grateful to them for their quick response and
immediate understanding of what was going on. Even with all the
technology we have today, it, many times, comes down to having
good friends in good places.

Anyway, it turns out the ISP that had acquired this prefix from
Cloud Innovation is based in Manila, Philippines. When IPDC
delivered their transit service to them in Manila, that ISP
informed them that they should use AS37353 for the eBGP session.
Yes, one could argue that IPDC should have done their checks to
ensure that the AS number being provided by their customer belongs
to them, but to be honest, I'm not too bothered about that
compared to Cloud Innovation's thinking that it was okay to use
another network's AS number in order to deliver their services to
their customers.

IPDC confirm that this service was activated for the Manila ISP in
December of 2019, right around the time Cloud Innovation's service
with SEACOM, in Cape Town, ended.

As it currently stands, the ISP in Manila is now off the Internet,
with some 12 paying customers currently without service. Their
excuse - they do not have an AS number of their own.

IPDC tried to find out why the ISP in Manila thought that it was
okay to use another network's AS number for their service, and as
it turns out, the network engineer at the Manila ISP that set this
up has since left the company. All the ones currently there do not
have any history about all of this.

Currently, 156.241.3.0/24 is not in the global BGP table:

*****

lg-01-ams.nl>sh ip bgp 156.241.3.0/24
% Network not in table
lg-01-ams.nl>

lg-01-nbo.ke>sh ip bgp 156.241.3.0/24
% Network not in table
lg-01-nbo.ke>

lg-01-cpt.za>sh ip bgp 156.241.3.0/24
% Network not in table
lg-01-cpt.za>

*****

That Cloud Innovation thought it was okay for them to use
MacroLan's AS number to originate their own prefixes into the
global BGP is as morally reprehensible as it is technologically
distasteful.

SEACOM have been working very closely with AFRINIC to delete
previous route objects from their IRR that certify a relationship
between Cloud Innovation's parent /16 aggregates that cover this
prefix, and AS37353, but this is one of those objects that cannot
be removed via the MyAFRINIC portal, and requires manual
intervention from AFRINIC.

I have not, personally, spoken to the proprietors of Cloud
Innovation and/or Outside Heaven, as I don't see how anything
could explain this with any degree of justification.

For now, I will find some beer to wipe the foul taste from my
mouth, while we (SEACOM) consider what to do about this.

And yes, for those who are wondering, RPKI's ROV would not have
prevented this, in its current form. This is AS hijacking, and
ROV, today, tries to solve the prefix-hijacking problem, first.

Thank you for your attention.

Mark.



_______________________________________________
Community-Discuss mailing list
Community-Discuss at afrinic.net
https://lists.afrinic.net/mailman/listinfo/community-discuss
-- --Kind regards.Lu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.afrinic.net/pipermail/community-discuss/attachments/20200523/0cb5fb50/attachment.html>

More information about the Community-Discuss mailing list