[rpd] Cloud Innovation Displays Very Poor, If Not Criminal, Netizenship

[Lu Heng]

To whom it may concern,

On May 8, Mark Think posted a claim to multiple lists that Cloud Innovation
was abusing an ASN (37353) that didn’t belong to them (Cloud Innovation)
but rather belonged to Seacom through their acquisition of MacroLAN.

While we regret this unfortunate incident, Mark’s claims that it was
criminal or bad netizenship on the part of Cloud Innovation is without
foundation and utterly incorrect.

As shown below in the attached document from Paul Wollner(Ex-CTO of
Macrolan who created IRR routes to allow Macrolan to announce Cloud
Innovation's prefix); letter from Link Infinity International Ltd. (Link
Infinity), A customer of Cloud Innovation; and attached LOA from LARUS
authorizing IPDC Solutions to announce the prefix with origin AS134190.
And a Letter from IPDC. This was an innocent mistake committed by third
parties and had nothing to do with any action by Cloud Innovation or LARUS.

Here’s what happened:

Cloud Innovation delegated a /24 to Link Infinity, an ISP in December 2019.


Link Infinity further delegated that same /24 to IPDC and asked Cloud
innovation to issue an LOA, which we did. The LOA specifically required
IPDC to use its own ASN to announce the space (AS134190).

IPDC subsequently authorized one of its customers to use the said prefix.


For reasons still unknown to Cloud Innovation, IPDC and their customer set
up a BGP session wherein their customer used AS37353 as the origin to
advertise prefix 156.241.3.0/24.


Upon discovering the announcement, rather than contact Cloud Innovation,
Mark contacted IPDC who provided him with an incomplete explanation blaming
their customer and Mark, not realizing that Cloud Innovation was not the
customer in question posted far and wide about the event. It is unclear to
us why he chose to do this rather than contact us to try and resolve the
issue.


A contributing factor to the erroneous BGP configuration by IPDC's customer
may have been data contained in some outdated IRR route objects for
156.241.0.0/16 which have subsequently been deleted.

As soon as we became aware of the problem (via Mark’s email), we began to
investigate the situation. As soon as it was clear that this was the result
of third-party actions, we reached out to Mark privately to let him know
what we knew and that we were still investigating. We delayed making a
public statement in order to try and ascertain all of the facts of the
situation. We prefer not to make public statements based on incomplete
information.

We apologize to the community for our small part in this unfortunate
incident and assure you that we work very hard to remain good netizens and
will address any concerns promptly when they come to our attention.


Sincerely,

Lu Heng
CEO
Cloud Innovations

Attached:
1. Letter from Paul Wollner
2. Letter from Link Infinity
3. LOA Issued to IPDC Solutions for announcing 156.241.3.0/24 from AS134190
        4.     Letter from IPDC

FYI: LARUS is proving IP management service for Cloud Innovation, while
LARUS is also providing IP management service to other third parties in all
regions, for full disclosure, LARUS and Cloud Innovation are headed by same
CEO.

Content of those letters have been posted here for your convince:

*IPDC:*

FOR IMMEDIATE RELEASE [Perusal of Cloud Innovation Ltd]


IPDC Solutions Pte Ltd (IPDC) is a customer of Cloud Innovation Ltd and
over the course of our corporate relationship we were given the authority
to use address block 156.241.3.0/24 since 9th December 2019.


On 12th December 2019, we have delegated that address block to our client.
Following which our client further instructed us to announce the prefix
with AS37353. In good will after minimal verification through WHOIS’ IP
Prefix we have proceeded to execute our client’s request.


On 7th May 2020 IPDC was then contacted by SEACOM, the legitimate holder of
record for that ASN and have since learned that the customer’s use of that
ASN was erroneous and not permitted by SEACOM and immediate action has been
taken to rectify this situation.


IPDC would like to apologize for our lack of attention in conducting
thorough verification and wish to highlight that the involvement of Cloud
Innovation Ltd in the entire process was providing the addresses to us and
that we truly apologize as we understand that this incident may have
indirectly implicated Cloud Innovation Ltd.


IPDC however, does not wish to disclose our client information and our
client information shall remain confidential under present circumstances.
Once again, we apologize for our shortcomings.


*Link Infinity:*


To whom it may concern,

We at HK Infinity International Ltd are a customer of Cloud Innovation and
in the course received rights to use 156.241.3.0/24 from them. Beginning
December, 2019, we delegated the right to announce this prefix to IPDC
Solutions Pte Ltd. (IPDC). We asked Cloud Innovation to provide an LOA
authorizing them to announce the space which was subsequently received.
IPDC subsequently and without our knowledge delegated this space to one of
their customers and allowed them to originate it from AS37353.

This announcement was not authorized by us, nor is it permitted by the LOA
provided by Cloud Innovation.

Unfortunately, we were not aware of the situation until after it had
already been resolved.

It was never our intent to violate the LOA or to allow the prefix to be
announced from a hijacked ASN. We do not condone this and apologize
sincerely to the community for what has happened here.

Sincere Apologies,


*Paul Wollner:*


8 May 2020

TO WHOM IT MAY CONCERN

In the light of the recent email on NAPAfrica mailing list, I would just
like to clear the air.


The IP route objects were created by myself for Cloud Innovation when they
signed up as a client of Macrolan ( now SEACOM) as they didn't have their
own AS.

At the time I was working for Macrolan (now SEACOM). I left the employment
of SEACOM in October 2019.

It appears that when Cloud Innovation's contract with SEACOM came to an
end, the route objects were never cleaned up.

This occurred after I left SEACOM's employment. Since leaving, I have no
access to these objects.

Regards

Paul Wollner
082-786-9776



On Fri, 8 May 2020 at 22:10, Mark Tinka <mark.tinka at seacom.mu> wrote:


> Hi all.

>

> I'm not one to b**ch & moan in public, but per subject, I could not let

> this one slide.

>

> And if you get this from multiple mailing lists, apologies for that - I'm

> just trying to make sure that this reaches as wide an audience as possible,

> on the continent.

>

> SEACOM (AS37100) acquired MacroLan (AS37353) a couple of years ago.

> MacroLan is now part of the SEACOM family, and while we are in the process

> of integrating that network into AS37100, some existing services continue

> to be delivered on AS37353 until that exercise is completed.

>

> One of the customers that AS37353 was providing services to was Cloud

> Innovation, in Cape Town. From a routing perspective, because Cloud

> Innovation had no AS number for this service, all of their IP address space

> was being originated by AS37353, on their behalf.

>

> In December of 2019, AS37353 ceased providing services to Cloud

> Innovation. That is 6 months ago.

>

> In recent days, it came to SEACOM's attention that some of Cloud

> Innovation's IP address space was being originated by AS37353 -

> specifically, 156.241.3.0/24 - even though we were sure that they were no

> longer a customer of AS37353 since December of 2019. At first, we thought

> this was a cached entry in a number of popular looking glasses, but then

> every looking glass had the same entry, which made this an unlikely bug.

>

> As of yesterday afternoon, see below what Telia's looking glass was saying

> about this prefix:

>

> *****

>

> Path #1: Received by speaker 0

>   4809 134190 37353

>     2.255.249.42 (metric 2134) from 2.255.253.101 (80.91.242.40)

>       Origin incomplete, localpref 200, valid, internal, best, group-best,

> import-candidate

> Communities:

>

> 1299:431

>     (RPKI state Unknown)

>

> 1299:1000 1299:30000 1299:30600 23456:20413 45352:4500 45352:9204

>

> *****

>

> But when I run a traceroute from my house to that prefix, it clearly was

> not ending up in Cape Town, where AS37353's main operation resides:

>

> *****

>

> MacBook-Pro-7:~ tinka$ traceroute -I 156.241.3.1

> traceroute to 156.241.3.1 (156.241.3.1), 64 hops max, 72 byte packets

>  1  172.16.0.254 (172.16.0.254)  14.824 ms  11.522 ms  3.525 ms

>  2  xe-1-3-0-1064.er-01-jnb.za.seacomnet.com (105.22.37.13)  5.620 ms

> 9.714 ms  9.887 ms

>  3  ce-0-2-0-0.cr-02-jnb.za.seacomnet.com (105.16.28.2)  175.232 ms

> 172.699 ms  175.896 ms

>  4  xe-0-0-0-8.cr-02-cpt.za.seacomnet.com (105.16.9.182)  164.496 ms

> 163.578 ms  163.546 ms

>  5  105.16.14.153 (105.16.14.153)  169.812 ms  171.272 ms  177.115 ms

>  6  xe-0-0-0-0.br-02-lhr.uk.seacomnet.com (105.16.34.253)  168.911 ms

> 172.958 ms  165.165 ms

>  7  82.112.115.169 (82.112.115.169)  172.700 ms  176.482 ms  174.375 ms

>  8  ae-17.r05.londen12.uk.bb.gin.ntt.net (129.250.2.147)  672.099 ms

> 613.617 ms  615.109 ms

>  9  ae-2.r24.londen12.uk.bb.gin.ntt.net (129.250.4.244)  181.952 ms

> 183.087 ms  187.302 ms

> 10  ae-16.r20.frnkge13.de.bb.gin.ntt.net (129.250.3.13)  190.511 ms

> 185.579 ms  187.058 ms

> 11  ae-3.r20.sngpsi07.sg.bb.gin.ntt.net (129.250.4.17)  520.882 ms

> 613.982 ms  615.275 ms

> 12  ae-9.r24.tkokhk01.hk.bb.gin.ntt.net (129.250.7.67)  612.301 ms

> 586.886 ms  436.711 ms

> 13  ae-1.r03.tkokhk01.hk.bb.gin.ntt.net (129.250.6.98)  614.470 ms

> 613.416 ms  614.281 ms

> 14  ce-0-3-0-3.r03.tkokhk01.hk.ce.gin.ntt.net (203.131.241.126)  614.128

> ms  613.661 ms  615.416 ms

> 15  * *     *

> 16  * * *

> 17  156.241.3.1 (156.241.3.1)  494.400 ms  410.180 ms *

> MacBook-Pro-7:~ tinka$

>

> *****

>

> So we, then, realized that this was a fraudulent use of MacroLan's

> AS37353, to which we had given no express permission.

>

> As luck would have it, due to my days living and working in Malaysia, I

> know the good folk that operate AS134190 (IPDC Solutions), who was the

> upstream providing transit for this prefix. So I rang them up yesterday

> afternoon, told them what was happening, and within the hour, they got that

> eBGP session shutdown. I am most grateful to them for their quick response

> and immediate understanding of what was going on. Even with all the

> technology we have today, it, many times, comes down to having good friends

> in good places.

>

> Anyway, it turns out the ISP that had acquired this prefix from Cloud

> Innovation is based in Manila, Philippines. When IPDC delivered their

> transit service to them in Manila, that ISP informed them that they should

> use AS37353 for the eBGP session. Yes, one could argue that IPDC should

> have done their checks to ensure that the AS number being provided by their

> customer belongs to them, but to be honest, I'm not too bothered about that

> compared to Cloud Innovation's thinking that it was okay to use another

> network's AS number in order to deliver their services to their customers.

>

> IPDC confirm that this service was activated for the Manila ISP in

> December of 2019, right around the time Cloud Innovation's service with

> SEACOM, in Cape Town, ended.

>

> As it currently stands, the ISP in Manila is now off the Internet, with

> some 12 paying customers currently without service. Their excuse - they do

> not have an AS number of their own.

>

> IPDC tried to find out why the ISP in Manila thought that it was okay to

> use another network's AS number for their service, and as it turns out, the

> network engineer at the Manila ISP that set this up has since left the

> company. All the ones currently there do not have any history about all of

> this.

>

> Currently, 156.241.3.0/24 is not in the global BGP table:

>

> *****

>

> lg-01-ams.nl>sh ip bgp 156.241.3.0/24

> % Network not in table

> lg-01-ams.nl>

>

> lg-01-nbo.ke>sh ip bgp 156.241.3.0/24

> % Network not in table

> lg-01-nbo.ke>

>

> lg-01-cpt.za>sh ip bgp 156.241.3.0/24

> % Network not in table

> lg-01-cpt.za>

>

> *****

>

> That Cloud Innovation thought it was okay for them to use MacroLan's AS

> number to originate their own prefixes into the global BGP is as morally

> reprehensible as it is technologically distasteful.

>

> SEACOM have been working very closely with AFRINIC to delete previous

> route objects from their IRR that certify a relationship between Cloud

> Innovation's parent /16 aggregates that cover this prefix, and AS37353, but

> this is one of those objects that cannot be removed via the MyAFRINIC

> portal, and requires manual intervention from AFRINIC.

>

> I have not, personally, spoken to the proprietors of Cloud Innovation

> and/or Outside Heaven, as I don't see how anything could explain this with

> any degree of justification.

>

> For now, I will find some beer to wipe the foul taste from my mouth, while

> we (SEACOM) consider what to do about this.

>

> And yes, for those who are wondering, RPKI's ROV would not have prevented

> this, in its current form. This is AS hijacking, and ROV, today, tries to

> solve the prefix-hijacking problem, first.

>

> Thank you for your attention.

>

> Mark.

> _______________________________________________

> RPD mailing list

> RPD at afrinic.net

> https://lists.afrinic.net/mailman/listinfo/rpd

>



-- 
--
Kind regards.
Lu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.afrinic.net/pipermail/rpd/attachments/20200509/0e9ad38e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 200508_Wollner_Paul_NAPAfrica.pdf
Type: application/pdf
Size: 942672 bytes
Desc: not available
URL: <https://lists.afrinic.net/pipermail/rpd/attachments/20200509/0e9ad38e/attachment-0004.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LOA.pdf
Type: application/pdf
Size: 335184 bytes
Desc: not available
URL: <https://lists.afrinic.net/pipermail/rpd/attachments/20200509/0e9ad38e/attachment-0005.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 200509 IPDC-CI Official Letter for IR.pdf
Type: application/pdf
Size: 67129 bytes
Desc: not available
URL: <https://lists.afrinic.net/pipermail/rpd/attachments/20200509/0e9ad38e/attachment-0006.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: HK LINK INFINITY LETTER.pdf
Type: application/pdf
Size: 569698 bytes
Desc: not available
URL: <https://lists.afrinic.net/pipermail/rpd/attachments/20200509/0e9ad38e/attachment-0007.pdf>