[rpd] [Community-Discuss] Cloud Innovation Displays Very Poor, If Not Criminal, Netizenship

[Arnaud AMELINA]

Hi, all communities

It was said that the hijacking of AS37353 was a configuration mistake,
coupled with poor oversight in checking the right to use of the ASN by a
certain IPDC customer.

It was also said that this would have been facilitated by old and stalled
IRR objects from MacroLAN before the service contract ended in December
2019.

It was also stated that the BGP announcement was stopped and some actions
taken to delete the stalled routing registry object from IRR.

Below  are some data which gives some extra information and confirm
previous claim that this incident was not accidental.

NB: All data are collected from their source, Today, 28 May 2020, at 10:00
UTC.

IRRs created in March 2020 with AS37353 as Origin for 156.241.3.0/24 by
HGC

 $ whois -h whois.radb.net '156.241.3.0/24'

route:      156.241.3.0/24 descr:      Proxy-registered route object
origin:     AS37353 notify:     matthew.chan at hgc.com.hk mnt-by:
 MAINT-HGC-INTL changed:    matthew.chan at hgc.com.hk 20200304 source:
 RADB

=====
route:          156.241.3.0/24
descr:          Proxy-registered route object origin:         AS37353
remarks:        This is a HGC customer route-object
remarks:        which is being exported under this origin AS.
remarks:
remarks:        This route object was created because no existing remarks:
      route object with the same origin was found.
remarks:
remarks:        Please contact radb at hutchcity.com if you have any remarks:
      questions regarding this object.
notify:         radb at hutchcity.com
mnt-by:         MAINT-AS9304
changed:        radb at hutchcity.com 20200304
source:         NTTCOM
====

https://www.radb.net/query?advanced_query=1&keywords=156.241.3.0%2F24&-T+option=&ip_option=&-i+option=

These objects havent been deleted.

An in-depth check of the routing status of 156.241.3.0/24 shows the
followings:

Hurricane Electric BGP tool

https://bgp.he.net/net/156.241.3.0/24

Origin AS  Announcement           Description AS37353
156.241.3.0/24 IRR Valid  HONGKONG LINK INFINITY TECHNOLOGY LIMITED


RIPESTAT

https://stat.ripe.net/widget/looking-glass#w.resource=156.241.3.0%2F24

AS37353 is seen as the origin by 1 peer.

103.200.115.1 ►103.200.115.1 is announcing route AS64271 AS9304 AS4809
AS4809 AS4809 AS134190 AS37353.

Even with the limited view, the prefix is still being originated and routed
and the route objects still exist.

Regards

--
Arnaud

Le sam. 9 mai 2020 à 09:44, Lu Heng <h.lu at anytimechinese.com> a écrit :


> To whom it may concern,

>

> On May 8, Mark Think posted a claim to multiple lists that Cloud

> Innovation was abusing an ASN (37353) that didn’t belong to them (Cloud

> Innovation) but rather belonged to Seacom through their acquisition of

> MacroLAN.

>

> While we regret this unfortunate incident, Mark’s claims that it was

> criminal or bad netizenship on the part of Cloud Innovation is without

> foundation and utterly incorrect.

>

> As shown below in the attached document from Paul Wollner(Ex-CTO of

> Macrolan who created IRR routes to allow Macrolan to announce Cloud

> Innovation's prefix); letter from Link Infinity International Ltd. (Link

> Infinity), A customer of Cloud Innovation; and attached LOA from LARUS

> authorizing IPDC Solutions to announce the prefix with origin AS134190.

> And a Letter from IPDC. This was an innocent mistake committed by third

> parties and had nothing to do with any action by Cloud Innovation or LARUS.

>

> Here’s what happened:

>

> Cloud Innovation delegated a /24 to Link Infinity, an ISP in December 2019.

>

>

> Link Infinity further delegated that same /24 to IPDC and asked Cloud

> innovation to issue an LOA, which we did. The LOA specifically required

> IPDC to use its own ASN to announce the space (AS134190).

>

> IPDC subsequently authorized one of its customers to use the said prefix.

>

>

> For reasons still unknown to Cloud Innovation, IPDC and their customer set

> up a BGP session wherein their customer used AS37353 as the origin to

> advertise prefix 156.241.3.0/24.

>

>

> Upon discovering the announcement, rather than contact Cloud Innovation,

> Mark contacted IPDC who provided him with an incomplete explanation blaming

> their customer and Mark, not realizing that Cloud Innovation was not the

> customer in question posted far and wide about the event. It is unclear to

> us why he chose to do this rather than contact us to try and resolve the

> issue.

>

>

> A contributing factor to the erroneous BGP configuration by IPDC's

> customer may have been data contained in some outdated IRR route objects

> for 156.241.0.0/16 which have subsequently been deleted.

>

> As soon as we became aware of the problem (via Mark’s email), we began to

> investigate the situation. As soon as it was clear that this was the result

> of third-party actions, we reached out to Mark privately to let him know

> what we knew and that we were still investigating. We delayed making a

> public statement in order to try and ascertain all of the facts of the

> situation. We prefer not to make public statements based on incomplete

> information.

>

> We apologize to the community for our small part in this unfortunate

> incident and assure you that we work very hard to remain good netizens and

> will address any concerns promptly when they come to our attention.

>

>

> Sincerely,

>

> Lu Heng

> CEO

> Cloud Innovations

>

> Attached:

> 1. Letter from Paul Wollner

> 2. Letter from Link Infinity

> 3. LOA Issued to IPDC Solutions for announcing 156.241.3.0/24 from

> AS134190

>         4.     Letter from IPDC

>

> FYI: LARUS is proving IP management service for Cloud Innovation, while

> LARUS is also providing IP management service to other third parties in all

> regions, for full disclosure, LARUS and Cloud Innovation are headed by same

> CEO.

>

> Content of those letters have been posted here for your convince:

>

> *IPDC:*

>

> FOR IMMEDIATE RELEASE [Perusal of Cloud Innovation Ltd]

>

>

> IPDC Solutions Pte Ltd (IPDC) is a customer of Cloud Innovation Ltd and

> over the course of our corporate relationship we were given the authority

> to use address block 156.241.3.0/24 since 9th December 2019.

>

>

> On 12th December 2019, we have delegated that address block to our client.

> Following which our client further instructed us to announce the prefix

> with AS37353. In good will after minimal verification through WHOIS’ IP

> Prefix we have proceeded to execute our client’s request.

>

>

> On 7th May 2020 IPDC was then contacted by SEACOM, the legitimate holder

> of record for that ASN and have since learned that the customer’s use of

> that ASN was erroneous and not permitted by SEACOM and immediate action has

> been taken to rectify this situation.

>

>

> IPDC would like to apologize for our lack of attention in conducting

> thorough verification and wish to highlight that the involvement of Cloud

> Innovation Ltd in the entire process was providing the addresses to us and

> that we truly apologize as we understand that this incident may have

> indirectly implicated Cloud Innovation Ltd.

>

>

> IPDC however, does not wish to disclose our client information and our

> client information shall remain confidential under present circumstances.

> Once again, we apologize for our shortcomings.

>

>

> *Link Infinity:*

>

>

> To whom it may concern,

>

> We at HK Infinity International Ltd are a customer of Cloud Innovation and

> in the course received rights to use 156.241.3.0/24 from them. Beginning

> December, 2019, we delegated the right to announce this prefix to IPDC

> Solutions Pte Ltd. (IPDC). We asked Cloud Innovation to provide an LOA

> authorizing them to announce the space which was subsequently received.

> IPDC subsequently and without our knowledge delegated this space to one of

> their customers and allowed them to originate it from AS37353.

>

> This announcement was not authorized by us, nor is it permitted by the LOA

> provided by Cloud Innovation.

>

> Unfortunately, we were not aware of the situation until after it had

> already been resolved.

>

> It was never our intent to violate the LOA or to allow the prefix to be

> announced from a hijacked ASN. We do not condone this and apologize

> sincerely to the community for what has happened here.

>

> Sincere Apologies,

>

>

> *Paul Wollner:*

>

>

> 8 May 2020

>

> TO WHOM IT MAY CONCERN

>

> In the light of the recent email on NAPAfrica mailing list, I would just

> like to clear the air.

>

>

> The IP route objects were created by myself for Cloud Innovation when they

> signed up as a client of Macrolan ( now SEACOM) as they didn't have their

> own AS.

>

> At the time I was working for Macrolan (now SEACOM). I left the employment

> of SEACOM in October 2019.

>

> It appears that when Cloud Innovation's contract with SEACOM came to an

> end, the route objects were never cleaned up.

>

> This occurred after I left SEACOM's employment. Since leaving, I have no

> access to these objects.

>

> Regards

>

> Paul Wollner

> 082-786-9776

> To whom it may concern,

>

> On May 8, Mark Think posted a claim to multiple lists that Cloud

> Innovation was abusing an ASN (37353) that didn’t belong to them (Cloud

> Innovation) but rather belonged to Seacom through their acquisition of

> MacroLAN.

>

> While we regret this unfortunate incident, Mark’s claims that it was

> criminal or bad netizenship on the part of Cloud Innovation is without

> foundation and utterly incorrect.

>

> As shown below in the attached document from Paul Wollner(Ex-CTO of

> Macrolan who created IRR routes to allow Macrolan to announce Cloud

> Innovation's prefix); letter from Link Infinity International Ltd. (Link

> Infinity), A customer of Cloud Innovation; and attached LOA from LARUS

> authorizing IPDC Solutions to announce the prefix with origin AS134190.

> And a Letter from IPDC. This was an innocent mistake committed by third

> parties and had nothing to do with any action by Cloud Innovation or LARUS.

>

> Here’s what happened:

>

> Cloud Innovation delegated a /24 to Link Infinity, an ISP in December 2019.

>

>

> Link Infinity further delegated that same /24 to IPDC and asked Cloud

> innovation to issue an LOA, which we did. The LOA specifically required

> IPDC to use its own ASN to announce the space (AS134190).

>

> IPDC subsequently authorized one of its customers to use the said prefix.

>

>

> For reasons still unknown to Cloud Innovation, IPDC and their customer set

> up a BGP session wherein their customer used AS37353 as the origin to

> advertise prefix 156.241.3.0/24.

>

>

> Upon discovering the announcement, rather than contact Cloud Innovation,

> Mark contacted IPDC who provided him with an incomplete explanation blaming

> their customer and Mark, not realizing that Cloud Innovation was not the

> customer in question posted far and wide about the event. It is unclear to

> us why he chose to do this rather than contact us to try and resolve the

> issue.

>

>

> A contributing factor to the erroneous BGP configuration by IPDC's

> customer may have been data contained in some outdated IRR route objects

> for 156.241.0.0/16 which have subsequently been deleted.

>

> As soon as we became aware of the problem (via Mark’s email), we began to

> investigate the situation. As soon as it was clear that this was the result

> of third-party actions, we reached out to Mark privately to let him know

> what we knew and that we were still investigating. We delayed making a

> public statement in order to try and ascertain all of the facts of the

> situation. We prefer not to make public statements based on incomplete

> information.

>

> We apologize to the community for our small part in this unfortunate

> incident and assure you that we work very hard to remain good netizens and

> will address any concerns promptly when they come to our attention.

>

>

> Sincerely,

>

> Lu Heng

> CEO

> Cloud Innovations

>

> Attached:

> 1. Letter from Paul Wollner

> 2. Letter from Link Infinity

> 3. LOA Issued to IPDC Solutions for announcing 156.241.3.0/24 from

> AS134190

>         4.     Letter from IPDC

>

> FYI: LARUS is proving IP management service for Cloud Innovation, while

> LARUS is also providing IP management service to other third parties in all

> regions, for full disclosure, LARUS and Cloud Innovation are headed by same

> CEO.

>

> Content of those letters have been posted here for your convince:

>

> *IPDC:*

>

> FOR IMMEDIATE RELEASE [Perusal of Cloud Innovation Ltd]

>

>

> IPDC Solutions Pte Ltd (IPDC) is a customer of Cloud Innovation Ltd and

> over the course of our corporate relationship we were given the authority

> to use address block 156.241.3.0/24 since 9th December 2019.

>

>

> On 12th December 2019, we have delegated that address block to our client.

> Following which our client further instructed us to announce the prefix

> with AS37353. In good will after minimal verification through WHOIS’ IP

> Prefix we have proceeded to execute our client’s request.

>

>

> On 7th May 2020 IPDC was then contacted by SEACOM, the legitimate holder

> of record for that ASN and have since learned that the customer’s use of

> that ASN was erroneous and not permitted by SEACOM and immediate action has

> been taken to rectify this situation.

>

>

> IPDC would like to apologize for our lack of attention in conducting

> thorough verification and wish to highlight that the involvement of Cloud

> Innovation Ltd in the entire process was providing the addresses to us and

> that we truly apologize as we understand that this incident may have

> indirectly implicated Cloud Innovation Ltd.

>

>

> IPDC however, does not wish to disclose our client information and our

> client information shall remain confidential under present circumstances.

> Once again, we apologize for our shortcomings.

>

>

> *Link Infinity:*

>

>

> To whom it may concern,

>

> We at HK Infinity International Ltd are a customer of Cloud Innovation and

> in the course received rights to use 156.241.3.0/24 from them. Beginning

> December, 2019, we delegated the right to announce this prefix to IPDC

> Solutions Pte Ltd. (IPDC). We asked Cloud Innovation to provide an LOA

> authorizing them to announce the space which was subsequently received.

> IPDC subsequently and without our knowledge delegated this space to one of

> their customers and allowed them to originate it from AS37353.

>

> This announcement was not authorized by us, nor is it permitted by the LOA

> provided by Cloud Innovation.

>

> Unfortunately, we were not aware of the situation until after it had

> already been resolved.

>

> It was never our intent to violate the LOA or to allow the prefix to be

> announced from a hijacked ASN. We do not condone this and apologize

> sincerely to the community for what has happened here.

>

> Sincere Apologies,

>

>

> *Paul Wollner:*

>

>

> 8 May 2020

>

> TO WHOM IT MAY CONCERN

>

> In the light of the recent email on NAPAfrica mailing list, I would just

> like to clear the air.

>

>

> The IP route objects were created by myself for Cloud Innovation when they

> signed up as a client of Macrolan ( now SEACOM) as they didn't have their

> own AS.

>

> At the time I was working for Macrolan (now SEACOM). I left the employment

> of SEACOM in October 2019.

>

> It appears that when Cloud Innovation's contract with SEACOM came to an

> end, the route objects were never cleaned up.

>

> This occurred after I left SEACOM's employment. Since leaving, I have no

> access to these objects.

>

> Regards

>

> Paul Wollner

> 082-786-9776

>

> On Fri, 8 May 2020 at 22:08, Mark Tinka <mark.tinka at seacom.mu> wrote:

>

>> Hi all.

>>

>> I'm not one to b**ch & moan in public, but per subject, I could not let

>> this one slide.

>>

>> And if you get this from multiple mailing lists, apologies for that - I'm

>> just trying to make sure that this reaches as wide an audience as possible,

>> on the continent.

>>

>> SEACOM (AS37100) acquired MacroLan (AS37353) a couple of years ago.

>> MacroLan is now part of the SEACOM family, and while we are in the process

>> of integrating that network into AS37100, some existing services continue

>> to be delivered on AS37353 until that exercise is completed.

>>

>> One of the customers that AS37353 was providing services to was Cloud

>> Innovation, in Cape Town. From a routing perspective, because Cloud

>> Innovation had no AS number for this service, all of their IP address space

>> was being originated by AS37353, on their behalf.

>>

>> In December of 2019, AS37353 ceased providing services to Cloud

>> Innovation. That is 6 months ago.

>>

>> In recent days, it came to SEACOM's attention that some of Cloud

>> Innovation's IP address space was being originated by AS37353 -

>> specifically, 156.241.3.0/24 - even though we were sure that they were

>> no longer a customer of AS37353 since December of 2019. At first, we

>> thought this was a cached entry in a number of popular looking glasses, but

>> then every looking glass had the same entry, which made this an unlikely

>> bug.

>>

>> As of yesterday afternoon, see below what Telia's looking glass was

>> saying about this prefix:

>>

>> *****

>>

>> Path #1: Received by speaker 0

>>   4809 134190 37353

>>     2.255.249.42 (metric 2134) from 2.255.253.101 (80.91.242.40)

>>       Origin incomplete, localpref 200, valid, internal, best,

>> group-best, import-candidate

>> Communities:

>>

>> 1299:431

>>     (RPKI state Unknown)

>>

>> 1299:1000 1299:30000 1299:30600 23456:20413 45352:4500 45352:9204

>>

>> *****

>>

>> But when I run a traceroute from my house to that prefix, it clearly was

>> not ending up in Cape Town, where AS37353's main operation resides:

>>

>> *****

>>

>> MacBook-Pro-7:~ tinka$ traceroute -I 156.241.3.1

>> traceroute to 156.241.3.1 (156.241.3.1), 64 hops max, 72 byte packets

>>  1  172.16.0.254 (172.16.0.254)  14.824 ms  11.522 ms  3.525 ms

>>  2  xe-1-3-0-1064.er-01-jnb.za.seacomnet.com (105.22.37.13)  5.620 ms

>> 9.714 ms  9.887 ms

>>  3  ce-0-2-0-0.cr-02-jnb.za.seacomnet.com (105.16.28.2)  175.232 ms

>> 172.699 ms  175.896 ms

>>  4  xe-0-0-0-8.cr-02-cpt.za.seacomnet.com (105.16.9.182)  164.496 ms

>> 163.578 ms  163.546 ms

>>  5  105.16.14.153 (105.16.14.153)  169.812 ms  171.272 ms  177.115 ms

>>  6  xe-0-0-0-0.br-02-lhr.uk.seacomnet.com (105.16.34.253)  168.911 ms

>> 172.958 ms  165.165 ms

>>  7  82.112.115.169 (82.112.115.169)  172.700 ms  176.482 ms  174.375 ms

>>  8  ae-17.r05.londen12.uk.bb.gin.ntt.net (129.250.2.147)  672.099 ms

>> 613.617 ms  615.109 ms

>>  9  ae-2.r24.londen12.uk.bb.gin.ntt.net (129.250.4.244)  181.952 ms

>> 183.087 ms  187.302 ms

>> 10  ae-16.r20.frnkge13.de.bb.gin.ntt.net (129.250.3.13)  190.511 ms

>> 185.579 ms  187.058 ms

>> 11  ae-3.r20.sngpsi07.sg.bb.gin.ntt.net (129.250.4.17)  520.882 ms

>> 613.982 ms  615.275 ms

>> 12  ae-9.r24.tkokhk01.hk.bb.gin.ntt.net (129.250.7.67)  612.301 ms

>> 586.886 ms  436.711 ms

>> 13  ae-1.r03.tkokhk01.hk.bb.gin.ntt.net (129.250.6.98)  614.470 ms

>> 613.416 ms  614.281 ms

>> 14  ce-0-3-0-3.r03.tkokhk01.hk.ce.gin.ntt.net (203.131.241.126)  614.128

>> ms  613.661 ms  615.416 ms

>> 15  * *     *

>> 16  * * *

>> 17  156.241.3.1 (156.241.3.1)  494.400 ms  410.180 ms *

>> MacBook-Pro-7:~ tinka$

>>

>> *****

>>

>> So we, then, realized that this was a fraudulent use of MacroLan's

>> AS37353, to which we had given no express permission.

>>

>> As luck would have it, due to my days living and working in Malaysia, I

>> know the good folk that operate AS134190 (IPDC Solutions), who was the

>> upstream providing transit for this prefix. So I rang them up yesterday

>> afternoon, told them what was happening, and within the hour, they got that

>> eBGP session shutdown. I am most grateful to them for their quick response

>> and immediate understanding of what was going on. Even with all the

>> technology we have today, it, many times, comes down to having good friends

>> in good places.

>>

>> Anyway, it turns out the ISP that had acquired this prefix from Cloud

>> Innovation is based in Manila, Philippines. When IPDC delivered their

>> transit service to them in Manila, that ISP informed them that they should

>> use AS37353 for the eBGP session. Yes, one could argue that IPDC should

>> have done their checks to ensure that the AS number being provided by their

>> customer belongs to them, but to be honest, I'm not too bothered about that

>> compared to Cloud Innovation's thinking that it was okay to use another

>> network's AS number in order to deliver their services to their customers.

>>

>> IPDC confirm that this service was activated for the Manila ISP in

>> December of 2019, right around the time Cloud Innovation's service with

>> SEACOM, in Cape Town, ended.

>>

>> As it currently stands, the ISP in Manila is now off the Internet, with

>> some 12 paying customers currently without service. Their excuse - they do

>> not have an AS number of their own.

>>

>> IPDC tried to find out why the ISP in Manila thought that it was okay to

>> use another network's AS number for their service, and as it turns out, the

>> network engineer at the Manila ISP that set this up has since left the

>> company. All the ones currently there do not have any history about all of

>> this.

>>

>> Currently, 156.241.3.0/24 is not in the global BGP table:

>>

>> *****

>>

>> lg-01-ams.nl>sh ip bgp 156.241.3.0/24

>> % Network not in table

>> lg-01-ams.nl>

>>

>> lg-01-nbo.ke>sh ip bgp 156.241.3.0/24

>> % Network not in table

>> lg-01-nbo.ke>

>>

>> lg-01-cpt.za>sh ip bgp 156.241.3.0/24

>> % Network not in table

>> lg-01-cpt.za>

>>

>> *****

>>

>> That Cloud Innovation thought it was okay for them to use MacroLan's AS

>> number to originate their own prefixes into the global BGP is as morally

>> reprehensible as it is technologically distasteful.

>>

>> SEACOM have been working very closely with AFRINIC to delete previous

>> route objects from their IRR that certify a relationship between Cloud

>> Innovation's parent /16 aggregates that cover this prefix, and AS37353, but

>> this is one of those objects that cannot be removed via the MyAFRINIC

>> portal, and requires manual intervention from AFRINIC.

>>

>> I have not, personally, spoken to the proprietors of Cloud Innovation

>> and/or Outside Heaven, as I don't see how anything could explain this with

>> any degree of justification.

>>

>> For now, I will find some beer to wipe the foul taste from my mouth,

>> while we (SEACOM) consider what to do about this.

>>

>> And yes, for those who are wondering, RPKI's ROV would not have prevented

>> this, in its current form. This is AS hijacking, and ROV, today, tries to

>> solve the prefix-hijacking problem, first.

>>

>> Thank you for your attention.

>>

>> Mark.

>> _______________________________________________

>> Community-Discuss mailing list

>> Community-Discuss at afrinic.net

>> https://lists.afrinic.net/mailman/listinfo/community-discuss

>>

>

>

> --

> --

> Kind regards.

> Lu

>

> _______________________________________________

> Community-Discuss mailing list

> Community-Discuss at afrinic.net

> https://lists.afrinic.net/mailman/listinfo/community-discuss

>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.afrinic.net/pipermail/rpd/attachments/20200528/bcd6f84e/attachment-0001.html>