[DBWG] RIR "stats" file(s) software tools

[Ronald F. Guilmette]

As promised, I am providing the community with a number of the small
software tools that I have written to assist me in analyzing both
IP address allocations and ASN allocations globally and in each of
the five regions.  I retain the copyrights on these tools, but I
hereby grant permission for them to be used freely in non-commercial
contexts or for research purposes.  Anyone wishing to use any of
these tools in a commercial context should contact me before doing
so in order to obtain an appropriate license.

Note that I am providing rather substantially more in the way of tools
than I had promised.  I am doing so in the hope that I may perhaps help
people to get a at least some understanding of the full process I have
been using to find bogon route objects in various IRRs.  (I am *not*
yet providing the additional script needed to perform that separate
task however.  I will do so if others are sincerly interested in
performing their own hunts for bogon route objects.)

I will first provide links to the software tools that I am sharing today.
Following those links below I also provide some small description of each,
including usage instructions.  Note that none of these tools are in any
sense "well documented".  I *do not* have and do not provide formal man
pages for any of them.  As far as I am concerned these tools are all too
small and trivial to require or warrant that kind of serious documentation.

get-rir-stats.bash
https://pastebin.com/raw/URX0KN35

isort.pl
https://pastebin.com/raw/peeZgmXE

stats2asns.pl
https://pastebin.com/raw/SUgGNRUA

stats2ipv4cidrs.pl
https://pastebin.com/raw/A5d5iTrb

stats2ipv6cidrs.pl
https://pastebin.com/raw/VaA5Z9Uf

Range2cidrs.pm
https://pastebin.com/raw/bU2RtNdL

Note that the final link above is to a Perl library module.  You may install
that anywhere you like on your system, but the path to the containing
directory, whatever that is, must then be substituted into line 11 of the
stats2ipv4cidrs.pl script. (In my case, I place library code on my system
into the directory /usr/local/rfg/lib and I install my actual tools into
the local directory /usr/local/rfg/bin which has also been added to my
${PATH}.)

The get-rir-stats.bash script is one that I am using, via a daily cron job,
to fetch all of the daily RIR stats files.  This script also fetches the
daily NRO stats file, however only the portion of that which pertains to
IANA reserved IP address blocks and IANA reserved ASNs is used because
experience has shown that the other material therein may not be as fresh
as what is contained in the daily stats files obtained from the five RIRs
themselves.

In addition to fetching the daily stats files, the get-rir-stats.bash script
also makes use of the various Perl scripts linked to above to perform some
rather trivial textual transformations on the daily stats file.  These
transformations simply eliminate non-useful data and render what remains
into a simpler and more easily used notation and format.

Before installing, you should edit the definition of 'workdir' on line 8
of the get-rir-stats.bash script to suit you local preferences.

On a relatively low-speed DSL line, the get-rir-stats.bash script takes
approximately 3 minutes to run.  If you are running the script from the
command line and NOT as a cron job, then it is recommended that you provide
the -v option when invoking get-rir-stats.bash.  If you do so, then
get-rir-stats.bash will dutifully pass that option down to the various
Perls tools when they are invoked, and they themselves will provide more
information about any unexpected anomalies in the input data (i.e. within
the daily stats files).  Anomalies in the daily stats file *do* happen,
but these are generally due to transient strangness relating to specific
IP blocks that have just been transfered between regions and that are
thus still in a state of flux.  (These occasional anomalies usually
disappear within 24 hours.)

The get-rir-stats.bash script assumes that you have the widely used curl
program installed and in your ${PATH}.  If you do not, then you will need
to get and install that before using get-rir-stats.bash.

Other external utility programs used by get-rir-stats.bash are as follows:

    sort             -- UNIX built-in
    isort            -- Perl script to sort lines in proper IPv4 order
    stats2asns       -- Perl script to extract ASN ranges from stats files
    stats2ipv4cidrs  -- Perl script to extract IPv4 CIDRs from stats files
    stats2ipv6cidrs  -- Perl script to extract IPv6 CIDRs from stats files

With the exception of the standard UNIX "sort" program, which you should
already have installed and in your ${PATH}, the other supporting utilities
are provided at the links given above.  Note that all of these utilities
must be installed with a normal sort of execute permission (generally 0755).
Also the pathnames given in the very first line of each script will likely
need to be edited to suit your local environment, and specifically the path
to your local Perl interpreter.  (This is true also in the case of the
get-rir-stats.bash script. You may need to edit the first line of that so
that the path given there will point to your local bash interpreter.)

By default each of the stats2*.pl scripts selects out of its input file(s)
only those IP blocks or ASNs that are listed in the input file(s) as being
"assigned".  This default mode is useful when hunting bogons, since when
one is hunting bogons, one basically wants to look at a set of candidate
IPs or IP block and then filter -out- all of the stuff associated with
RIR "assigned" blocks.  Whatever is left over is bogons.

The various stats2* Perl scripts support a few common options which are
worthy of note, specifically:

      -v    -- verbosely report anomalies to stderr
      -a    -- include in the output items marked as "allocated"
      -r    -- include in the output items marked as "reserved"
      -d    -- include in the output items marked as any of the following:
		assigned, allocated, available, reserved

See the get-rir-stats.bash script for examples of how these options may be
used in practice to achieve various results.

After running the get-rir-stats.bash script you should be left with the
following set of files in your chosen ${workdir}:

    all.asns.assigned
    all.asns.delegated
    all.ipv4cidrs.assigned
    all.ipv4cidrs.delegated
    all.ipv6cidrs.assigned
    all.ipv6cidrs.delegated
    rir.asns.assigned
    rir.ipv4cidrs.assigned
    rir.ipv6cidrs.assigned
    stats.afrinic
    stats.apnic
    stats.arin
    stats.iana
    stats.lacnic
    stats.nro
    stats.ripe

The various stats.* files are the daily stats files as fetched from the
revelant sources.  Note that the stats.iana file is artificial however.
This file is composed of just the lines from the stats.nro file that
begin with "iana".  (This file is thus a lot smaller than the whole
stats.nro file, so it is esiler and faster to work with.)

The rir.*.assigned files contain summarized lists of the number resources
that have been assigned by any actual RIR to any actual resource member.
In the case of IPv4 address blocks these will have been helpfully converted
into traditional IPv4 CIDR notation.  (The contents of these files are used
in my bogon route object hunting efforts.)

The all.*.delegated files contain lists of everything that has been delegated
by IANA to the various RIRs *plus* any number resources, or blocks thereof,
that are reserved by IANA itself.  (The contents of these files are useful
for directing the actions of my own "ipw" -- IP WHOIS utility -- so that
it will direct each query about any arbitrary IP address straight to the
specific RIR WHOIS server that either has or that should have the relevant
WHOIS record, or which will return some definitive result that, in effect,
says "this IP isn't assigned to any resource member at present".)

The all.*.assigned files contain lists of everything that has been assigned
by any RIR, plus those number resources that are reserved by IANA.  (I don't
actually remember what I am using these files for anymore, or if I am even
using them at all.)

That's it.  I will try to answer questions if there are any, but please
DO NOT assume that my time is free.

One last thing. The original point of interest that prompted me to post
these tools was that some folks wanted to be able to get a nicely IPv4
CIDR-ized list of every IPv4 block that is currently under the administration
of AFRINIC, specifically, i.e. every IPv4 block that has been delegated by
IANA to AFRINIC at the present time.  Assuming that one has already run
the get-rir-stats.bash script (and thus has already fetched the various
daily stats files) then to get such a list, you should be able to just
do the following:

	cd ${workdir}
	stats2ipv4cidrs -d stats.afrinic

Output will be written to stdout.

Enjoy!


Regards,
rfg


P.S.  Note that if you add up all of the stuff in the all.ipv4cidrs.delegated
file, you get the entire IPv4 address range from 0.0.0.0 to 255.255.255.255,
with no gaps and no overlaps.  I know.  I checked.  (I do not know yet if
the same can also be said in the case of the all.ipv6cidrs.delegated file
or the all.asns.delegated file.  I haven't checked those yet.)

P.P.S.  Writing & running these tools has revealed a few glitches in things
globally.  The scripts try to correct for these errors.  For example, both
ARIN and IANA try to lay claim to AS112.  My belief is that only IANA's
claim is a valid one.

In a similar vein, both ARIN and IANA claim 192.175.48.0/24, but in my
opinion it rightly only belongs to ARIN.  (This block is *not* reserved
by any RFC as far as I know, and thus IANA should butt out.)

Lastly, APNIC has effectively tried to claim two IPv4 blocks reserved by
RFC 5737, namely 198.51.100.0/24 and 203.0.113.0/24.  But so has IANA!
In my opinion, the blocks have global meaning as laid out in RFC 5737,
and thus APNIC is in the wrong here, and the blocks effectively belong
only to IANA.

I will be notifying APNIC about that issue shortly.