RIPE Atlas Tools (Magellan)¶
The official command-line client for RIPE Atlas.
Why This Exists¶
[RIPE Atlas](https://atlas.ripe.net) is a powerful Internet measurements platform that until recently was only accessible via the website and the RESTful API. The reality however is that a great many people using RIPE Atlas are most comfortable on the command-line, so this project is an attempt to fill that gap.
Contents:
Requirements & Installation¶
This is a Linux-based tool, though it should work just fine in a BSD variant.
Windows is definitely not supported. In terms of the actual installation,
only Python’s package manager (pip) is currently supported, and the
installation process may require some system packages to be installed in order
for everything to work.
System Requirements¶
Some of the dependencies need to be compiled, so you’ll need a compiler on your system, as well as the development libraries for Python. In the Linux world, this typically means a few packages need to be installed from your standard package manager, but in true Linux fashion, each distribution does things slightly differently.
The most important thing to know is that you need Python 2.7 or 3. Python 2.6 will never be supported because it’s old, ugly, and needs to die.
Distribution Specific Requirements¶
Debian/Ubuntu¶
The following has been tested on Debian Jessie.
Debian-based distributions require two system packages to be installed first:
# apt-get install python-dev libffi-dev
You’ll also need either virtualenv (recommended), or if you’re not
comfortable with that, at the very least, you’ll need pip:
# apt-get install python-virtualenv python-pip
CentOS¶
This following has been tested on CentOS 7.
Since we require Python’s pip, we first need to install the epel-release
repository:
# yum install epel-release
You’ll also need the following system libraries:
# yum install gcc libffi-devel openssl-devel
Once that’s finished, you’ll need access to virtualenv (recommended), or if
you’re not comfortable with that, at the very least, you’ll need pip:
# yum install python-virtualenv python-pip
Gentoo¶
If you’re a Gentoo user, you never have to worry about development libraries, but if you intend to use the bleeding-edge version of this package (and what self-respecting Gentoo user wouldn’t?) then you’ll probably want to make sure that git is built with curl support:
# USE="curl" emerge git
If you’re not going bleeding edge, or if you’re just going to use SSH to get the code from GitHub, then Gentoo will have everything ready for you.
Apple OSX¶
These instructions expect that you’ve got Python’s pip installed, so if you
have no idea what that is, or simply don’t have it yet, you should be able to
install pip with one easy command (as root):
# easy_install pip
Outside of that, a few of the Python dependencies require that you have a compiler on your system. For this, you need only get a free copy of Xcode from the app store, and from there you should be good to go.
Python Requirements¶
Importantly, Magellan requires Python 2.7 or higher. For most desktop users, this shouldn’t be a problem, but for some older servers like CentOS 6 and lower, this may cause some pain. Thankfully, for most such systems, there are usually work-arounds that allow you to install a more modern version of Python in parallel.
Magellan depends on two other RIPE Atlas libraries, Cousteau and Sagan, which in
turn depend on a reasonable number of Python libraries. Thankfully, Python’s
package manager, pip should handle all of these for you:
- ripe.atlas.cousteau
- ripe.atlas.sagan
- tzlocal
- pyyaml
Installation¶
From PyPi¶
Python’s pip program can be used to install packages globally (not a good
idea since it conflicts with your system package manager) or on a per-user
basis. Typically, this is done with virtualenv, but if you don’t want to use
that, you can always pass --user to the pip program and it’ll install a
user-based copy in ${HOME}/.local/.
# From within a virtualenv
$ pip install ripe.atlas.tools
# In your user's local environment
$ pip install --user ripe.atlas.tools
Or if you want to live on the edge and perhaps try submitting a pull request of your own:
One day, we want this process to be as easy as installing any other command-line
program, that is, with apt, dfn, or emerge, but until that day,
Python’s standard package manager, pip does the job nicely.
From GitHub¶
If you’re feeling a little more daring and want to go bleeding-edge and use
our master branch on GitHub, you can have pip install right from there::
$ pip install git+https://github.com/RIPE-NCC/ripe.atlas.tools.git
If you think you’d like to contribute back to the project, we recommend the use
of pip’s -e flag, which will place the Magellan code in a directory where
you can edit it, and see the results without having to go through a new install
procedure every time. Simply clone the repo on GitHub and install it like so::
$ pip install -e git+https://github.com/your-username/ripe.atlas.tools.git
From a Tarball¶
If for some reason you want to just download the source and install it manually,
you can always do that too. Simply un-tar the file and run the following in the
same directory as setup.py.:
$ python setup.py install
Troubleshooting¶
If you’re using Mac OSX, the installation of Sagan, one of Magellan’s dependencies may give you trouble, especially in how Apple handles PyOpenSSL on their machines. Workarounds and proper fixes for this issue can be found in the Sagan installation documentation.
Quickstart¶
This is a very fast break down of everything you need to start using Ripe Atlas on the command line. Viewing public data is quick & easy, while creation is a little more complicated, since you need to setup your authorisation key.
Viewing Public Data¶
- Install the toolkit.
- View help with:
ripe-atlas --help - View a basic report for a public measurement:
ripe-atlas report <measurement_id> - View the live stream for a measurement:
ripe-atlas stream <measurement_id> - Get a list of probes in ASN 3333:
ripe-atlas probes --asn 3333 - Get a list of measurements with the word “wikipedia” in them:
ripe-atlas measurements --search wikipedia
Creating a Measurement¶
- Log into RIPE Atlas. If you don’t have an account, you can create one there for free.
- Visit the API Keys page and create a new key
with the permission
Create a new user defined measurement - Install the toolkit as below.
- Configure the toolkit to use your key with
ripe-atlas configure --set authorisation.create=MY_API_KEY - View the help for measurement creation with
ripe-atlas measure --help - Create a measurement with
ripe-atlas measure ping --target example.com
Advanced Use¶
Refer to the complete usage documentation for more advanced options.
How to Use the RIPE Atlas Toolkit¶
Configure¶
For most features, Magellan will work out-of-the-box, but if you’d like to customise the experience, or if you want to use this tool to create a measurement of your own, then you’ll need to configure it.
Thankfully, configuration is easy by way of the configure command::
$ ripe-atlas configure --help
Options¶
| Option | Arguments | Explanation |
|---|---|---|
--editor |
Invoke ${EDITOR} to edit the configuration directly | |
--set |
path=value | Permanently set a configuration value so it can be used in the future. |
--init |
Create a configuration file and save it
into your home directory at:
${HOME}/.config/ripe-atlas-tools/rc |
Examples¶
Create a standard configuration file. Note that this typically isn’t necessary:
$ ripe-atlas configure --init
Invoke your editor of choice to manually fiddle with the configuration file:
$ ripe-atlas configure --editor
Set an arbitrary value within the configuration file. You can use dot-separated notation to dictation the value you wish to change:
$ ripe-atlas configure --set authorisation.create=YOUR_API_KEY
Quick Measurement Information (go)¶
For the impatient, and for those looking to see how they might write their own
plugins, we have a simple go command::
$ ripe-atlas go <measurement-id>
This will open a web browser and take you to the detail page for the measurement id provided.
Measure¶
The most complicated command we have, this will create a measurement (given a plethora of options) and begin streaming the results back to you in a standardised rendered form.
It’s invoked by using a special positional argument that dictates the type of measurement you want to create. This also unlocks special options, specific to that type. See the examples for more information.
Options¶
All measurements share a base set of options.
| Option | Arguments | Explanation |
|---|---|---|
--renderer |
One of: dns, http, ntp, ping, raw, ssl_consistency, sslcert, traceroute, traceroute_aspath, aggregate_ping | The renderer you want to use. If this isn’t defined, an appropriate renderer will be selected. |
--dry-run |
Do not create the measurement, only show its definition. | |
--auth |
An API key | The API key you want to use to create the measurement. |
--af |
One of: 4, 6 | The address family, either 4 or 6. The default is a guess based on the target, favouring 6. |
--description |
A free-form string | The description/name of your new measurement. |
--target |
A domain or IP | The target, either a domain name or IP address. If creating a DNS measurement, the absence of this option will imply that you wish to use the probe’s resolver. |
--no-report |
Don’t wait for a response from the measurement, just return the URL at which you can later get information about the measurement. | |
--interval |
An integer | Rather than run this measurement as
a one-off (the default), create this
measurement as a recurring one, with
an interval of n seconds between
attempted measurements. This option
implies --no-report. |
--from-area |
One of: WW, West, North-Central, South-Central, North-East, South-East | The area from which you’d like to select your probes. |
--from-country |
A two-letter ISO country code | The country from which you’d like to select your probes. |
--from-prefix |
A prefix string | The prefix from which you’d like to select your probes. |
--from-asn |
An ASN number | The ASN from which you’d like to select your probes. |
--from-probes |
A comma-separated list of probe ids | Probes you want to use in your measurement. |
--from-measurement |
A measurement id | A measurement id which you want to use as the basis for probe selection in your new measurement. This is a handy way to re-create a measurement under conditions similar to another measurement. |
--probes |
An integer | The number of probes you want to use. |
--include-tag |
A tag name | Include only probes that are marked with this tag. Note that this option may be repeated. |
--exclude-tag |
A tag name | Exclude probes that are marked with this tag. Note that this option may be repeated. |
Ping-Specific Options¶
| Option | Arguments | Explanation |
|---|---|---|
--packets |
An integer | The number of packets sent |
--size |
An integer | The size of packets sent |
--packet-interval |
An integer |
Traceroute-Specific Options¶
| Option | Arguments | Explanation |
|---|---|---|
--packets |
An integer | The number of packets sent |
--size |
An integer | The size of packets sent |
--protocol |
One of: ICMP, UDP, TCP | The protocol used. For DNS measurements, this is limited to UDP and TCP, but traceroutes may use ICMP as well. |
--timeout |
An integer | The timeout per-packet |
--dont-fragment |
Don’t Fragment the packet | |
--paris |
An integer | Use Paris. Value must be between 0 and 64.If 0, a standard traceroute will be performed. |
--first-hop |
An integer | Value must be between 1 and 255. |
--max-hops |
An integer | Value must be between 1 and 255. |
--port |
An integer | Destination port, valid for TCP only. |
--destination-option-size |
An integer | IPv6 destination option header. |
--hop-by-hop-option-size |
An integer | IPv6 hop by hop option header. |
DNS-Specific Options¶
| Option | Arguments | Explanation |
|---|---|---|
--query-class |
One of: IN, CHAOS | The query class. The default is “IN” |
--query-type |
One of: A, SOA, TXT, SRV, SSHFP, TLSA, NSEC, DS, AAAA, CNAME, DNSKEY, NSEC3, PTR, HINFO, NSEC3PARAM, NS, MX, RRSIG, ANY | The query type. The default is “A” |
--query-argument |
A string | The DNS label to query. |
--set-cd-bit |
Set the DNSSEC Checking Disabled flag (RFC4035) | |
--set-do-bit |
Set the DNSSEC OK flag (RFC3225) | |
--set-nsid-bit |
Include an EDNS name server. ID request with the query. | |
--udp-payload-size |
An integer | May be any integer between 512 and 4096 inclusive. |
--set-rd-bit |
Set the Recursion Desired flag. | |
--retry |
An integer | Number of times to retry. |
Examples¶
The simplest of measurements. Create a ping with 50 probes to example.com:
$ ripe-atlas measure ping --target example.com
The same, but don’t actually create it, just show what would be done:
$ ripe-atlas measure ping --target example.com --dry-run
Be more specific about which address family you want to target:
$ ripe-atlas measure ping --target example.com --af 6
Ask for 20 probes from Canada:
$ ripe-atlas measure ping --target example.com --probes 20 --from-country ca
Or ask for 20 Canadian probes that definitely support IPv6:
$ ripe-atlas measure ping --target example.com --probes 20 \
--from-country ca --include-tag system-ipv6-works
Rather than creating a one-off create a recurring measurement:
$ ripe-atlas measure ping --target example.com --interval 3600
Moving onto DNS measurements, do a lookup for example.com. Since we’re not
specifying --target here, this implies that we want to use the probe’s
resolver:
$ ripe-atlas measure dns --query-argument example.com
Getting a little more complicated, let’s set a few special bits and make a more complex query:
$ ripe-atlas measure dns --query-type AAAA --query-argument example.com \
--set-nsid-bit --set-rd-bit --set-do-bit --set-cd-bit
Measurements¶
A querying tool for finding existing measurements in the RIPE Atlas database. You can request a table-formatted list of measurements based on search-string lookups, type, start time, etc.
Options¶
| Option | Arguments | Explanation |
|---|---|---|
--search |
A free-form string | This could match the target or description. |
--status |
One of: scheduled, stopped, ongoing | The measurement status. |
--af |
One of: 4, 6 | The address family. |
--type |
One of: ping, traceroute, dns, sslcert, ntp, http | The measurement type. |
--field |
One of: status, target, url, type, id, description | The field(s) to display. Invoke multiple times for multiple fields. The default is id, type, description, and status. |
--ids-only |
Display a list of measurement ids matching your filter criteria. | |
--limit |
An integer | The number of measurements to return. The number must be between 1 and 1000 |
| –started-before | An ISO timestamp | Filter for measurements that started before a specific date. The format required is YYYY-MM-DDTHH:MM:SS |
| –started-after | An ISO timestamp | Filter for measurements that started after a specific date. The format required is YYYY-MM-DDTHH:MM:SS |
| –stopped-before | An ISO timestamp | Filter for measurements that stopped before a specific date. The format required is YYYY-MM-DDTHH:MM:SS |
| –stopped-after | An ISO timestamp | Filter for measurements that stopped after a specific date. The format required is YYYY-MM-DDTHH:MM:SS |
Examples¶
Get a list of measurements:
$ ripe-atlas measurements
Filter that list by status=ongoing:
$ ripe-atlas measurements --status ongoing
Further filter it by getting measurements that conform to IPv6:
$ ripe-atlas measurements --status ongoing --af 6
Get that same list, but strip out everything but the measurement ids:
$ ripe-atlas measurements --status ongoing --af 6 --ids-only
Limit that list to 200 entries:
$ ripe-atlas measurements --status ongoing --af 6 --limit 200
Get that list, but show only the id, url and target fields:
- $ ripe-atlas measurements –status ongoing –af 6
- –field id –field url –field target
Filter for measurements of type dns that started after January 1, 2015:
$ ripe-atlas measurements --type dns --started-after 2015-01-01
Probes¶
Just like the measurements command, but for probes, and a lot more powerful.
You can use this command to find probes within an ASN, prefix, or geographical
region, and then aggregate by country, ASN, and/or prefix.
Options¶
| Option | Arguments | Explanation |
|---|---|---|
--limit |
An integer | Return limited number of probes. |
--field |
One of: status, description, address_v6, address_v4, asn_v4, is_public, asn_v6, id, prefix_v4, prefix_v6, is_anchor, country, coordinates | The field(s) to display. Invoke multiple times for multiple fields. The default is id, asn_v4, asn_v6, country, and status. |
--aggregate-by |
country, asn_v4, asn_v6, prefix_v4, prefix_v6 | Aggregate list of probes based on all specified aggregations. Multiple aggregations supported. |
--all |
Fetch ALL probes. That will give you a loooong list. | |
--max-per-aggregation |
An integer | Maximum number of probes per aggregated bucket. |
--ids-only |
Print only IDs of probes. Useful to pipe it to another command. | |
--asn |
An integer | Filter the list by an ASN |
--asnv4 |
An integer | Filter the list by an ASN |
--asnv6 |
An integer | Filter the list by an ASN |
--prefix |
A prefix string | Filter the list by a prefix |
--prefixv4 |
A prefix string | Filter the list by a prefix |
--prefixv6 |
A prefix string | Filter the list by a prefix |
--location |
A free-form string | The location of probes as a string i.e. ‘Amsterdam’ |
--center |
A pair of geographic coordinates | Location as <lat>,<lon>-string, i.e. “48.45,9.16” |
--radius |
An integer | Radius in km from specified center/point. |
--country |
A two-letter ISO country code | The country in which the probes are located. |
Examples¶
Get a list of probes within ASN 3333:
$ ripe-atlas probes --asn 3333
Further filter that list to show only probes in ASN 3333 from the Netherlands:
$ ripe-atlas probes --asn 3333 --country nl
Change the limit from the default of 25 to 200:
$ ripe-atlas probes --asn 3333 --limit 200
Aggregate the probes by country, and then by ASN:
$ ripe-atlas probes --asn 3333 --aggregate-by country --aggregate-by asn
Show the id, url, target, description, and whether the probe is public or not:
$ ripe-atlas probes --asn 3333 --field id --field url --field description \
--field is_public
Render¶
Sometimes you already have a large collection of measurement results and you
just want Magellan to render them nicely for you. In these cases, render is
your friend.
You can use the --renderer flag to target specific renderers too if the
default isn’t enough for you.
Options¶
| Option | Arguments | Explanation |
|---|---|---|
--renderer |
One of: dns, http, ntp, ping, raw, ssl_consistency, sslcert, traceroute, traceroute_aspath, aggregate_ping | The renderer you want to use. If this isn’t defined, an appropriate renderer will be selected. |
--probes |
A comma-separated list of probe ids | Limit the results to those returned from specific probes |
--from-file |
A file path | The source of the data to be rendered. If nothing is specified, we assume “-” or, standard in (the default). |
--aggregate-by |
One of: dns, ntp, http, ping, raw, ssl_consistency, sslcert, traceroute, traceroute_aspath, aggregate_ping, | Tell the rendering engine to aggregate the results by the selected option. Note that if you opt for aggregation, no output will be generated until all results are received, and if large data sets may explode your system. |
Examples¶
Pipe the contents of an arbitrary file file into the renderer. The rendering engine will be guessed from the first line of input:
$ cat /path/to/file/full/of/results | ripe-atlas render
The same, but point Magellan to a file deliberately rather than using a pipe:
$ ripe-atlas render --from-file /path/to/file/full/of/results
Specify a particular renderer:
$ cat /path/to/file/full/of/results | ripe-atlas render --renderer ping
Aggregate the output by country:
$ cat /path/to/file/full/of/results | ripe-atlas render --aggregate-by country
Report¶
A means to generate a simple text-based report based on the results from a
measurement. Typically, this is used to get the latest results of a measurement
in a human-readable format, but with the --start-time and --stop-time
options, you can get results from any time range you like.
Options¶
| Option | Arguments | Explanation |
|---|---|---|
--probes |
A comma-separated list of probe ids | Limit the report to only results obtained from specific probes. |
--renderer |
One of: dns, http, ntp, ping, raw, ssl_consistency, sslcert, traceroute, traceroute_aspath, aggregate_ping | The renderer you want to use. If this isn’t defined, an appropriate renderer will be selected. |
--aggregate-by |
One of: status, prefix_v4, prefix_v6, country, rtt-median, asn_v4, asn_v6 | Tell the rendering engine to aggregate the results by the selected option. Note that if you opt for aggregation, no output will be generated until all results are received. |
--start-time |
An ISO timestamp | The start time of the report. The format should conform to YYYY-MM-DDTHH:MM:SS |
--stop-time |
An ISO timestamp | The stop time of the report. The format should conform to YYYY-MM-DDTHH:MM:SS |
Examples¶
Get the latest results of measurement 1001:
$ ripe-atlas report 1001
The same, but specifically request the ping renderer:
$ ripe-atlas report 1001 --renderer ping
Aggregate those results by country:
$ ripe-atlas report 1001 --aggregate-by country
Get results from the same measurement, but show all results from the first week of 2015:
$ ripe-atlas report 1001 --start-time 2015-01-01 --stop-time 2015-01-07
Get results from the first day of 2015 until right now:
$ ripe-atlas report 1001 --start-time 2015-01-01
Stream¶
Connect to the streaming API and render the results in real-time as they come in.
Options¶
| Option | Arguments | Explanation |
|---|---|---|
--limit |
A number < 1000 | The maximum number of results you want
to stream. The default is to stream
forever until you hit Ctrl+C. |
--renderer |
One of: dns, http, ntp, ping, raw, ssl_consistency, sslcert, traceroute, traceroute_aspath, aggregate_ping | The renderer you want to use. If this isn’t defined, an appropriate renderer will be selected. |
Examples¶
Stream the results from measurement #1001:
$ ripe-atlas stream 1001
Limit those results to 500:
$ ripe-atlas stream 1001 --limit 500
Specify a renderer:
$ ripe-atlas stream 1001 --renderer ping
Combine for fun and profit:
$ ripe-atlas stream 1001 --renderer ping --limit 500
Troubleshooting¶
Sometimes things don’t go as planned. In these cases, this page is here to help.
InsecurePlatformWarning¶
On older systems (running Python versions <2.7.10), you may be presented with a warning message that looks like this::
/path/to/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:100:
InsecurePlatformWarning: A true SSLContext object is not available. This
prevents urllib3 from configuring SSL appropriately and may cause certain
SSL connections to fail. For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
This is due to the insecure way older versions of Python handle secure connections and a visit to the above URL will tell you that the fix is one of three options:
- Upgrade to a modern version of Python
- Install three Python packages:
pyopenssl,ndg-httpsclient, andpyasn1 - Suppress the warnings. Don’t do that though.