Easy Network Diagnostics with SLATE

Network testing is hugely important to diagnose problems within and between sites. We’re trying to make network diagnostics dead simple with SLATE, and to do that we’ve SLATE-ified pieces of the excellent perfSONAR software stack. This application has recently landed into the SLATE stable catalog, and so we thought we would give you a little tour of this handy tool!

Let’s dive right in - here’s what the configuration looks like:

[09:31]:~ $ slate app get-conf perfsonar-testpoint
# Default values for perfsonar-testpoint.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

Instance: ''

# Whether to run only on specially marked nodes. 
# If nodeSelection is true, this service will only run on a node
# which has the `perfsonar: enabled` label applied to it. 
# Otherwise, it will allow itself to be scheduled on any node. 
nodeSelection: false

So in this particular app, there’s not much configuration. Note that this is only the perfSONAR testpoint, so it only contains the pieces needed to launch tests, rather than the entire perfSONAR suite.

The one configuration parameter we do have is nodeSelection, what’s this for? From the in-line comments, we see that perfSonar be configured to run on a dedicated node, provided that the node has a label perfsonar: enabled. You might want to do this for long-lived network measurement endpoints that will see some heavy-duty testing, where the network interface won’t be shared with other applications.

Installing it on an endpoint is simple. Let’s say we want to do a measurement between the University of Chicago and the University of Michigan. You can find the endpoints with slate cluster list and a little grep action:

[09:43]:~ $ slate cluster list | grep -E uchicago\|umich
umich-prod           slate-dev             cluster_WRb0f8mH9ak
uchicago-its-fiona01 slate-dev             cluster_gWCytq-5yaU
uchicago-prod        slate-dev             cluster_yZroQR5mfBk
uchicago-river-v2    ssl                   cluster_iL8D7abxCM8

So for this particular example we’ll use umich-prod and uchicago-prod. To stand up the endpoints, first we’ll do UMich:

[09:46]:~ $ slate app install perfsonar-testpoint --cluster umich-prod --group slate-dev
Installing application...
Successfully installed application perfsonar-testpoint as instance slate-dev-perfsonar-testpoint with ID instance_LAgguQo0KVY

We’ll also want to get its instance info, to see where it’s running:

[09:47]:~  $ slate instance info instance_LAgguQo0KVY
Name                Started                         Group     Cluster    ID                  
perfsonar-testpoint 2019-Oct-04 15:17:23.774216 UTC slate-dev umich-prod instance_LAgguQo0KVY

Services: (none)

    Status: Running
    Created: 2019-10-04T15:17:24Z
    Host: sl-um-es3.slateci.io
    Host IP:

Likewise for UChicago:

[09:48]:~ $ slate app install perfsonar-testpoint --cluster uchicago-prod --group slate-dev
Installing application...
Successfully installed application perfsonar-testpoint as instance slate-dev-perfsonar-testpoint with ID instance_Wy1saT5eVmM

[09:50]:~ $ slate instance info instance_Wy1saT5eVmM
Name                Started                         Group     Cluster       ID                  
perfsonar-testpoint 2019-Oct-04 14:47:55.102212 UTC slate-dev uchicago-prod instance_Wy1saT5eVmM

Services: (none)

    Status: Running
    Created: 2019-10-04T14:47:55Z
    Host: sl-uc-es1.slateci.io
    Host IP:

So now we know both IP addresses, and we ostensibly have perfSONAR running both places. As far as SLATE goes, that’s all you need to do to get testpoints running at two sites.

In order to see that things are actually running, we’ll need to have a copy of the pScheduler software to ask both endpoints to execute a test. I don’t have these tools on my laptop, but perfSONAR does provide a docker container that does. You’ll want to launch a version with the pscheduler daemon, and then exec into it:

[10:22]:~ $ docker run -d perfsonar/testpoint
[10:22]:~ $ docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                                                                     NAMES
ce0b0926b2ff        perfsonar/testpoint   "/bin/sh -c '/usr/..."   4 seconds ago       Up 2 seconds        443/tcp, 861-862/tcp, 5000-5001/tcp, 5101/tcp, 5201/tcp, 8760-9960/tcp, 18760-19960/tcp   festive_wright
[10:22]:~/blog $ sudo docker exec -ti ce0b0926b2ff /bin/sh

Once you have a shell, you can ask pscheduler to start a test between the endpoints:

sh-4.2# pscheduler task trace --source-node --dest
Submitting task...
Task URL:
Running with tool 'traceroute'
Fetching first run...

Next scheduled run:
Starts 2019-10-04T12:09:14-04 (~2 seconds)
Ends   2019-10-04T12:09:22-04 (~7 seconds)
Waiting for result...

1	gw-shinano.aglt2.org ( AS229 0.2 ms
	  MERIT-AS-6 - Merit Network Inc., US
2	esnet-lhc1-a-aglt2.es.net ( AS291 6.1 ms
3	uchicago-lhc1-esnet.es.net ( AS291 6.4 ms
4 AS160 6.6 ms
	  U-CHICAGO-AS - University of Chicago, US
5	sl-uc-es1.slateci.io ( AS160 6.6 ms
	  U-CHICAGO-AS - University of Chicago, US

No further runs scheduled.

Likewise, you can look at the throughput between the sites:

sh-4.2# pscheduler task throughput --source-node --dest
Submitting task...
Task URL:
Running with tool 'iperf3'
Fetching first run...

Next scheduled run:
Starts 2019-10-04T11:48:55-04 (~7 seconds)
Ends   2019-10-04T11:49:14-04 (~18 seconds)
Waiting for result...

* Stream ID 5
Interval       Throughput     Retransmits    Current Window 
0.0 - 1.0      524.95 Mbps    49             393.71 KBytes  
1.0 - 2.0      524.57 Mbps    14             348.97 KBytes  
2.0 - 3.0      526.64 Mbps    24             492.14 KBytes  
3.0 - 4.0      886.30 Mbps    13             671.10 KBytes  
4.0 - 5.0      870.31 Mbps    35             232.65 KBytes  
5.0 - 6.0      482.35 Mbps    2              760.58 KBytes  
6.0 - 7.0      1.09 Gbps      19             411.61 KBytes  
7.0 - 8.0      639.64 Mbps    22             885.85 KBytes  
8.0 - 9.0      901.76 Mbps    21             590.57 KBytes  
9.0 - 10.0     849.28 Mbps    19             796.37 KBytes  

Interval       Throughput     Retransmits    
0.0 - 10.0     729.63 Mbps    218

No further runs scheduled.

Pretty cool. We didn’t have to ask admins at either site to install perfSonar infrastructure, but we were able to schedule and run tests between both sites and learn something about the connectivity between them.

If you want to try this out for yourself, you can find the perfSONAR testpoint application in the catalog. There are many other tests you might want to run against sites, such as rtt, latency, dns and so on. You can find out more about pScheduler on the perfSONAR website.

The SLATE Team