Stitching SLATE Clusters on Chameleon

Posted on Monday, June 14 2021.

Chameleon is an OpenStack-based research platform for provisioning compute and networking resources. Chameleon has the capability to create a virtual LAN with hosts stitched over two geographically disparate sites. In this blog post, we demonstrate how to setup two SLATE clusters at separate sites, stitched together with a VLAN. We will also run various tests to determine network performance over this stitched link.

For a more basic SLATE-on-Chameleon setup, please see the Deploying a SLATE Cluster on Chameleon blog post. If you have no experience with Chameleon, we recommend that you read through and follow this blog post first to familiarize yourself with the platform.

Requirements

To run a SLATE cluster on Chameleon, you must first have access to a Chameleon account, as well as be on an existing Chameleon project. The Chameleon Getting Started Guide contains lots of useful information regarding this. Other helpful Chameleon documentation includes this page on network stitching.

Additionally, you must have a SLATE account with an access token.

Chameleon Blazar Client

Due to Chameleon limitations, a stitched network can only be created with the OpenStack CLI. Install the OpenStack/Chameleon Blazar client for creating leases. It can be downloaded with the following command:

pip install git+https://github.com/ChameleonCloud/python-blazarclient.git

It is important to be aware of different client versions, as Chameleon expects version 2.2.2 to be used.

Application Credential

You must also have an unrestricted application credential for both Chameleon sites, “CHI@UC” and “CHI@TACC”.

To do this, log into the first site’s portal.

  1. In the left sidebar, navigate to the “Identity” tab, and click “Application Credentials”.
  2. Then, click “Create Application Credential”, fill out the name field, and check the “Unrestricted” box.
  3. Click “Create Application Credential” again, and download the openrc file somewhere safe.

Then, repeat these steps for the other site.

Chameleon Setup

Network Lease

A stitch-able network lease must be created at each Chameleon site. First, authenticate with a Chameleon site by running this command:

source /path/to/openrc.sh

This openrc.sh file is the application credential you created earlier. You should have one for each site.

Then, use the following command to create a network lease:

`blazar lease-create --reservation resource_type=network,network_name=<network_name_here>,resource_properties='["==","$physical_network","exogeni"]' --start-date <start_date> --end-date "<end_date>" <lease_name>`

Note that the start and end date parameters must include a time as well. The time and date should be specified with the following format:

"YYYY-MM-DD HH:MM"

The time zone is UTC, and a 24-hour clock should be used. Additionally, the network lease command needs to have a start time later than the current time, and an end time no later than 7 days after the start time. Note that the networks we will create are only going to be active during the time the lease is active, so ensure you have allocated sufficient time.

Repeat these steps for the other Chameleon site. Note that these networks or leases cannot be named the same thing.

Once the leases have instantiated correctly, a network will automatically be created at each site. This can be verified on the online portal, using the “Networks” tab.

Once both networks are up, a ticket will need to be submitted to the Chameleon help desk. This ticket should request the creation of an AL2S circuit between the sites, and should specify the Project ID, network name, and network ID of each network. This information can be found by clicking on each network name. Additionally, the lifetime of the stitched network should also be specified. Set this lifetime to be the same as the lifetime of your network leases. This circuit will enable our hosts at different physical sites to talk to each other as if they were on the same LAN.

Submit a ticket containing this information, and wait for it to be approved.

Network Setup

Create a router

Next, a router must be created to serve as a default gateway for our instances to access the rest of the internet.

  1. Login to one of the Chameleon portals.
  2. Navigate to the “Routers” section under the “Network” tab on the left sidebar.
  3. Then, on the right-hand side, click the “Create Router” button.
  4. Next, name this router. We like to use slate-router.
  5. Click the external network drop-down menu, and select “public”.
  6. Leave everything else as-is, and click “Create Router”.
  7. Repeat these steps for the other Chameleon site.

Create a subnet

Next, we will create a subnet under our existing network.

  1. Login to one of the Chameleon portals.
  2. Navigate to the left sidebar and select “Network”, then select “Networks” underneath this.
  3. Locate the network that was already created for you, and click the drop down menu next to your network on the right side.
  4. From the drop down menu, select “Create Subnet”.
  5. The only value that needs to be changed here is the “Network Address” parameter (no subnet name is needed). Use 192.168.1.0/24 for the network address.
  6. Following this, click “Next” again. You will be brought to the “Subnet Details” section. Leave this as-is, and click “Create”.
  7. Repeat these steps on the other Chameleon site/portal.

If you would like to learn more about networks in Chameleon, more information can be found in the Chameleon documentation.

Connect your router to your new network

Here, we will connect the router object we made to our custom network.

  1. Login to one of the Chameleon portals.
  2. Navigate back to the “Routers” section under the “Network” tab on the left sidebar.
  3. Click on the name of the router you created earlier (most likely called slate-router).
  4. Select the “Interfaces” tab, and then click “Add Interface”
  5. Under the “Subnet” drop-down menu, select the network that was created earlier.
  6. Leave everything else as-is, and click “Submit”.
  7. Repeat these steps on the other Chameleon site/portal.

Instance Setup

Now, we need to launch two compute nodes, on the subnets that we already created. More detailed instructions regarding creating instances and associating IP addresses can be found in the Getting Started Guide.

Launch Instances

  1. First, login to one of the Chameleon portals.
  2. Click the “Reservations” tab on the left side, and select the “Leases” menu underneath it.
  3. Click “Create Lease”, and configure a lease for as much time as you need.
  4. In the lease-creation box, switch to the “Hosts” tab, and check the “Reserve Hosts” box.
  5. Set the minimum and maximum number of hosts to 1.
  6. Set the node type property to compute_haswell or compute_skylake.
  7. Click through the remainder of the lease creation wizard, until a reservation is created.
  8. Next, navigate to the “Instances” page under the “Compute” menu on the left-hand side.
  9. Then, on the right side of the page, click the “Launch Instance” button.
  10. Under the “Details” tab, give this instance a name (we like slate-instance). Additionally, select the reservation that was previously created.
  11. Under the “Source” tab, select “Image” under the “Select Boot Source” drop-down menu. Then, select the CC-CentOS7 image.
  12. Under the “Networks” tab, make sure that the only network that is selected is our new network.
  13. Under the “Key Pair” tab, make sure you have configured the correct SSH keys. This is explained in more detail in this documentation.
  14. Click the “Launch Instance” button, and wait for the instance to spin up. This may take 5 to 10 minutes.
  15. Repeat these steps for the other site.

Associate Public IP and Log In

To access our instance, we need to NAT a floating public IP to our instance.

  1. First, login to one Chameleon portal.
  2. Next, go to the “Network” tab on the left and select “Floating IPs”.
  3. Click the “Allocate IP to Project” button located on the right side.
  4. Leave the default settings and click “Allocate IP”.
  5. Once an IP address has been allocated, click the “Associate” button to the right of the IP address.
  6. From the “Port to be Associated” menu, select the instance you created earlier.
  7. Click “Associate”
  8. Repeat for the other instance on the other Chameleon site.

For more information regarding associating floating IP addresses, visit the Chameleon Getting Started Guide.

To login to any Chameleon node, log in as user cc, with ssh cc@<PUBLIC_INSTANCE_IP>. The public instance IP is the floating public IP that we assigned earlier. This user should have password-less sudo access.

Disable Firewall

Before you go any further, make sure any firewalls are disabled, as they will impact cluster creation. On Chameleon, ufw is often running, even on CentOS. Disable it with sudo ufw disable.

Assign Cluster Hostnames

Our testing software, perfSONAR, requires a DNS name, not an IP address. To get around this, we are simply going to edit the /etc/hosts file on both of our clusters. Open this file on both nodes, and append the following lines:

<node_1_internal_ip> cluster1.slateci.net
<node_2_internal_ip> cluster2.slateci.net

The internal IP of each node can be found from each node’s respective web portal. These IPs will be assigned from the 192.168.1.0/24 subnet that we created earlier.

Testing

Test Basic Connectivity

First, use the ping command on each node to reach the other node.

From the first cluster, run: ping cluster2.slateci.net.

From the second cluster, run: ping cluster1.slateci.net.

If these ping commands succeed, we have successfully stitched the two Chameleon sites together!

Testing with iPerf

Next, as a preliminary method of verifying throughput, iperf can be used. If iperf is not already installed, it can be installed with:

sudo yum install iperf

Note that both nodes will need to have iperf installed.

Afterwards, run an iPerf server on the first node with this command:

iperf -s

Then, run the iPerf client on the second node and connect to the server on the first node with this command:

iperf -c <node_1_internal_ip>

After a little while, the test will complete, and you should see bandwidth results. If desired, you can run this test in reverse by swapping the locations of the client and server. Alternately, iPerf’s -r or -d options can be used for running bi-directional tests.

Testing with perfSONAR

The perfsonar-testpoint application can be installed with its default values. Do this with the following command wherever you have the slate client available:

slate app install perfsonar-testpoint --cluster <cluster_2> --group <your_group>

The perfsonar-checker application will require a small amount of configuration. First, to fetch its configuration file, run:

slate app get-conf --dev perfsonar-checker > perfsonar-checker.conf

Navigate to the Instance section, and give your application an appropriate instance tag. Next, navigate to the PerfsonarChecker section, and change the Dest1 parameter to the appropriate hostname that was set up earlier in /etc/hosts/. The Dest2 and Dest3 values can be commented out, or left as-is, if you would also like to run tests to those endpoints.

Now, we are ready to install this application with this command:

slate app install --dev perfsonar-checker --cluster <cluster_1> --group <your_group> --conf perfsonar-checker.conf

After the application finishes installing, an instance ID will be printed out. Take note of this ID. Then, give the application some time to run, and check the results with this command:

slate instance logs --max-lines 0 <perfsonar_checker_instance_id>

More information about perfSONAR and the expected results can be found in the PerfSONAR Checker Blog Post.

Other Testbeds

Similar setups can be created on other testbeds as well. The testing methods outlined in the post will also work on other testbeds. Additionally, the SLATE Nginx application (without ingress controller) can be used to test. Host an Nginx instance on both nodes, and access each instance from the opposing node over NodePort using curl.

CloudLab-to-CloudLab setups have been verified to work, as well as GENI-to-Chameleon@UC.

More documentation about stitching networks between Chameleon and other testbeds/external domains can be found in the Chameleon documentation.

Contact Us

If you have any additional comments or questions, please contact our team!

The SLATE Team
Edit This Page on GitHub