Welcome to another round of SLATE project updates! It’s been quite some time since our last update, so this post will be especially long. On the plus side, we have a lot of cool and interesting new things to tell you about!
JupyterLab is a great tool for data analysis, visualization, machine learning and much more. It allows users to run code interactively via its web notebook interface and thus iterate changes quickly. Often users need to scale up their work and thus require submission to a backend cluster from the notebook. We show how this can be done with HTCondor using SLATE.
An OSG Compute Element (CE) is an application that allows a site to contribute HPC or HTC compute resources to the Open Science Grid. The CE is responsible for receiving jobs from the grid and routing them to your local cluster(s). Jobs from the Open Science Grid are preemptible, and can be configured to run only when resources would have otherwise been idle. Resource providers can use OSG to backfill their cluster(s) to efficiently utilize resources and contribute to the shared national cyberinfrastructure.
The simplest way to start contributing resources to the OSG, for many sites, is via the “Hosted” CE. In the hosted case, installation and setup of the Compute Element is done by the OSG team, usually on a machine outside of your cluster, and uses standard OpenSSH as a transport for submitting jobs to your resources. With SLATE, we have simplified Hosted CE installation and made a shared operations model possible. Now the Compute Element can be hosted on your Kubernetes infrastructure on-prem and cooperatively managed by OSG and your local team. In this article, we’ll go through the steps for connecting a SLURM cluster to the Open Science Grid with our newly stabilized OSG Hosted CE Helm Chart.
Sometimes when we run containerized applications, we want a little more than the usual stdout/stderr log aggregation that Kubernetes or SLATE gives us by default. For many applications, logs may be exceptionally verbose, or split across many files corresponding to different parts of the software, necessitating some other mechanism by which to retrieve logs. In this post we’ll show you how to add a logging sidecar to your Helm chart, complete with HTTP basic auth and ingress for ease of access.