Reading Running ML jobs in Compute Canada clusters

Throughout your script, make sure to have plots written to file and not displayed on screen.

You can develop your script directly in the cluster using an interactive job, or on your local machine (working on your machine at this stage, using very little data and few epochs—only to develop and test the code—is a convenient approach).

Ultimately, you will submit your script to sbatch .

The Compute Canada clusters are optimized for very large files and are slowed by large collections of small files. Datasets with many small files need to be turned into single-file archives with tar . Failing to do so will affect performance not just for you, but for all users of the cluster.

$ tar cf <data>.tar <path/to/dataset/directory>/*

Notes:

• If you want to also compress the files, replace tar cf with tar czf
• As a modern alternative to tar , you can use Dar

To use Python for ML on the cluster, you will need to load the relevant modules.

This is done with the Lmod tool through the module command. Here are some key commands:

# Get help on the module command
$ module help

# List modules that are already loaded
$ module list

# See which modules are available for Python
$ module avail python

# Load Python, CUDA, and cuDNN
$ module load python/3.8.2 cudacore/.10.1.243 cuda/10 cudnn/7.6.5

NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks.

You also need Python packages.

For this, create a virtual environment on compute nodes (this improves I/O performance) in which you install packages with pip .

The variable $SLURM_TMPDIR is created by Slurm on the compute node where a job is running. Its path is /localscratch/<user>.<jobid>.0 . Anything in it gets deleted when the job is done.

It may be a good idea to run an interactive job to create a requirements file and test your code:

$ salloc --account=def-<user> --gres=gpu:1 --cpus-per-task=6 --mem=32000 --time=1:00

$ virtualenv --no-download $SLURM_TMPDIR/env

$ source $SLURM_TMPDIR/env/bin/activate

(env) $ pip install --no-index --upgrade pip

(env) $ pip install --no-index torch torchvision

(env) $ pip freeze > ~/<path/project/dir/>requirements.txt

(env) $ mkdir $SLURM_TMPDIR/data

(env) $ tar xf ~/projects/def-<user>/<data>.tar -C $SLURM_TMPDIR/data

(env) $ python

>>> import torch
>>> ...

(env) $ deactivate

To submit a job to Slurm (the job scheduler used by the Compute Canada clusters), you need to write an sbatch script. Here is an example script:

#!/bin/bash
#SBATCH --job-name=<name>*			  # job name
#SBATCH --account=def-<user>
#SBATCH --time=<time>				  # max walltime in D-HH:MM or HH:MM:SS
#SBATCH --cpus-per-task=<number>      # number of cores
#SBATCH --gres=gpu:<type>:<number>    # type and number of GPU(s) per node
#SBATCH --mem=<mem>					  # max memory (default unit is MB) per node
#SBATCH --output=<file%j.out>*		  # file name for the output
#SBATCH --error=<file%j.err>*		  # file name for errors
					                  # %j gets replaced by the job number
#SBATCH --mail-user=<email_address>*
#SBATCH --mail-type=ALL*

# Load modules
module load python/3.8.2 cudacore/.10.1.243 cuda/10 cudnn/7.6.5

# Create variable with the directory for your ML project
SOURCEDIR=~/<path/project/dir>

# Create and activate a virtual environment on compute node
virtualenv --no-download $SLURM_TMPDIR/env
source $SLURM_TMPDIR/env/bin/activate

# Install Python packages
pip install --no-index -r $SOURCEDIR/requirements.txt

# Transfer and extract data
mkdir $SLURM_TMPDIR/data
tar xf ~/projects/def-<user>/<data>.tar -C $SLURM_TMPDIR/data

# Run Python script on the data
python $SOURCEDIR/<mlscript>.py $SLURM_TMPDIR/data

Notes:

• If you compressed your data with tar czf , you need to extract it with tar xzf
• SBATCH options marked with a * are optional
• There are various other options for email notifications.

Long jobs should have a checkpoint at least every 24 hours. This ensures that an outage won't lead to days of computation lost and it will help get the job started sooner by the scheduler.

For instance, you might want to have checkpoints every n epochs (choose n so that n epochs take less than 24 hours to run).

In PyTorch, you can create dictionaries with all the information necessary and save them as .tar files with torch.save() . You can then load them back with torch.load() .

The information you want to save in each checkpoint includes the model’s state_dict, the optimizer’s state_dict, the epoch at which you stopped, the latest training loss, and anything else needed to restart training where you left off.

Example:

Saving a checkpoint during training could look something like this:

torch.save({
    'epoch': <last epoch run>,
    'model_state_dict': net.state_dict(),
    'optimizer_state_dict': optimizer.state_dict(),
    'loss': <latest loss>,
}, <path/to/file.tar>)

To restart, initialize the model and optimizer, load the dictionary, and resume training:

# Initialize the model and optimizer
model = <your model>
optimizer = <your optimizer>

# Load the dictionary
checkpoint = torch.load(<path/to/file.tar>)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

# Resume training
model.train()

A number of ML tasks (e.g. hyperparameter optimization) require running several instances of similar jobs. Grouping them into a single job with GLOST or GNU Parallel reduces the stress on the scheduler.

If you want to visually track your model metrics (e.g. loss, accuracy, model graph, etc.) with TensorBoard, you need to launch it in the background (using & ) before your python script by adding to your sbatch script:

tensorboard --logdir=/tmp/<your log dir> --host 0.0.0.0 &

So our example script would look like:

#!/bin/bash
#SBATCH ...
...

tensorboard --logdir=/tmp/<your log dir> --host 0.0.0.0 &
python $SOURCEDIR/<mlscript>.py $SLURM_TMPDIR/data

Once the job is running, you need to create a connection between the compute node running TensorBoard and your computer.

First, you need to find the hostname of the compute node running the Tensorboard server. This is the value under NODELIST for your job when you run:

$ sq

Then, from your computer, enter this ssh command:

[local]$ ssh -N -f -L localhost:6006:<node hostname>:6006 <user>@<cluster>.computecanada.ca

(Replace <node hostname> by the compute node hostname you just identified, <user> by your user name, and <cluster> by the name of the Compute Canada cluster hostname (e.g. beluga, cedar, graham).)

You can now open a browser and go to http://localhost:6006.

Note: TensorBoard requires a lot of processing power. Please do not run it from the login node. Only run it in a job.