Cheatsheet for CSD3 (Cambridge HPC)

The documentation for the Cambridge HPC is good, but I've tried to summarize the most relevant parts below.

Logging In

You need to be on a Cambridge network or logged into the VPN. Then, from your terminal:

# For CPU cluster
ssh <CSRID>@login-cpu.hpc.cam.ac.uk

# For GPU cluster
ssh <CSRID>@login-gpu.hpc.cam.ac.uk

Basic conda on CSD3

You can install conda packages as follows:

# Load the conda module
module load miniconda3-4.5.4-gcc-5.4.0-hivczbz

# Create conda environment with python 3.7. 
# You can change the name to something appropriate
conda create --name testenv python=3.7

# Activate environment. 
# Replace USER with your username and testenv with the name of your environemnt
source activate /home/USER/.conda/envs/testenv/

#Note that you can also see the location of all conda environments using..
conda info --envs

# Install needed packages. Note that this takes a long time
conda install -c conda-forge graph-tool

# Test graph tools
python3 
from graph_tool.all import *

Submitting a job to SLURM

Below I have modified the standard slurm_submit.peta4-skylake SLURM script to run a graph tool job. The things I changed:

Line 15: Put the name of the project. This can be found by typing mybalanceinto the command line when logged into CSD3
Nodes (line 17) and tasks (line 20) are changed to one for the test
Line 58-60 load the conda module and virtual environment
Line 63 specifies the application to run (test_graph_tool.py is just a simple script that imports graph-tool to see if it works)
I commented out line 93 and and uncommented line 97 to just run the basic application instead of using MPI

More information can be found here.

#!/bin/bash
#!
#! Example SLURM job script for Peta4-Skylake (Skylake CPUs, OPA)
#! Last updated: Mon 13 Nov 12:25:17 GMT 2017
#!

#!#############################################################
#!#### Modify the options in this section as appropriate ######
#!#############################################################

#! sbatch directives begin here ###############################
#! Name of the job:
#SBATCH -J cpujob
#! Which project should be charged:
#SBATCH -A test-projec
#! How many whole nodes should be allocated?
#SBATCH --nodes=1
#! How many (MPI) tasks will there be in total? (<= nodes*32)
#! The skylake/skylake-himem nodes have 32 CPUs (cores) each.
#SBATCH --ntasks=1
#! How much wallclock time will be required?
#SBATCH --time=00:01:00
#! What types of email messages do you wish to receive?
#SBATCH --mail-type=FAIL
#! Uncomment this to prevent the job from being requeued (e.g. if
#! interrupted by node failure or system downtime):
##SBATCH --no-requeue

#! For 6GB per CPU, set "-p skylake"; for 12GB per CPU, set "-p skylake-himem": 
#SBATCH -p skylake

#! sbatch directives end here (put any additional directives above this line)

#! Notes:
#! Charging is determined by core number*walltime.
#! The --ntasks value refers to the number of tasks to be launched by SLURM only. This
#! usually equates to the number of MPI tasks launched. Reduce this from nodes*32 if
#! demanded by memory requirements, or if OMP_NUM_THREADS>1.
#! Each task is allocated 1 core by default, and each core is allocated 5980MB (skylake)
#! and 12030MB (skylake-himem). If this is insufficient, also specify
#! --cpus-per-task and/or --mem (the latter specifies MB per node).

#! Number of nodes and tasks per node allocated by SLURM (do not change):
numnodes=$SLURM_JOB_NUM_NODES
numtasks=$SLURM_NTASKS
mpi_tasks_per_node=$(echo "$SLURM_TASKS_PER_NODE" | sed -e  's/^\([0-9][0-9]*\).*$/\1/')
#! ############################################################
#! Modify the settings below to specify the application's environment, location 
#! and launch method:

#! Optionally modify the environment seen by the application
#! (note that SLURM reproduces the environment at submission irrespective of ~/.bashrc):
. /etc/profile.d/modules.sh                # Leave this line (enables the module command)
module purge                               # Removes all modules still loaded
module load rhel7/default-peta4            # REQUIRED - loads the basic environment

#! Insert additional module load commands after this line if needed:
module load miniconda3-4.5.4-gcc-5.4.0-hivczbz
source activate /home/kcmf2/.conda/envs/testenv/
conda info --envs

#! Full path to application executable: 
application="test_graph_tool.py"

#! Run options for the application:
options=""

#! Work directory (i.e. where the job will run):
workdir="$SLURM_SUBMIT_DIR"  # The value of SLURM_SUBMIT_DIR sets workdir to the directory
                             # in which sbatch is run.

#! Are you using OpenMP (NB this is unrelated to OpenMPI)? If so increase this
#! safe value to no more than 32:
export OMP_NUM_THREADS=1

#! Number of MPI tasks to be started by the application per node and in total (do not change):
np=$[${numnodes}*${mpi_tasks_per_node}]

#! The following variables define a sensible pinning strategy for Intel MPI tasks -
#! this should be suitable for both pure MPI and hybrid MPI/OpenMP jobs:
export I_MPI_PIN_DOMAIN=omp:compact # Domains are $OMP_NUM_THREADS cores in size
export I_MPI_PIN_ORDER=scatter # Adjacent domains have minimal sharing of caches/sockets
#! Notes:
#! 1. These variables influence Intel MPI only.
#! 2. Domains are non-overlapping sets of cores which map 1-1 to MPI tasks.
#! 3. I_MPI_PIN_PROCESSOR_LIST is ignored if I_MPI_PIN_DOMAIN is set.
#! 4. If MPI tasks perform better when sharing caches/sockets, try I_MPI_PIN_ORDER=compact.


#! Uncomment one choice for CMD below (add mpirun/mpiexec options if necessary):

#! Choose this for a MPI code (possibly using OpenMP) using Intel MPI.
#CMD="mpirun -ppn $mpi_tasks_per_node -np $np $application $options"

#! Choose this for a pure shared-memory OpenMP parallel program on a single node:
#! (OMP_NUM_THREADS threads will be created):
CMD="$application $options"

#! Choose this for a MPI code (possibly using OpenMP) using OpenMPI:
#CMD="mpirun -npernode $mpi_tasks_per_node -np $np $application $options"


###############################################################
### You should not have to change anything below this line ####
###############################################################

cd $workdir
echo -e "Changed directory to `pwd`.\n"

JOBID=$SLURM_JOB_ID

echo -e "JobID: $JOBID\n======"
echo "Time: `date`"
echo "Running on master node: `hostname`"
echo "Current directory: `pwd`"

if [ "$SLURM_JOB_NODELIST" ]; then
        #! Create a machine file:
        export NODEFILE=`generate_pbs_nodefile`
        cat $NODEFILE | uniq > machine.file.$JOBID
        echo -e "\nNodes allocated:\n================"
        echo `cat machine.file.$JOBID | sed -e 's/\..*$//g'`
fi

echo -e "\nnumtasks=$numtasks, numnodes=$numnodes, mpi_tasks_per_node=$mpi_tasks_per_node (OMP_NUM_THREADS=$OMP_NUM_THREADS)"

echo -e "\nExecuting command:\n==================\n$CMD\n"

eval $CMD

A note, you need to first deactivate the environment before submitting a job..

Sbatch commands for submitting jobs via slurm

squeue      -> show global cluster information
squeue -u kcmf2 --start -> estimate start time for you jobs
sinfo       -> show global cluster information
sview       -> show global cluster information
scontrol show job nnnn -> examine the job with jobid nnnn
scontrol show node nodename -> examine the node with name nodename
sbatch      -> submits an executable script to the queueing system
sintr       -> submits an interactive job to the queueing system
srun        -> run a command either as a new job or within an existing job
scancel     -> delete a job
mybalance   -> show current balance of core hour credits

Docker and Singularity

There is a strange bug with the standard temporary directory used in pulling containers. So, to pull a container, make a temporary directory in your home folder. For example:

mkdir .singularity_tmp

Then, you can run the singularity pull command (similar to Docker pull):

TMPDIR=~/.singularity_tmp singularity pull docker://marcosfelt/summit

Replace marcosfelt/summit with the name of the image you want to pull.

To-Do

~~Set up conda and see if you can properly install graph tool~~

~~Figure out how to submit jobs to SLURM using conda~~

Figure out how to use DMPTC for long running jobs: https://docs.hpc.cam.ac.uk/hpc/user-guide/long.html

Consider testing jupyterlab: https://docs.hpc.cam.ac.uk/hpc/software-packages/jupyter.html