ACF Cluster changing to SLURM que manager November 7

In case you did not hear yet, the current scheduler, which was implemented February 2017, is being replaced by SLURM, on November 8. There will be a cluster shutdown Nov. 6- Nov 7 and after that the submission scripts that used "MSUB" notation will no longer work.

Here is how we are preparing in CRMDA.

  1. We have a test cluster for people who want to practice SLURM code. We reallocated 5 notes out of the CRMDA share into this temporary test system:

    slurmsubmit.hpc.crc.ku.edu

    Users who currently are in the CRMDA cluster group should be able to log in there and test revisions. Because we ONLY have 5 nodes there, one should not submit MASSIVE jobs in slurmsubmit. It is intended primarily for testing and learning SLURM to prepare for the transition.

  2. Our set of working examples for high performance computing, hpcexample, is being re-worked. The webpage for the Gitlab repository is https://gitlab.crmda.ku.edu/crmda/hpcexample The master branch on the repository is the currently used MSUB notation. We have started a new branch slurm and on that branch we are putting in revised submission scripts. The slurm branch You can tell if an example has been updated because the notation #MSUB will be changed to #SBATCH.

The Center for Research Computing has a help page for the transition (https://crc.ku.edu/hpc/slurm/how-to) and there have been training sessions (https://crc.ku.edu/hpc/meetings).

To give you a rough idea of what is needed, we compare an old MSUB-style submission script (in this case, the submission for Example 65 (https://gitlab.crmda.ku.edu/crmda/hpcexample/tree/master/Ex65-R-parallel)

#!/bin/sh
#
#
#MSUB -M your-name_here@ku.edu
#MSUB -N RParallelHelloWorld 
#MSUB -q sixhour 
#MSUB -l nodes=1:ppn=11:ib
#MSUB -l walltime=00:05:00
#MSUB -m bea

cd $PBS_O_WORKDIR

## Please check your ~/.bash_profile to make sure
## the correct modules will be loaded with new shells.
## See discussion:
## http://www.crmda.dept.ku.edu/timeline/archives/184
## In a terminal on a compute node, you can test this
## by running
## $ module list
## It is not helpful or necessary to insert module
## commands in this script (contrary to previous advice).
   mpiexec -n 1 R --vanilla -f parallel-hello.R

To the SLURM style, like so:

#!/bin/sh
#
#
#SBATCH --mail-user=your_name_here@ku.edu
#SBATCH --job-name=RParallelHelloWorld 
#SBATCH --partition=sixhour
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=11
#SBATCH --time=00:05:00
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --error="%x.e%j"
#SBATCH --output="%x.o%j"

cd $SLURM_SUBMIT_DIR

## Please check your ~/.bash_profile to make sure
## the correct modules will be loaded with new shells.
## See discussion:
## http://www.crmda.dept.ku.edu/timeline/archives/184
## In a terminal on a compute node, you can test this
## by running
## $ module list
## It is not helpful or necessary to insert module
## commands in this script (contrary to previous advice).

mpiexec -n 1 R --vanilla -f parallel-hello.R

Then, to actually launch the job into the cluster, we will no longer use "msub", instead use "sbatch".

After November 7, the slurmsubmit.hpc.crc.ku.edu test login will no longer function, users will just need to go to the address that we currently use hpc.crc.ku.edu.

The slurm branch in hpcexample has just a few fully converted directories, please check Ex65 for a fully working example if you need it today. However, the other examples are rapidly approaching a presentable state.

Other chores we need to do:

  1. See about creating "shortcuts" to create user sessions. In SLURM system, to request an interactive session (as explained in the CRC slurm/how-to document) can be obtained thus:

    $ srun --time=4:00:00 --ntasks=1 --nodes=1 --partition=sixhour --x11 --pty /bin/bash

    I would have an easier-to-remember way to launch that, like our old qlogin script. But I don't yet know how we can make that available to the users of CRMDA-based accounts. We'll figure it out.

  2. Did you notice that our old MSUB script asked for Infiniband (ib) nodes, but the new one does not. Its like that because I don't know the right approach. Find out if it is smart to ask for or avoid the infiniband nodes. Several years ago, we were avoiding IB because it made jobs hang. More recently, we've been asking for IB nodes because it was a way to make sure that a job's calculations were not sent to a very old compute node (only newer had IB). I don't know what is the best advice now.

  3. If we have a job that requires 11 cores, are we smarter to ask for 1 node with 11 tasks, or are we smarter to allow cluster to send the 11 cores-worth of work out to 11 different nodes. Over time, the best advice has changed. In our old cluster, the latter was faster. In the community cluster, we noticed that there were errors caused by slightly different types of compute nodes and we were smarter to try to keep all computations on the same system (or a small number of systems). I don't know what the best advice will be in the new SLURM framework. We'll figure it out.

  4. A wholesale overhaul of our training materials in http://crmda.ku.edu/computing will need to occur. That's the most daunting part in all of this.

This entry was posted in Data Analysis. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *