Interactive sessions on HPC

Danger: new smaller memory default!

At the user meeting on April 12, we found out that requesting 1 core will automatically provide only 500MB of memory. This is a BIG change, because in older cluster we received 2GB per core and that was generally sufficient. That is to say, we almost always did not specify memory.

The default interactive session is not likely to be sufficient, so it will be required to specify memory.

As a result, the command to ask for 1 node with 1 processor (core) on that node would be

msub -X -I -l nodes=1:ppn=1,pmem=2048m 

This asks for graphics X11 forwarding (-X). The memory can also be specified as "2gb".

If you only want 1 core on 1 node, the simpler notation would be to use the flag "procs".

msub -X -I -l procs=1,pmem=2048m 

To ask for several cores on 1 node (test multicore project), run

msub -X -I -l nodes=1:ppn=5,pmem=2048m

** Specify a queue **

Interactive jobs can be run on any queue. By default, they go to the user's nodes.

The default queue is displayed with 'mystats'. If you wish to run on a node that is not in your owner group, like a GPGPU node, you will then need to specify the sixhour queue and the node name. You will only have a maximum of 6 hours on this node. There is no time limit to your default queue.

msub -X -I -l nodes=1:ppn=5,pmem=2048m -q sixhour

One can specify a particular node, "g0001", with a request likee:

msub -X -I -lnodes=g001:ppn=1 -q sixhour

CRC made a page regarding queues and has relocated it at

Update 20170413

We requested a simpler way to launch the usual type of interactive session--one node, one core--as we had in the old cluster. The administrators created a script "qxlogin" which the user can run from the login node.

$ qxlogin
qsub: waiting for job 40565091.sched to start
qsub: job 40565091.sched ready

We suggest caution with this, since the new memory default limit is 500MB and CRMDA users have regularly reported frustration with unanticipated job failures.

In case you want to write your own login script, you can take an example from the new qxlogin, which I found is installed in /usr/local/bin on the new cluster.

$ cat /usr/local/bin/qxlogin


/opt/moab/bin/msub -X -I -lnodes=1:ppn=1 $ARGS

If you want more interactive nodes, or more ppn, just change the 1's. To test that, suppose you save it as "qxlogin2", then run

$ sh qxlogin2

If you enjoy the result, save that file in your $HOME/bin directory, make it executable, and then it will be more generally available within your sessions. After that, there is no need to run "sh" before "qxlogin2". Try it out, let me know if there is trouble.

This entry was posted in Data Analysis. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *