News Brief

24/10/2012 - Wolfpack Cluster's launched! Check out the User Guide now!

ubuntu Cluster & Grid Computing

Cluster Decommissioning

The very first job:

==============================================================
qname        amd.q
hostname     sgeqexec02.garvan.unsw.edu.au
group        FacilityBioinformatics
owner        warkap
project      NONE
department   defaultdepartment
jobname      smallMemTest.py
jobnumber    1
taskid       undefined
account      sge
priority     0
qsub_time    Mon Feb  7 12:02:10 2011
start_time   Mon Feb  7 12:02:37 2011
end_time     Mon Feb  7 12:02:37 2011
granted_pe   NONE
slots        1
failed       0
exit_status  1
ru_wallclock 0
ru_utime     0.000
ru_stime     0.000
ru_maxrss    1216
ru_ixrss     0
ru_ismrss    0
ru_idrss     0
ru_isrss     0
ru_minflt    1206
ru_majflt    4
ru_nswap     0
ru_inblock   776
ru_oublock   24
ru_msgsnd    0
ru_msgrcv    0
ru_nsignals  0
ru_nvcsw     27
ru_nivcsw    11
cpu          0.000
mem          0.000
io           0.000
iow          0.000
maxvmem      0.000
arid         undefined

Usage Statistic:

#
#
#
Job Count       Queue
--------------------------
114128          xs.mem.q
89330           s.mem.q
5507            gp.q
2597            m.mem.q
1432            dev.q
625             qa.q
0               l.mem.q

Total Job Count: 213619
#
#
#
Total System Usage
    WALLCLOCK         UTIME         STIME           CPU             MEMORY                 IO                IOW
================================================================================================================
    167603488 155969787.310    297208.060 164762489.430       65708477.703         141009.945              0.000
#
#
#
OWNER      WALLCLOCK         UTIME         STIME           CPU             MEMORY                 IO                IOW
=======================================================================================================================
aarsta      14512720  17921918.730    108065.200  18896325.740       28715114.548          80477.720              0.000
danrod         57476    121899.300      6087.400    127987.460         287091.975           4344.408              0.000
darstr          8431         0.630         0.570         1.200              0.000              0.000              0.000
derlin           611       187.500        50.670       238.170             19.843              0.031              0.000
elezot       5740613   1090293.670     11732.890   1102026.560        2207446.671          14402.379              0.000
marcow      25432704  22404409.180     74961.590  23570468.330        8259307.172          25611.727              0.000
marpin      50099730  49654391.860     35597.680  49807467.790        8849315.276           6819.245              0.000
nicarm      62251366  62172647.920       304.150  62172952.070       10708604.355              0.612              0.000
pwbcad       9481740   2513952.360     59996.150   8994524.190        6441911.257           9320.558              0.000
rimcha           372         0.030         0.020         0.050              0.000              0.000              0.000
warkap         17725     90086.130       411.740     90497.870         239666.607             33.263              0.000

SGE's Fact Sheet

This section provides the most current configuration of the cluster. Please check out this fact sheet regularly to make sure you are up-to-date.

The Queues

  • xs.mem.q - extra small memory queue. 24 CPUs, 4GB/CPU. (SGEQEXEC0000-4999)
  • s.mem.q - small memory queue. 16 CPUs, 8GB/CPU. (SGEQEXEC5000-8999)
  • m.mem.q - medium memory queue. 8 CPUs, 16GB/CPU. (SGEQEXEC9000-9999)
  • l.mem.q - large memory queue. 4 CPUs, 32GB/CPU. (SGEQEXEC000-599)
  • xl.mem.q - extra large memory. N/A CPUs, 64GB/CPU. TBA (SGEQEXEC600-999)
  • dev.q - development queue. Very unstable. Do *NOT* use.
  • qa.q - quality assurance queue. Do *NOT* use.

:!: Memory per CPU is a hard limit. If a job over this designated limit, it will be terminated automatically.
:!: The queue name must be specified for a job but ONLY if it's a single-thread, non-parallel job.

The Parallel Environment (PE)

A PE, is a configuration to facilitate a parallel job to be run on the SGE. See SGE in Parallel Environment (PE) to learn how to run a parallel job. Each queue has already had a standard PE attached to it. The name of a PE is the same as its queue name except it has “orte” instead of “q” in the last bit. For example, dev.q has dev.orte PE. qa.q has qa.orte PE.

:!: For a parallel job, a PE name MUST be specified instead of the queue name.
:!: If a PE with more capacity is required, please contact PWBC administrator.

The Cluster's Storages

The cluster accesses data (like input, output) via NFS protocol. Cluster provides each user a Home directory which is cluster-wide accessible. That means, a user's Home directory is mounted to every host in the cluster via NFS.

gagri:/ClusterHome/USER_ID

:!: ClusterHome is NOT backed up. Its sole objective is for the cluster to process job data. It means nothing more than an user's private working directory. Do NOT store any data in permanent basis.

Sun Grid Engine (SGE) Basics

The Sun Grid Engine (SGE) is a job management service intended to accept, schedule, dispatch, and manage remote execution of a large number of standalone, parallel or interactive user jobs. The current version is 6.2u5. Documentation can be found in here.

Submission Host

A job is submitted from a authorized submission host with a set of SGE commands. Beside the submission feature, the submission host itself is identical to the execution hosts on the cluster. Thus it's very useful for checking other details such as:

  • The software, toolkits and binaries the cluster supports.
  • The NFS mount points available on the cluster.

For example, to find out if cluster's R-2.12.2 contains Rmpi library, just run R-2.12.2 on the submission host and check.

For logging in, just SSH to:

sgeqsub01.garvan.unsw.edu.au

:!: NEVER run any job on the submission directly.
:!: NEVER save any of your personal file inside home directory (i.e. /home/user).

Check the Status

SGE provides qstat to allow users to find out the current status of the cluster such as queues, running jobs or a job's summary etc.

#
# The simplest way, display all available queues: 
#
pwbcad@sgeqsub01:~$ qstat -f
queuename                      qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------------------------
dev.q@sgeqexec01.garvan.unsw.e BIP   0/0/8          0.00     lx26-amd64
---------------------------------------------------------------------------------
gp.q@sgeqexec04.garvan.unsw.ed BIP   0/0/2          0.00     lx26-amd64
---------------------------------------------------------------------------------
gp.q@sgeqexec05.garvan.unsw.ed BIP   0/0/2          0.00     lx26-amd64
---------------------------------------------------------------------------------
gp.q@sgeqexec06.garvan.unsw.ed BIP   0/0/2          0.00     lx26-amd64
---------------------------------------------------------------------------------
gp.q@sgeqexec07.garvan.unsw.ed BIP   0/0/2          0.00     lx26-amd64
---------------------------------------------------------------------------------
qa.q@sgeqexec02.garvan.unsw.ed BIP   0/0/8          0.01     lx26-amd64
---------------------------------------------------------------------------------
s.mem.q@sgeqexec08.garvan.unsw BIP   0/0/8          0.00     lx26-amd64
---------------------------------------------------------------------------------
s.mem.q@sgeqexec09.garvan.unsw BIP   0/0/8          0.00     lx26-amd64
---------------------------------------------------------------------------------
xs.mem.q@sgeqexec04.garvan.uns BIP   0/0/6          0.00     lx26-amd64
---------------------------------------------------------------------------------
xs.mem.q@sgeqexec05.garvan.uns BIP   0/0/6          0.00     lx26-amd64
---------------------------------------------------------------------------------
xs.mem.q@sgeqexec06.garvan.uns BIP   0/0/6          0.00     lx26-amd64
---------------------------------------------------------------------------------
xs.mem.q@sgeqexec07.garvan.uns BIP   0/0/6          0.00     lx26-amd64
#
# To display one specific queue and report all jobs that are running on it:
#
pwbcad@sgeqsub01:~$ qstat -q dev.q -f
queuename                      qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------------------------
dev.q@sgeqexec01.garvan.unsw.e BIP   0/1/8          0.09     lx26-amd64
    290 0.50000 gp_dev_job pwbcad       r     05/06/2011 12:30:46     1
#
# Display detail of a job:
#
pwbcad@sgeqsub01:~$ qstat -j 290
==============================================================
job_number:                 290
exec_file:                  job_scripts/290
submission_time:            Fri May  6 12:30:32 2011
owner:                      pwbcad
uid:                        6828
group:                      FacilityBioinformatics
gid:                        3012
sge_o_home:                 /home/pwbcad
....
#
# it's a good way to check why a job fails to run
#
script_file:                ./cal_pi_auto.sh
parallel environment:  dev.orte range: 4
scheduling info:            cannot run in queue "qa.q" because PE "dev.orte" is not in pe list
                            cannot run in queue "gp.q" because PE "dev.orte" is not in pe list
                            has no permission for cluster queue "dev.q"
                            cannot run in queue "l.mem.q" because PE "dev.orte" is not in pe list
                            cannot run in queue "m.mem.q" because PE "dev.orte" is not in pe list
                            cannot run in queue "s.mem.q" because PE "dev.orte" is not in pe list
                            cannot run in queue "xs.mem.q" because PE "dev.orte" is not in pe list
                            cannot run in PE "dev.orte" because it only offers 0 slots

Submit a Batch Job

A batch job can be submitted to the cluster via command qsub. Try

pwbcad@sgeqsub01:myjob$ qsub -q qa.q -cwd -b y echo "This a job runs on a SGE cluster."
Your job 292 ("echo") has been submitted
pwbcad@sgeqsub01:myjob$ ls
echo.e292  echo.o292
pwbcad@sgeqsub01:myjob$ cat echo.o292
This a job ran on a SGE cluster.

A job can also be launched by using SGE job script. Any qsub command option can be specified inside the script prefixed with #$. It's very handy if a job needs to specify many options. A job script for running the same job above can be:

#!/bin/sh
#
# sge_launcher.sh
#
# Specify the queue to run on
#$ -q qa.q
# Use current working directory
#$ -cwd
# Indicates a binary will be executed 
#$ -b y
# This is the actual command for the job
echo "This is a job runs on a SGE cluster."
#
# Notice there is no more qsub options in the command line
#
pwbcad@sgeqsub01:myjob$ qsub ./sge_launcher.sh
Your job 293 ("sge_launcher.sh") has been submitted
pwbcad@sgeqsub01:myjob$ cat sge_launcher.sh.o293
This is a job runs on a SGE cluster.

Submit an Interactive Job

SGE supports scheduling a interactive job in the cluster. This is particularly useful for many R jobs. Command qrsh is used to accomplish this. Try:

#
# R will be launched as if it's running locally, but in fact, the R session is running on a execution host in a cluster.
#
pwbcad@sgeqsub01:myjob$ qrsh -q qa.q /usr/local/SGE_EXEC_BIN/R-2.12.2/lib64/R/bin/R --save

:!: NEVER try to install R packages via an interactive R job. You are NOT authorized to do so and it will not work due to the file permissions.
:?: Compare to qsub, qrsh does not work too well with SGE job script?

Submit a Job with Email Notification

SGE is capable to notify a user in regard of a job's status (issue #78).

#
# -m bes" means send email at beginning/end/job suspension
# -M must be specified if -m is enabled
#
qsub -b y -cwd -m bes -M f.last@garvan.org.au cmd

Remove a Running Job

A running job can be terminated with qdel command given the job ID:

pwbcad@sgeqsub01:~$ qdel 203

Sun Grid Engine (SGE) Advance