CMS - Jarasandh - HPC Reference Manual



  • Introduction
This document summarizes hardware details, access, tools, commands for Jarasandh.cms.unipune.ernet.in - compute cluster based on rocks 4.3 located at ground floor of Centre for Modeling and Simulation (in Computer Science Building).

The status webpage of the cluster can be accessed using following URL
 http://cms.unipune.ernet.in:9080/ganglia


back
  • Hardware details
The setup consists of 9 x HP Proliant DL 360 Servers, 1 x Dell PowerEdge 2950, 6 x Dell PowerEdge 1950 Server, 1 x 24 Port Voltaire Infiniband Switch, 1 x D-Link Giga Switch, 1 x HP Storage MSA 60(12 Hard disk , each 146 GB), 1 x USB DVD writer & 1 x 16 Port D-Link, 1 Benq Monitor, 1 x Logitech mouse, 1 x Logitech keyboard .

All the machines has processor Xeon running at 3 Ghz. Dell machines are dual processor dual core and HP machines are dual processor quad core. Every machine has 1 GB RAM per core. Thus entire cluster including master node has 32 CPUs/ 100 cores and 100 GB of RAM.

Usable space on the storage MSA 60 is 673 GB


back
  • Logging in - out
Jarasandh is located behind CMS server in university network.

  • For logging onto Machine give following command from your Linux desktop shell
ssh -p 2020 username@cms.unipune.ernet.in
The username and password is the same, which is on jarasandh machine

  • For logging out just type
exit


back
For transferring your files from local machine to jarasandh,
    scp -P 2020 your_file username@cms.unipune.ernet.in:
This will copy "your_file" to cluster. The "username" in above command is your username on jarasandh and the password when prompted should be given same as that of your password on jarasandh (its abc if you have not changed)
scp -P 2020 -r your_dir username@cms.unipune.ernet.in:
Above command should be used to recurrsivly copy files from "your_dir" to jarasandh.

For transferring your files from jarasandh to local machine.
    scp -P 2020 username@cms.unipune.ernet.in:~/file_on_jarasandh .
This will copy "file_on_jarasandh" to your local machine. The "username" in above command is your username on jarasandh and the password when prompted should be given same as that of your password on jarasandh (its username123 if you have not changed)
scp -P 2020 -r username@cms.unipune.ernet.in:~/dir_on_jarasandh .
Above command should be used to recurrsivly copy files from "dir_on_jarasandh" to your local machine.

back
Sr. No.
Compilers / Libraries
Version
Installation path
1
Intel Fortran compiler
10.1
/opt/intel/fce/10.1.011/bin/ifort
2
Intel C++ compiler
10.1
/opt/intel/cce/10.1.011/bin/icc
3
Intel Mathematical Kernel libraries
10.0
/opt/intel/mkl/10.0.1.014/lib/
4
GNU C compiler
4.1.0
/usr/local/bin/gcc
5
GNU fortran compiler

/usr/bin/g77
6
ATLAS (Automatically Tuned Linear Algebra Software)

/opt/Libraries/ATLAS
7
LAPACK (Linear Algibra PACKage)

for dynamic linking
/usr/lib/liblapack.so
/usr/lib64/liblapack.so
for static linking
/usr/lib/liblapack.a
/usr/lib64/liblapack.a
8
ScaLAPACK (Scalable LAPACK)

/opt/Libraries/SCALAPACK
9
BLACS (Basic Linear Algebra Communication Subprograms)

/opt/Libraries/BLACS
10
FFTW
3.2.1
/opt/Libraries/fftw-3.2.1

back
Gaussian 03

Gaussian 03 is the latest in the Gaussian series of electronic structure programs. Gaussian 03 is
used by chemists, chemical engineers, biochemists, physicists and others for research in
established and emerging areas of chemical interest. Starting from the basic laws of quantum
mechanics, Gaussian predicts the energies, molecular structures, and vibrational frequencies of
molecular systems, along with numerous molecular properties derived from these basic
computation types. It can be used to study molecules and reactions under a wide range of
conditions, including both stable species and compounds which are difficult or impossible to
observe experimentally such as short-lived intermediates and transition structures. This article
introduces several of its new and enhanced features.
Location of Gaussian application:/opt/apps/g03
Submit gaussina job via sge
#vi Gaussian.sh
#!/bin/bash
#$ -S /bin/bash
#$ -N gaussian
#$ -cwd
#$ -j y
#$ -l h_vmem=6G
#$ -v g03root=/opt/apps
#$ -v GAUSS_EXEDIR=/opt/apps/g03/bsd:/opt/apps/g03/linda-exe:/opt/apps/g03
#$ -v GAUSS_LEXEDIR=/opt/apps/g03/linda-exe
#$ -v GDVBASIS=/opt/apps/g03/basis
#$ -v GAUSS_SCRDIR=/scratch
#$ -v PATH=/opt/apps/g03:/opt/apps/g03/bsd:$PATH
#$ -q all.q
/bin/hostname
/bin/date
export g03root GAUSS_EXEDIR GAUSS_LEXEDIR GDVBASIS GAUSS_SCRDIR PATH
/opt/apps/g03/g03 < ./gaussian.in > ./gaussian.out
/bin/date

Save and exit
#qsub Gaussian.sh
#qstat
#qstat -f -q all.q
For deleting job
#qdel <job id>

back
Create a file ".mpd.conf" in your home directory using your favorite editor. The file should contain following line
    MPD_USE_ROOT_MPD=1

Sun Grid Engine (SGE) is the scheduler and resource manager installed on jarasandh

Submitting Serial Job
Let a.out be the executable file. To submit this job
1. Create a shell script file submit.sh (say), which contains
/home/<user>/path/a.out > <outputfile>
2. Submit the job..
qsub -cwd submit.sh
(Since cwd is given, the output will be put in current working directory)

Submitting Parallel Job
Let a.out be the executable file. To submit this job
1. Create a shell script file submit.sh (say), which contains
mpirun -np <number of processors> ./a.out > <outputfile>
2. To submit parallel job through the SGE use the following command
qsub -cwd -pe mpich <no. of processors> submit.sh

Knowing the status of job
qstat  -f -u <username> -q <queue_name>
-f: gives the full detail of the job status
-u: specify the particular user
-q: specify queue name
qstat gives the following columnar details:
jobid     queuename     qtype     used/tot. load_avg     arch     states
The state of the job is mentioned in the "states" column. This column tells you whether job is running, queued, suspended or terminated or terminated.


Killing a job
qdel <jobid>
jobid is the pid of the job submitted which can be found when you use the qstat command.


back
For any queries/problems please send a mail to hpc@unipune.ernet.in

back