How to develop and test code with Dell dlx
The dlx is the University of Kentucky's Dell commodity cluster. it is described here. The system blog is here.
OpenMPI is available.
Slurm is the batch mode job scheduler. Examples are sbatch and squeue.
compilers
We have access to icc 11.1, gcc 4.1.2, and open MPI 1.2rc1 by default.
To build non-mpi exec, execute make in source (for g++), sys_gcc (for g++), or sys_icc (for icc).
To build MPI enabled, do make in source/sys_mpi_icc (the default mpiCC is based on icc 11.1 on the dlx).
Jerry G. email of 2010 Aug 26 says OK to use make -j 4
running the test suite
To run the test suite as a parallel batch job do
sbatch -n <nn> run_parallel.pl sys_gcc dlx
where <nn> is the number of cpus you want. The run_parallel.pl script has a complete description of how to run on this machine. The number of processors <nn> is set by the load leveling across the test suite. Peter suggests <nn> = 12 (i.e. one node) on the regular compute nodes for all test suites.
running a single model in batch mode
I created a script, brun, which is on my path. It contains
sbatch /home/gary/cloudy/trunk/source/sys_icc/cloudy.exe -r $1
Then the script model.in could be computed with the command
brun model
a multi-way MPI rid/optimization run
Make sure you have the correct module loaded. Do
module load mpi/openmpi/icc/default
You will want this in your login script.
Use sys_mpi_icc under source. Running the mpi job is a two step process. First create a batch job, then submit it using the batch processor.
The minimal batch script to run mpi cloudy would be something like this:
#!/bin/csh mpirun /home/gary/cloudy/trunk/source/sys_mpi_icc/cloudy.exe -r $1
If this is contained in mpirun.cs then it would be submitted as follows:
sbatch -n <nn> mpirun.cs feii
where <nn> is the number of processors you want. This will use an input file given by the last parameter on the srun command, feii.in in this case.
srun options
srun is not supported on this machine
sbatch
We need to update the remainder of this page - these were the srun options - do not know what exists for sbatch
There are also the following options to the srun command to do something when a batch job when an event occurs:
--mail-type=type
Notify user by email when certain event types occur.
Valid type values are BEGIN, END, FAIL, ALL (any state change). The user to be notified
is indicated with --mail-user.
--mail-user=user
User to receive email notification of state changes as
defined by --mail-type. The default value is the submitting user.
So adding "--mail-type=END --mail-user=rporter@xxx" to the srun command would announce the end of the batch job.
other slurm options
squeue -uname
will list all jobs in the queue belonging to user name
scancel <nn>
will cancel the run with job ID number <nn>
Other queues
The hi-mem queue requires "-p FatComp"
Return to DeveloperPages
Return to main wiki page
