HP XC System 4.x Software manual Example 5-7 Submitting an MPI Job

variable that was set by LSF; this environment variable is equivalent to the number provided by the -noption of the bsub command.

Any additional SLURM srun options are job specific, not allocation-specific.

The mpi-jobnameis the executable file to be run. The mpi-jobnamemust be compiled with the appropriate HP-MPI compilation utility. For more information, see the section titled Compiling applications in the HP-MPI User's Guide.

Example 5-7shows an MPI job that runs a hello world program on 4 cores on 2 compute nodes.

Example 5-7 Submitting an MPI Job

$ bsub -n4 -I mpirun -srun ./hello_world

Job <24> is submitted to default queue <normal>. <<Waiting for dispatch ...>>

<<Starting on lsfhost.localdomain>> Hello world!

Hello world! I'm 1 of 4 on host1 Hello world! I'm 3 of 4 on host2 Hello world! I'm 0 of 4 on host1 Hello world! I'm 2 of 4 on host2

You can use the LSF-SLURM External Scheduler option to add capabilities at the job level and queue level by including several SLURM options in the command line. For example, you can use it to submit a job to run one task per node, or to submit a job to run on specific nodes.

“LSF-SLURM External Scheduler” discusses this option.

Example 5-8shows an MPI job that uses the LSF-SLURM External Scheduler option to run the same hello world program on each of 4 compute nodes.

Example 5-8 Submitting an MPI Job with the LSF-SLURM External Scheduler Option

$ bsub -n4 -ext "SLURM[nodes=4]" -I mpirun -srun ./hello_world Job <27> is submitted to default queue <normal>.

<<Waiting for dispatch ...>> <<Starting on lsfhost.localdomain>> Hello world!

Hello world! I'm 1 of 4 on host1 Hello world! I'm 2 of 4 on host2 Hello world! I'm 3 of 4 on host3 Hello world! I'm 4 of 4 on host4

Some preprocessing may need to be done:

•If the MPI job requires the use of an appfile

•If there is another reason that prohibits the use of the srun command as the task launcher

This preprocessing should determine the node host names to which mpirun's standard task launcher should launch the tasks. In such scenarios, you need to write a batch script; there are several methods available for determining the nodes in an allocation. One method is to use the SLURM_JOBID environment variable with the squeue command to query the nodes. Another method is to use the LSF environment variables such as LSB_HOSTS and LSB_MCPU_HOSTS, which are prepared by the HP XC job starter script.

5.3.3 Submitting a Parallel Job Using the SLURM External Scheduler

The LSF-SLURM external scheduler option provides additional capabilities at the job level and queue level by allowing the inclusion of several SLURM options in the LSF command line. With LSF integrated with SLURM, you can use the LSF-SLURM External Scheduler to specify SLURM

5.3 Submitting a Parallel Job 53