$ bsub -n4 -I ./myscript.sh | HP XC System 4.x Software specification

Example 5-15 Submitting a Batch Script with the LSF-SLURM External Scheduler Option

$ bsub -n4 -ext "SLURM[nodes=4]" -I ./myscript.sh

Job <79> is submitted to			default queue <normal>.
<<Waiting for	dispatch ...>>
<<Starting on	lsfhost.localdomain>>
n1
n2
n3
n4
Hello world!	I'm 0	of 4	on	n1
Hello world!	I'm 1	of 4	on	n2
Hello world!	I'm 2	of 4	on	n3
Hello world!	I'm 3	of 4	on	n4

Example 5-16and Example 5-17show how the jobs inside the script can be manipulated within the allocation.

Example 5-16 Submitting a Batch Job Script That Uses a Subset of the Allocation

$ bsub -n4 -ext "SLURM[nodes=4]" -I ./myscript.sh Job <80> is submitted to default queue <normal>. <<Waiting for dispatch ...>>

<<Starting on lsfhost.localdomain>> n1

n2

Hello world! I'm 0 of 2 on n1 Hello world! I'm 1 of 2 on n2

Example 5-17 Submitting a Batch job Script That Uses the srun --overcommit Option

$ bsub -n4 -I ./myscript.sh

Job <81> is submitted to default queue <normal>. <<Waiting for dispatch ...>>

<<Starting on lsfhost.localdomain>> n1

n1

n1

n1

n2

n2

n2

n2

Hello world! I'm 0 of 8 on n1 Hello world! I'm 1 of 8 on n1 Hello world! I'm 2 of 8 on n1 Hello world! I'm 3 of 8 on n1 Hello world! I'm 4 of 8 on n2 Hello world! I'm 5 of 8 on n2 Hello world! I'm 6 of 8 on n2 Hello world! I'm 7 of 8 on n2

Example 5-18shows some of the environment variables that are available in a batch script. The LSB_HOSTS and LSB_MCPU_HOSTS environment variables are defined in Platform LSF Reference. The SLURM_JOBID and SLURM_NPROCS environment variables are defined in the SLURM Reference Manual.

5.4 Submitting a Batch Job or Job Script

57

Image 57

HP XC System 4.x Software manual $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh, $ bsub -n4 -I ./myscript.sh

Contents

HP XC System Software XC User Guide Page Table of Contents Compiling and Linking Serial Applications Configuring Your Environment with Modulefiles Srun Roles Srun Modes Debugging Applications 115 107 131 125 List of Figures 10-111-1 List of Tables List of Examples Intended Audience About This DocumentNew and Changed Information in This Edition Typographic Conventions HP XC and Related HP Products Information HP XC System Software Users Guide Related Information Https//computing.llnl.gov/linux/slurm/documentation.html Http//systemimager.org Http//www-unix.mcs.anl.gov/mpi Manpages HP Encourages Your Comments $ man discover $ man 8 discover$ man -k keyword Page Page Node Specialization Only to the administrator of the HP XC systemDetermining the Node Platform /proc/cpuinfo file is dynamic File System Storage and I/OSAN Storage Local Storage System Interconnect Network File System LayoutHP XC System Interconnects User Environment Determining System Configuration InformationNetwork Address Translation NAT 1 LVS Commands Application Development EnvironmentParallel Applications Serial Applications Run-Time EnvironmentSlurm Load Sharing Facility LSF How LSF and Slurm Interact Requested by the HP-MPI mpirun commandHP-MPI Nodes for the job Components, Tools, Compilers, Libraries, and Debuggers Page LVS Login Routing Using the SystemUsing the Secure Shell to Log Logging In to the System Getting Information About Queues IntroductionGetting Information About Resources $ bqueues Performing Other Common User Tasks Determining the LSF Cluster Name and the LSF Execution Host Getting System Help and Information$ lsid $ man sinfo Overview of Modules Configuring Your Environment with Modulefiles Supplied Modulefiles Supplied Modulefiles HP-MPI Viewing Available Modulefiles Modulefiles Automatically Loaded on the SystemViewing Loaded Modulefiles Loading a Modulefile Unloading a Modulefile Viewing Modulefile-Specific HelpModulefile Conflicts $ module load modules $ man modulefile Creating a Modulefile Application Development Environment Overview Developing Applications Compiler Commands CompilersMPI Compiler Examining Nodes and Partitions Before Running Jobs Setting Debugging OptionsInterrupting a Job Developing Serial Applications Serial Application Build Environment Developing Parallel ApplicationsBuilding Serial Applications Parallel Application Build Environment HP-MPI ModulefilesOpenMP Pthreads Intel Fortran and C/C++Compilers MPI LibraryPGI Fortran and C/C++ Compilers GNU C and C++ Compilers Building Parallel Applications To compile and link a C application using the mpicc command Developing LibrariesDesigning Libraries for the CP4000 Platform Linkcommand 32-bit -L/opt/mypackage/lib/i686 -lmystuff Linkcommand 64-bit -L/opt/mypackage/lib/x8664 -lmystuffTo build a 64-bit application, you might enter Developing Applications Submitting a Serial Job with the LSF bsub Command Submitting a Serial Job Using LSFSubmitting Jobs Overview of Job Submission Submitting a Serial Job Through Slurm Only Example 5-2 Submitting a Serial Job Using LSFExample 5-1 Submitting a Job from the Standard Input $ bsub -I srun hostname Submitting a Parallel Job Following is the command line used to compile this programSubmitting a Non-MPI Parallel Job To submit a parallel job $ bsub -n4 -I srun hostname Example 5-5 Submitting a Non-MPI Parallel Job $ bsub -n4 -I mpirun -srun ./helloworld Example 5-7 Submitting an MPI Job Nodelist=list of nodes Arguments for the Slurm External Scheduler $ bsub -n 10 -ext SLURMnodelist=n1-10 srun hostname Example 5-14 Submitting a Job Script Submitting a Batch Job or Job Script$ bsub -n 10 -ext SLURMconstraint=dualcore -I srun hostname $ bqueues -l dualcore grep Slurm $ bsub -n4 -I ./myscript.sh $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh Using a Makefile to Submit Multiple Jobs Using a Script to Submit Multiple JobsUse the squeue command to acquire information on the jobs Submitting Multiple MPI Jobs Across the Same Set of Nodes Following command line makes the program and executes it $ cat mymake$ tail 113.out $ cat 117.out Running Preexecution Programs Submitting a Job from a Host Other Than an HP XC HostType=SLINUX64 $ bsub -R type=SLINUX64 -n4 -I srun hostnamePage Debugging Serial Applications Debugging ApplicationsDebugging Parallel Applications TotalView Debugging with TotalView Setting Up TotalViewSSH and TotalView Module load mpi module load totalview Using TotalView with LSF Using TotalView with SlurmSetting TotalView Preferences $ srun -Nx-A $ mpirun -tv -srun application $ mpirun -tv -srun -n2 ./Psimple Debugging an Application Debugging Running Applications $ scancel --user usernameExiting TotalView Run the applicationPage Xtools Utilities Monitoring Node Activity Where Running Performance Health Tests $ ovp -l You can list the available tests with the ovp -lcommand$ ovp --verify=perfhealth/cpuusage $ ovp --verbose --verify=perfhealth/cpuusage HOMEDIRECTORY/ovpn16mmddyy.log HOMEDIRECTORY/ovpn16mmddyyr1.log Page Tuning Applications Using the Intel Trace Collector and Intel Trace AnalyzerBuilding a Program Intel Trace Collector and HP-MPI Example 8-1 The vtjacobic Example Program Running a Program Intel Trace Collector and HP-MPIFor more information, see the following Web site Example 8-2 C Example Running the vtjacobic Example Program Installation Kit Intel Trace Collector and Analyzer with HP-MPI on HP XCHP-MPI and the Intel Trace Collector Following is a Fortran example called vtjacobif Running a Program Running a Program Across Nodes Using LSF Visualizing Data Intel Trace Analyzer and HP-MPI# bsub -n4 -I mpirun.mpich -np 2 ./vtjacobic Page Launching Jobs with the srun Command Using SlurmIntroduction to Slurm Slurm Utilities Using the srun Command with HP-MPI Monitoring Jobs with the squeue CommandUsing the srun Command with LSF Srun Roles and Modes Terminating Jobs with the scancel Command Getting System Information with the sinfo CommandExample 9-5cancels all pending jobs Job Accounting Fault ToleranceSecurity $ sinfo -R Information for LSF Using LSF $ squeue --jobs $SLURMJOBID Overview of LSF Integrated with Slurm Example 10-1 Examples of LSF Job Launch Lsfadmin@n16 ~$ bsub -n4 -I srun hostname Differences Between LSF and LSF Integrated with Slurm Lshosts and lsload commands display for each of these items Connection back to the terminal from which the job was Job TerminologySubmitted. This job may run immediately, or it may run $ ssh n15 lshosts Default, the output is written to Batch jobs are submitted with the srun -bcommand. ByAre submitted with the bsub -Icommand Batch system scheduling policies Useful Commands Using LSF Integrated with Slurm in the HP XC EnvironmentSubmitting Jobs Job Startup and Job Control LSF-SLURM External Scheduler How LSF and Slurm Launch and Manage a JobLSF with Slurm Job Launch Exit Codes How LSF and Slurm Launch and Manage a Job User logs in to login node n16 Determining Available System Resources Determining the LSF Execution Host Following example shows the output from the lshosts command Examining System Core StatusGetting Information About the LSF Execution Host Node Getting Host Load Information Getting Information About JobsExamining System Queues Getting Information About the lsf Partition This allocation string has the following values Getting Job Allocation InformationLSF job with Slurm allocated resources Than what the job requests Example 10-5 Using the bjobs Command Short Output Examining the Status of a JobExample 10-4 Job Allocation Information for a Finished Job Example 10-7 Using the bhist Command Short Output Example 10-6 Using the bjobs Command Long OutputOutput Provided by the bhist Command Viewing the Historical Information for a Job Example 10-8 Using the bhist Command Long Output Use the bjobs command to view the Slurm JobidTranslating Slurm and LSF JOBIDs $ sacct -j Working Interactively Within an Allocation$ bsub -I -n4 -ext SLURMnodes=4 /bin/bash $ bjobs -l 124 grep slurm Example 10-9 Launching an Interactive MPI Job This example assumes 2 cores per nodeAlternatively, you can use the following 3describes the srun options and lists their LSF equivalents LSF Equivalents of Slurm srun OptionsLSF Equivalents of Slurm srun Options $ unset Slurmjobid $ unset Slurmnprocs Requests a specific list of nodes. Slurm Suppress informational message Enabling Remote Execution with OpenSSH Advanced TopicsRunning an X Terminal Session from a Remote Node Determining IP Address of Your Local Machine Running an X terminal Session Using Slurm Running an X terminal Session Using LSFLogging in to HP XC System Options used in this command are Allocate 4 cores Using the GNU Parallel Make CapabilityRun the job on 1 core Monitors display server address $ cd subdir srun -n1 -N1 $MAKE -j4 $ cd subdir srun $MAKE $ make PREFIX=’srun -n1 -N1 MAKEJ=-j4 Example Procedure Local Disks on Compute Nodes Modified Makefile is invoked as follows$ make PREFIX=srun -n1 -N1 MAKEJ=-j4 11.5 I/O Performance Considerations Using Mpich on the HP XC SystemCommunication Between Nodes Shared File View Using Mpich with Slurm Allocation Using Mpich with LSF AllocationBsub command launches the wrapper script Building and Running a Serial Application ExamplesLaunching a Serial Interactive Shell Through LSF Examine the LSF execution host information Running LSF Jobs with a Slurm Allocation Request Example 2. Four Cores on Two Specific Nodes Launching a Parallel Interactive Shell Through LSF $ hostname n16 $ srun hostname n5 $ bjobs Examine the running jobs information Examine the finished jobs information Submitting a Simple Job Script with LSFShow the environment Display the script Run the job Submitting an Interactive Job with LSFSubmit the job Show the job allocation Show the Slurm job ID Run some commands from the pseudo-terminalExit the pseudo-terminal View the interactive jobs Submitting an HP-MPI Job with LSF View the node stateView the running job Using a Resource Requirements String in an LSF Command View the finished job$ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob 124 Network Availability set Glossary Fcfs Ipmi LVS PXE SVA Index Index LVS PGI 135