$ cat 117.out | HP XC System 4.x Software manual

1This line attempts to submit a program that does not exist.

The following command line makes the program and executes it:

$ bsub -o %J.out -n2 -ext "SLURM[nodes=2]" make -j3 \ -f ./mymake PPR_ARGS=100000

Job <117> is submitted to default queue <normal>.

The output file contains error messages related to the attempt to launch the nonexistent program.

$ cat 117.out

.

.

.

mpirun -srun -N 2 -n 4 ./ping_pong_ring 100000 mpirun -srun -N 2 -n 4 ./ping_pong_ring 100000 mpirun -srun -N 2 -n 4 ./ping_bogus 100000

slurmstepd: [19.3]: error: execve(): ./ping_bogus: No such file or directory

slurmstepd: [19.3]: error: execve(): ./ping_bogus: No such file or directory

srun: error: n14: task0: Exited with exit code 2

srun: Terminating job

slurmstepd: [19.3]: error: execve(): ./ping_bogus: No such file or directory

slurmstepd: [19.3]: error: execve(): ./ping_bogus: No such file or directory

make: *** [run3] Error 2

make: *** Waiting for unfinished jobs....

[0:n14] ping-pong 100000 bytes ...

100000 bytes: 99.06 usec/msg

100000 bytes: 1009.51 MB/sec [0:n14] ping-pong 100000 bytes ...

100000 bytes: 99.76 usec/msg

100000 bytes: 1002.43 MB/sec [1:n14] ping-pong 100000 bytes ...

100000 bytes: 1516.83 usec/msg

100000 bytes: 65.93 MB/sec [1:n14] ping-pong 100000 bytes ...

100000 bytes: 1519.73 usec/msg

100000 bytes: 65.80 MB/sec [2:n15] ping-pong 100000 bytes ...

100000 bytes: 108.65 usec/msg

100000 bytes: 920.38 MB/sec [2:n15] ping-pong 100000 bytes ...

100000 bytes: 99.44 usec/msg

100000 bytes: 1005.65 MB/sec [3:n15] ping-pong 100000 bytes ...

100000 bytes: 1877.35 usec/msg

100000 bytes: 53.27 MB/sec [3:n15] ping-pong 100000 bytes ...

100000 bytes: 1888.22 usec/msg

100000 bytes: 52.96 MB/sec

The sacct command, which displays SLURM accounting information, reflects the error:

[lsfadmin@n16 ~]$ sacct -j 19
Jobstep	Jobname	Partition	Ncpus	Status	Error
----------	------------------	---------- -------		---------- -----
19	hptclsf@117	lsf	8	CANCELLED	2
19.0	hptclsf@117	lsf	0	FAILED	2
19.1	hptclsf@117	lsf	8	COMPLETED	0
19.2	hptclsf@117	lsf	8	COMPLETED	0
19.3	hptclsf@117	lsf	8	FAILED	2

60 Submitting Jobs

Image 60

HP XC System 4.x Software manual $ cat 117.out

Contents

HP XC System Software XC User Guide Page Table of Contents Configuring Your Environment with Modulefiles Compiling and Linking Serial Applications Debugging Applications Srun Roles Srun Modes 107 115 125 131 List of Figures 10-111-1 List of Tables List of Examples About This Document Intended AudienceNew and Changed Information in This Edition Typographic Conventions HP XC and Related HP Products Information Related Information HP XC System Software Users Guide Https//computing.llnl.gov/linux/slurm/documentation.html Http//systemimager.org Manpages Http//www-unix.mcs.anl.gov/mpi HP Encourages Your Comments $ man discover $ man 8 discover$ man -k keyword Page Page Only to the administrator of the HP XC system Node SpecializationDetermining the Node Platform /proc/cpuinfo file is dynamic Storage and I/O File SystemSAN Storage Local Storage System Interconnect Network File System LayoutHP XC System Interconnects Determining System Configuration Information User EnvironmentNetwork Address Translation NAT 1 LVS Commands Application Development EnvironmentParallel Applications Run-Time Environment Serial ApplicationsSlurm Load Sharing Facility LSF Requested by the HP-MPI mpirun command How LSF and Slurm InteractHP-MPI Nodes for the job Components, Tools, Compilers, Libraries, and Debuggers Page Using the System LVS Login RoutingUsing the Secure Shell to Log Logging In to the System Introduction Getting Information About QueuesGetting Information About Resources $ bqueues Performing Other Common User Tasks Getting System Help and Information Determining the LSF Cluster Name and the LSF Execution Host$ lsid $ man sinfo Configuring Your Environment with Modulefiles Overview of Modules Supplied Modulefiles Supplied Modulefiles HP-MPI Modulefiles Automatically Loaded on the System Viewing Available ModulefilesViewing Loaded Modulefiles Loading a Modulefile Unloading a Modulefile Viewing Modulefile-Specific HelpModulefile Conflicts Creating a Modulefile $ module load modules $ man modulefile Developing Applications Application Development Environment Overview Compiler Commands CompilersMPI Compiler Setting Debugging Options Examining Nodes and Partitions Before Running JobsInterrupting a Job Developing Serial Applications Developing Parallel Applications Serial Application Build EnvironmentBuilding Serial Applications Parallel Application Build Environment Modulefiles HP-MPIOpenMP Pthreads MPI Library Intel Fortran and C/C++CompilersPGI Fortran and C/C++ Compilers GNU C and C++ Compilers Building Parallel Applications To compile and link a C application using the mpicc command Developing LibrariesDesigning Libraries for the CP4000 Platform Linkcommand 32-bit -L/opt/mypackage/lib/i686 -lmystuff Linkcommand 64-bit -L/opt/mypackage/lib/x8664 -lmystuffTo build a 64-bit application, you might enter Developing Applications Submitting a Serial Job Using LSF Submitting a Serial Job with the LSF bsub CommandSubmitting Jobs Overview of Job Submission Example 5-2 Submitting a Serial Job Using LSF Submitting a Serial Job Through Slurm OnlyExample 5-1 Submitting a Job from the Standard Input $ bsub -I srun hostname Following is the command line used to compile this program Submitting a Parallel JobSubmitting a Non-MPI Parallel Job To submit a parallel job Example 5-5 Submitting a Non-MPI Parallel Job $ bsub -n4 -I srun hostname Example 5-7 Submitting an MPI Job $ bsub -n4 -I mpirun -srun ./helloworld Arguments for the Slurm External Scheduler Nodelist=list of nodes $ bsub -n 10 -ext SLURMnodelist=n1-10 srun hostname Submitting a Batch Job or Job Script Example 5-14 Submitting a Job Script$ bsub -n 10 -ext SLURMconstraint=dualcore -I srun hostname $ bqueues -l dualcore grep Slurm $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh $ bsub -n4 -I ./myscript.sh Using a Script to Submit Multiple Jobs Using a Makefile to Submit Multiple JobsUse the squeue command to acquire information on the jobs Submitting Multiple MPI Jobs Across the Same Set of Nodes Following command line makes the program and executes it $ cat mymake$ tail 113.out $ cat 117.out Submitting a Job from a Host Other Than an HP XC Host Running Preexecution ProgramsType=SLINUX64 $ bsub -R type=SLINUX64 -n4 -I srun hostnamePage Debugging Applications Debugging Serial ApplicationsDebugging Parallel Applications TotalView Setting Up TotalView Debugging with TotalViewSSH and TotalView Module load mpi module load totalview Using TotalView with Slurm Using TotalView with LSFSetting TotalView Preferences $ srun -Nx-A $ mpirun -tv -srun application Debugging an Application $ mpirun -tv -srun -n2 ./Psimple $ scancel --user username Debugging Running ApplicationsExiting TotalView Run the applicationPage Monitoring Node Activity Xtools Utilities Running Performance Health Tests Where You can list the available tests with the ovp -lcommand $ ovp -l$ ovp --verify=perfhealth/cpuusage $ ovp --verbose --verify=perfhealth/cpuusage HOMEDIRECTORY/ovpn16mmddyy.log HOMEDIRECTORY/ovpn16mmddyyr1.log Page Tuning Applications Using the Intel Trace Collector and Intel Trace AnalyzerBuilding a Program Intel Trace Collector and HP-MPI Running a Program Intel Trace Collector and HP-MPI Example 8-1 The vtjacobic Example ProgramFor more information, see the following Web site Example 8-2 C Example Running the vtjacobic Example Program Installation Kit Intel Trace Collector and Analyzer with HP-MPI on HP XCHP-MPI and the Intel Trace Collector Running a Program Following is a Fortran example called vtjacobif Running a Program Across Nodes Using LSF Visualizing Data Intel Trace Analyzer and HP-MPI# bsub -n4 -I mpirun.mpich -np 2 ./vtjacobic Page Using Slurm Launching Jobs with the srun CommandIntroduction to Slurm Slurm Utilities Monitoring Jobs with the squeue Command Using the srun Command with HP-MPIUsing the srun Command with LSF Srun Roles and Modes Terminating Jobs with the scancel Command Getting System Information with the sinfo CommandExample 9-5cancels all pending jobs Fault Tolerance Job AccountingSecurity $ sinfo -R Using LSF Information for LSF Overview of LSF Integrated with Slurm $ squeue --jobs $SLURMJOBID Lsfadmin@n16 ~$ bsub -n4 -I srun hostname Example 10-1 Examples of LSF Job Launch Lshosts and lsload commands display for each of these items Differences Between LSF and LSF Integrated with Slurm Job Terminology Connection back to the terminal from which the job wasSubmitted. This job may run immediately, or it may run $ ssh n15 lshosts Batch jobs are submitted with the srun -bcommand. By Default, the output is written toAre submitted with the bsub -Icommand Batch system scheduling policies Using LSF Integrated with Slurm in the HP XC Environment Useful CommandsSubmitting Jobs Job Startup and Job Control LSF-SLURM External Scheduler How LSF and Slurm Launch and Manage a JobLSF with Slurm Job Launch Exit Codes User logs in to login node n16 How LSF and Slurm Launch and Manage a Job Determining the LSF Execution Host Determining Available System Resources Following example shows the output from the lshosts command Examining System Core StatusGetting Information About the LSF Execution Host Node Getting Information About Jobs Getting Host Load InformationExamining System Queues Getting Information About the lsf Partition Getting Job Allocation Information This allocation string has the following valuesLSF job with Slurm allocated resources Than what the job requests Example 10-5 Using the bjobs Command Short Output Examining the Status of a JobExample 10-4 Job Allocation Information for a Finished Job Example 10-6 Using the bjobs Command Long Output Example 10-7 Using the bhist Command Short OutputOutput Provided by the bhist Command Viewing the Historical Information for a Job Example 10-8 Using the bhist Command Long Output Use the bjobs command to view the Slurm JobidTranslating Slurm and LSF JOBIDs Working Interactively Within an Allocation $ sacct -j$ bsub -I -n4 -ext SLURMnodes=4 /bin/bash $ bjobs -l 124 grep slurm Example 10-9 Launching an Interactive MPI Job This example assumes 2 cores per nodeAlternatively, you can use the following LSF Equivalents of Slurm srun Options 3describes the srun options and lists their LSF equivalentsLSF Equivalents of Slurm srun Options $ unset Slurmjobid $ unset Slurmnprocs Requests a specific list of nodes. Slurm Suppress informational message Advanced Topics Enabling Remote Execution with OpenSSHRunning an X Terminal Session from a Remote Node Determining IP Address of Your Local Machine Running an X terminal Session Using Slurm Running an X terminal Session Using LSFLogging in to HP XC System Using the GNU Parallel Make Capability Options used in this command are Allocate 4 coresRun the job on 1 core Monitors display server address $ cd subdir srun $MAKE $ cd subdir srun -n1 -N1 $MAKE -j4 Example Procedure $ make PREFIX=’srun -n1 -N1 MAKEJ=-j4 Local Disks on Compute Nodes Modified Makefile is invoked as follows$ make PREFIX=srun -n1 -N1 MAKEJ=-j4 Using Mpich on the HP XC System 11.5 I/O Performance ConsiderationsCommunication Between Nodes Shared File View Using Mpich with Slurm Allocation Using Mpich with LSF AllocationBsub command launches the wrapper script Examples Building and Running a Serial ApplicationLaunching a Serial Interactive Shell Through LSF Examine the LSF execution host information Running LSF Jobs with a Slurm Allocation Request Launching a Parallel Interactive Shell Through LSF Example 2. Four Cores on Two Specific Nodes Examine the running jobs information $ hostname n16 $ srun hostname n5 $ bjobs Submitting a Simple Job Script with LSF Examine the finished jobs informationShow the environment Display the script Submitting an Interactive Job with LSF Run the jobSubmit the job Show the job allocation Run some commands from the pseudo-terminal Show the Slurm job IDExit the pseudo-terminal View the interactive jobs Submitting an HP-MPI Job with LSF View the node stateView the running job Using a Resource Requirements String in an LSF Command View the finished job$ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob 124 Glossary Network Availability set Fcfs Ipmi LVS PXE SVA Index Index LVS PGI 135