Show the Slurm job ID | HP XC System 4.x Software instruction

Show the SLURM job ID:

$ env grep SLURM SLURM_JOBID=74 SLURM_NPROCS=8

Run some commands from the pseudo-terminal:

$ srun hostname n13

n13

n14

n14

n15

n15

n16

n16

$ srun -n3 hostname n13

n14

n15

Exit the pseudo-terminal:

$ exit exit

View the interactive jobs:

$ bjobs -l 1008

Job <1008>, User smith, Project <default>, Status <DONE>, Queue <normal>, Interactive pseudo-terminal mode, Command </bin/sh>

date and time stamp: Submitted from host n16,

CWD <$HOME/tar_drop1/test>, 8 Processors Requested; date and time stamp: Started on 8Hosts/Processors<8*lsfhost.localdomain>; date and time stamp: slurm_id=74;ncpus=8;slurm_alloc=n16,n14,n13,n15; date and time stamp: Done successfully. The CPU time used is 0.1 seconds.

SCHEDULING	PARAMETERS:
	r15s	r1m	r15m	ut	pg	io	ls	it	tmp	swp	mem
loadSched	-	-	-	-	-	-	-	-	-	-	-
loadStop	-	-	-	-	-	-	-	-	-	-	-

View the finished jobs:

$ bhist -l 1008

Job <1008>, User smith, Project <default>,

Interactive pseudo-terminal mode, Command </bin/sh>

date and time stamp: Submitted from host n16, to Queue <normal>,

CWD <$HOME/tar_drop1/test>, 8 Processors Requested;

date and time stamp: Dispatched to 8 Hosts/Processors <8*lsfhost.localdomain>;

date and time stamp: slurm_id=74;ncpus=8;slurm_alloc=n16,n14,n13,n15;

date and time stamp: Starting (Pid 26446);

date and time stamp: Done successfully. The CPU time used is 0.1 seconds;

date and time stamp: Post job process done successfully;

Summary of time in seconds spent in various states by date and time

PEND	PSUSP	RUN	USUSP	SSUSP	UNKWN	TOTAL
12	0	93	0	0	0	105

A.6 Submitting an Interactive Job with LSF 121

Image 121

HP XC System 4.x Software Show the Slurm job ID, Run some commands from the pseudo-terminal, Exit the pseudo-terminal

Contents

HP XC System Software XC User Guide Page Table of Contents Compiling and Linking Serial Applications Configuring Your Environment with Modulefiles Srun Roles Srun Modes Debugging Applications 115 107 131 125 10-1 List of Figures11-1 List of Tables List of Examples Intended Audience About This DocumentNew and Changed Information in This Edition Typographic Conventions HP XC and Related HP Products Information HP XC System Software Users Guide Related Information Https//computing.llnl.gov/linux/slurm/documentation.html Http//systemimager.org Http//www-unix.mcs.anl.gov/mpi Manpages $ man discover $ man 8 discover HP Encourages Your Comments$ man -k keyword Page Page Node Specialization Only to the administrator of the HP XC systemDetermining the Node Platform /proc/cpuinfo file is dynamic File System Storage and I/OSAN Storage Local Storage File System Layout System Interconnect NetworkHP XC System Interconnects User Environment Determining System Configuration InformationNetwork Address Translation NAT 1 LVS Application Development Environment CommandsParallel Applications Serial Applications Run-Time EnvironmentSlurm Load Sharing Facility LSF How LSF and Slurm Interact Requested by the HP-MPI mpirun commandHP-MPI Nodes for the job Components, Tools, Compilers, Libraries, and Debuggers Page LVS Login Routing Using the SystemUsing the Secure Shell to Log Logging In to the System Getting Information About Queues IntroductionGetting Information About Resources $ bqueues Performing Other Common User Tasks Determining the LSF Cluster Name and the LSF Execution Host Getting System Help and Information$ lsid $ man sinfo Overview of Modules Configuring Your Environment with Modulefiles Supplied Modulefiles Supplied Modulefiles HP-MPI Viewing Available Modulefiles Modulefiles Automatically Loaded on the SystemViewing Loaded Modulefiles Loading a Modulefile Viewing Modulefile-Specific Help Unloading a ModulefileModulefile Conflicts $ module load modules $ man modulefile Creating a Modulefile Application Development Environment Overview Developing Applications Compilers Compiler CommandsMPI Compiler Examining Nodes and Partitions Before Running Jobs Setting Debugging OptionsInterrupting a Job Developing Serial Applications Serial Application Build Environment Developing Parallel ApplicationsBuilding Serial Applications Parallel Application Build Environment HP-MPI ModulefilesOpenMP Pthreads Intel Fortran and C/C++Compilers MPI LibraryPGI Fortran and C/C++ Compilers GNU C and C++ Compilers Building Parallel Applications Developing Libraries To compile and link a C application using the mpicc commandDesigning Libraries for the CP4000 Platform Linkcommand 64-bit -L/opt/mypackage/lib/x8664 -lmystuff Linkcommand 32-bit -L/opt/mypackage/lib/i686 -lmystuffTo build a 64-bit application, you might enter Developing Applications Submitting a Serial Job with the LSF bsub Command Submitting a Serial Job Using LSFSubmitting Jobs Overview of Job Submission Submitting a Serial Job Through Slurm Only Example 5-2 Submitting a Serial Job Using LSFExample 5-1 Submitting a Job from the Standard Input $ bsub -I srun hostname Submitting a Parallel Job Following is the command line used to compile this programSubmitting a Non-MPI Parallel Job To submit a parallel job $ bsub -n4 -I srun hostname Example 5-5 Submitting a Non-MPI Parallel Job $ bsub -n4 -I mpirun -srun ./helloworld Example 5-7 Submitting an MPI Job Nodelist=list of nodes Arguments for the Slurm External Scheduler $ bsub -n 10 -ext SLURMnodelist=n1-10 srun hostname Example 5-14 Submitting a Job Script Submitting a Batch Job or Job Script$ bsub -n 10 -ext SLURMconstraint=dualcore -I srun hostname $ bqueues -l dualcore grep Slurm $ bsub -n4 -I ./myscript.sh $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh Using a Makefile to Submit Multiple Jobs Using a Script to Submit Multiple JobsUse the squeue command to acquire information on the jobs Submitting Multiple MPI Jobs Across the Same Set of Nodes $ cat mymake Following command line makes the program and executes it$ tail 113.out $ cat 117.out Running Preexecution Programs Submitting a Job from a Host Other Than an HP XC HostType=SLINUX64 $ bsub -R type=SLINUX64 -n4 -I srun hostnamePage Debugging Serial Applications Debugging ApplicationsDebugging Parallel Applications TotalView Debugging with TotalView Setting Up TotalViewSSH and TotalView Module load mpi module load totalview Using TotalView with LSF Using TotalView with SlurmSetting TotalView Preferences $ srun -Nx-A $ mpirun -tv -srun application $ mpirun -tv -srun -n2 ./Psimple Debugging an Application Debugging Running Applications $ scancel --user usernameExiting TotalView Run the applicationPage Xtools Utilities Monitoring Node Activity Where Running Performance Health Tests $ ovp -l You can list the available tests with the ovp -lcommand$ ovp --verify=perfhealth/cpuusage $ ovp --verbose --verify=perfhealth/cpuusage HOMEDIRECTORY/ovpn16mmddyy.log HOMEDIRECTORY/ovpn16mmddyyr1.log Page Using the Intel Trace Collector and Intel Trace Analyzer Tuning ApplicationsBuilding a Program Intel Trace Collector and HP-MPI Example 8-1 The vtjacobic Example Program Running a Program Intel Trace Collector and HP-MPIFor more information, see the following Web site Example 8-2 C Example Running the vtjacobic Example Program Intel Trace Collector and Analyzer with HP-MPI on HP XC Installation KitHP-MPI and the Intel Trace Collector Following is a Fortran example called vtjacobif Running a Program Visualizing Data Intel Trace Analyzer and HP-MPI Running a Program Across Nodes Using LSF# bsub -n4 -I mpirun.mpich -np 2 ./vtjacobic Page Launching Jobs with the srun Command Using SlurmIntroduction to Slurm Slurm Utilities Using the srun Command with HP-MPI Monitoring Jobs with the squeue CommandUsing the srun Command with LSF Srun Roles and Modes Getting System Information with the sinfo Command Terminating Jobs with the scancel CommandExample 9-5cancels all pending jobs Job Accounting Fault ToleranceSecurity $ sinfo -R Information for LSF Using LSF $ squeue --jobs $SLURMJOBID Overview of LSF Integrated with Slurm Example 10-1 Examples of LSF Job Launch Lsfadmin@n16 ~$ bsub -n4 -I srun hostname Differences Between LSF and LSF Integrated with Slurm Lshosts and lsload commands display for each of these items Connection back to the terminal from which the job was Job TerminologySubmitted. This job may run immediately, or it may run $ ssh n15 lshosts Default, the output is written to Batch jobs are submitted with the srun -bcommand. ByAre submitted with the bsub -Icommand Batch system scheduling policies Useful Commands Using LSF Integrated with Slurm in the HP XC EnvironmentSubmitting Jobs Job Startup and Job Control How LSF and Slurm Launch and Manage a Job LSF-SLURM External SchedulerLSF with Slurm Job Launch Exit Codes How LSF and Slurm Launch and Manage a Job User logs in to login node n16 Determining Available System Resources Determining the LSF Execution Host Examining System Core Status Following example shows the output from the lshosts commandGetting Information About the LSF Execution Host Node Getting Host Load Information Getting Information About JobsExamining System Queues Getting Information About the lsf Partition This allocation string has the following values Getting Job Allocation InformationLSF job with Slurm allocated resources Than what the job requests Examining the Status of a Job Example 10-5 Using the bjobs Command Short OutputExample 10-4 Job Allocation Information for a Finished Job Example 10-7 Using the bhist Command Short Output Example 10-6 Using the bjobs Command Long OutputOutput Provided by the bhist Command Viewing the Historical Information for a Job Use the bjobs command to view the Slurm Jobid Example 10-8 Using the bhist Command Long OutputTranslating Slurm and LSF JOBIDs $ sacct -j Working Interactively Within an Allocation$ bsub -I -n4 -ext SLURMnodes=4 /bin/bash $ bjobs -l 124 grep slurm This example assumes 2 cores per node Example 10-9 Launching an Interactive MPI JobAlternatively, you can use the following 3describes the srun options and lists their LSF equivalents LSF Equivalents of Slurm srun OptionsLSF Equivalents of Slurm srun Options $ unset Slurmjobid $ unset Slurmnprocs Requests a specific list of nodes. Slurm Suppress informational message Enabling Remote Execution with OpenSSH Advanced TopicsRunning an X Terminal Session from a Remote Node Determining IP Address of Your Local Machine Running an X terminal Session Using LSF Running an X terminal Session Using SlurmLogging in to HP XC System Options used in this command are Allocate 4 cores Using the GNU Parallel Make CapabilityRun the job on 1 core Monitors display server address $ cd subdir srun -n1 -N1 $MAKE -j4 $ cd subdir srun $MAKE $ make PREFIX=’srun -n1 -N1 MAKEJ=-j4 Example Procedure Modified Makefile is invoked as follows Local Disks on Compute Nodes$ make PREFIX=srun -n1 -N1 MAKEJ=-j4 11.5 I/O Performance Considerations Using Mpich on the HP XC SystemCommunication Between Nodes Shared File View Using Mpich with LSF Allocation Using Mpich with Slurm AllocationBsub command launches the wrapper script Building and Running a Serial Application ExamplesLaunching a Serial Interactive Shell Through LSF Examine the LSF execution host information Running LSF Jobs with a Slurm Allocation Request Example 2. Four Cores on Two Specific Nodes Launching a Parallel Interactive Shell Through LSF $ hostname n16 $ srun hostname n5 $ bjobs Examine the running jobs information Examine the finished jobs information Submitting a Simple Job Script with LSFShow the environment Display the script Run the job Submitting an Interactive Job with LSFSubmit the job Show the job allocation Show the Slurm job ID Run some commands from the pseudo-terminalExit the pseudo-terminal View the interactive jobs View the node state Submitting an HP-MPI Job with LSFView the running job View the finished job Using a Resource Requirements String in an LSF Command$ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob 124 Network Availability set Glossary Fcfs Ipmi LVS PXE SVA Index Index LVS PGI 135