Submitting an HP-MPI Job with LSF-HPC | HP XC System 3.x Software

n15

n15

n16

n16

$ srun -n3 hostname n13

n14

n15

Exit the pseudo-terminal:

$ exit exit

View the interactive jobs:

$ bjobs -l 1008

Job <1008>, User smith, Project <default>, Status <DONE>, Queue <normal>, Interactive pseudo-terminal mode, Command </bin/sh>

date and time stamp: Submitted from host n16,

CWD <$HOME/tar_drop1/test>, 8 Processors Requested; date and time stamp: Started on 8Hosts/Processors<8*lsfhost.localdomain>; date and time stamp: slurm_id=74;ncpus=8;slurm_alloc=n16,n14,n13,n15; date and time stamp: Done successfully. The CPU time used is 0.1 seconds.

SCHEDULING	PARAMETERS:
	r15s	r1m	r15m	ut	pg	io	ls	it	tmp	swp	mem
loadSched	-	-	-	-	-	-	-	-	-	-	-
loadStop	-	-	-	-	-	-	-	-	-	-	-

View the finished jobs:

$ bhist -l 1008

Job <1008>, User smith, Project <default>,

Interactive pseudo-terminal mode, Command </bin/sh>

date and time stamp: Submitted from host n16, to Queue <normal>,

CWD <$HOME/tar_drop1/test>, 8 Processors Requested;

date and time stamp: Dispatched to 8 Hosts/Processors <8*lsfhost.localdomain>;

date and time stamp: slurm_id=74;ncpus=8;slurm_alloc=n16,n14,n13,n15;

date and time stamp: Starting (Pid 26446);

date and time stamp: Done successfully. The CPU time used is 0.1 seconds;

date and time stamp: Post job process done successfully;

Summary of time in seconds spent in various states by date and time

PEND	PSUSP	RUN	USUSP	SSUSP	UNKWN	TOTAL
12	0	93	0	0	0	105

View the node state:

$ sinfo
PARTITION AVAIL		TIMELIMIT	NODES	STATE	NODELIST
lsf	up	infinite	4	idle	n[13-16]

Submitting an HP-MPI Job with LSF-HPC

This example shows how to run an MPI job with the bsub command.

Show the environment:

$ lsid

Platform LSF HPC 6.0 for SLURM, Sep 23 2004

Exit the pseudo-terminal: 105

Image 105

HP XC System 3.x Software manual Submitting an HP-MPI Job with LSF-HPC, Exit the pseudo-terminal, View the interactive jobs

Contents

HP XC System Software Users Guide Page Table of Contents Configuring Your Environment with Modulefiles Developing ApplicationsSubmitting Jobs Using Slurm Tuning ApplicationsUsing LSF Debugging Applications Advanced Topics ExamplesGlossary 109 Index 115 List of Figures Page Determining the Node Platform List of TablesPage Submitting a Job Script List of ExamplesPage Intended Audience About This DocumentDocument Organization This document is organized as follows HP XC Information $ man lsfcommandname For More InformationSupplementary Information Manpages Related Information$ man discover $ man 8 discover $ man -k keyword Related MPI Web Sites Related Linux Web SitesRelated Compiler Web Sites Additional Publications HP Encourages Your Comments Typographic ConventionsEnvironment Variable User input System Architecture Overview of the User EnvironmentHP XC System Software Operating System Storage and I/O Node Specialization SAN Storage File SystemLocal Storage File System Layout Determining System Configuration Information System Interconnect NetworkNetwork Address Translation NAT Commands User EnvironmentModules Run-Time Environment Application Development EnvironmentParallel Applications Serial Applications Load Sharing Facility LSF-HPC Standard LSFHow LSF-HPC and Slurm Interact Components, Tools, Compilers, Libraries, and Debuggers Mpirun command LVS Login Routing Using the SystemUsing the Secure Shell to Log Logging In to the System Introduction Getting Information About QueuesGetting Information About Resources Performing Other Common User Tasks $ man sinfo Getting System Help and Information Overview of Modules Configuring Your Environment with Modulefiles Supplied Modulefiles Viewing Available Modulefiles Modulefiles Automatically Loaded on the SystemViewing Loaded Modulefiles Loading a Modulefile Unloading a Modulefile Automatically Loading a Modulefile at LoginModulefile Conflicts Loading a Modulefile for the Current Session Viewing Modulefile-Specific Help Creating a Modulefile$ module load modules $ man modulefile $ module help totalviewPage Developing Applications Application Development Environment OverviewCompilers Interrupting a Job Examining Nodes and Partitions Before Running JobsMPI Compiler Partition Avail Timelimit Nodes State Nodelist Developing Serial Applications Setting Debugging OptionsSerial Application Build Environment Building Serial Applications Parallel Application Build Environment Developing Parallel ApplicationsModulefiles OpenMP Quadrics Shmem PthreadsMPI Library Intel Fortran and C/C++Compilers Building Parallel Applications Developing Libraries Designing Libraries for the CP4000 PlatformExamples of Compiling and Linking HP-MPI Applications Linkcommand 32-bit -L/opt/mypackage/lib/i686 -lmystuff Linkcommand 64-bit -L/opt/mypackage/lib/x8664 -lmystuffTo build a 64-bit application, you might enter Submitting Jobs Overview of Job SubmissionExtSLURMslurm-arguments Submitting a Serial Job Using LSF-HPC Submitting a Serial Job Using Standard LSFSubmitting a Serial Job with the LSF bsub Command $ bsub hostname Submitting a Serial Job Through Slurm only $ bsub -n4 -I srun hostname Submitting a Non-MPI Parallel JobBsub -nnum-procsbsub-optionsmpijob Mpirun mpirun--options-srunsrun-optionsmpi-jobname Submitting a Batch Job or Job Script $ bsub -n4 -I mpirun -srun ./helloworldBsub -nnum-procs bsub-optionsscript-name Srun hostname mpirun -srun hellompi $ cat myscript.sh #!/bin/sh$ bsub -I -n4 Myscript.sh $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh Running Preexecution Programs $ bsub -n4 -I ./myscript.sh$ cat ./envscript.sh #!/bin/sh name=`hostname` Opt/hptc/bin/srun Mypreexec Debugging Serial Applications Debugging ApplicationsDebugging Parallel Applications TotalView Using TotalView with Slurm Setting Up TotalViewSSH and TotalView Module load mpimodule load totalview Using TotalView with LSF-HPC Setting TotalView PreferencesDebugging an Application Debugging Running Applications Sourcefile initfdte.f was not found, using assembler modeDirectories in File ⇒ Search Path $ mpirun -srun -n2 Psimple $ scancel --user username Exiting TotalView$ squeue Page Tuning Applications Using the Intel Trace Collector and Intel Trace AnalyzerBuilding a Program Intel Trace Collector and HP-MPI Visualizing Data Intel Trace Analyzer and HP-MPI Running a Program Intel Trace Collector and HP-MPILibs Cldflags Using the Intel Trace Collector and Intel Trace Analyzer Page Launching Jobs with the srun Command Using SlurmSrun Squeue Scancel Sinfo Scontrol Introduction to Slurm Using the srun Command with HP-MPI Monitoring Jobs with the squeue CommandUsing the srun Command with LSF-HPC Srun Roles and Modes Terminating Jobs with the scancel Command Getting System Information with the sinfo CommandJob Accounting Fault Tolerance Security# chmod a+r /hptccluster/slurm/job/jobacct.log Using LSF Using Standard LSF on an HP XC SystemUsing LSF-HPC Overview of LSF-HPC Introduction to LSF-HPC in the HP XC Environment Hostname Differences Between LSF-HPC and Standard LSFResources Hostname Status JL/U MAX Njobs RUN Ssusp Ususp RSV Job Terminology $ ssh n15 lshostsUnknown Unknown SLURMnodelist =nodelist if specified HP XCCompute Node Resource Support $ bsub -n 10 -ext SLURMnodes=10 -I srun hostname $ bsub -n 10 -I srun hostname$ bsub -n 10 -ext SLURMnodes=10exclude=n16 -I srun hostname $ bsub -n 10 -ext SLURMconstraint=dualcore -I srun hostname $ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript How LSF-HPC and Slurm Launch and Manage a Job #!/bin/sh hostname srun hostname Mpirun -srun ./hellompi Job Startup and Job Control Determining Available LSF-HPC System Resources Determining the LSF Execution HostGetting the Status of LSF-HPC Getting Information About LSF Execution Host Node Examining LSF-HPC System Queues Getting Host Load InformationGetting Information About the lsf Partition SLINUX6 Summary of the LSF bsub Command Format $ sinfo -p lsf$ sinfo -p lsf -lNe For information about running scripts LSF-SLURM External SchedulerBsub -n num-procs-ext SLURMslurm-arguments \ Bsub-options srun srun-optionsjobname job-options Starting on lsfhost.localdomain n6 Submitting a Job from a Non-HP XC HostWaiting for dispatch ... Starting on lsfhost.localdomain n1 Type=SLINUX64 Getting Job Allocation Information Getting Information About JobsSlurmid=slurmjobidncpus=slurmnprocsslurmalloc=nodelist $ bjobs -l $ bhist -l Examining the Status of a JobTime stamp $ bjobs $ bhist Viewing the Historical Information for a JobSummary of time in seconds spent Various States Jobid User Jobname Pend Psusp RUN Ususp Ssusp Unkwn Total Translating Slurm and LSF-HPC JOBIDs $ bsub -I -n4 -ext SLURMnodes=4 /bin/bash Working Interactively Within an LSF-HPC Allocation$ bjobs -l 124 grep slurm $ srun --jobid=150 hostname Alternatively, you can use the following $ unset Slurmjobid$ export SLURMJOBID=150 $ export SLURMNPROCS=4 $ unset Slurmjobid $ unset Slurmnprocs LSF-HPC Equivalents of Slurm srun Options Job 125 is submitted to the default queue normal$ srun --jobid=250 uptime $ bsub -n4 -ext SLURMnodes=4 -o %J.out sleep Bsub -iinputfile Mpi=mpitype Quit-on-interrupt Page Enabling Remote Execution with OpenSSH Advanced TopicsRunning an X Terminal Session from a Remote Node Determining IP Address of Your Local Machine Running an X terminal Session Using Slurm Running an X terminal Session Using LSF-HPCLogging in to HP XC System $ bsub -n4 -Ip srun -n1 xterm -display Using the GNU Parallel Make Capability$ srun -n4 hostname n46 $ srun -n2 hostname n46 $ cd subdir srun -n1 -N1 $MAKE -j4 $ make PREFIX=’srun -n1 -N1 MAKEJ=-j4 Example Procedure Performance Considerations Local Disks on Compute NodesModified Makefile is invoked as follows $ make PREFIX=srun -n1 -N1 MAKEJ=-j4 Shared File View Communication Between NodesPrivate File View Fp = fopen myfile, a+Page Building and Running a Serial Application Appendix a ExamplesLaunching a Serial Interactive Shell Through LSF-HPC Examine the LSF execution host information Running LSF-HPC Jobs with a Slurm Allocation Request Example 2. Four cores on Two Specific Nodes Launching a Parallel Interactive Shell Through LSF-HPCR15s r1m r15m It tmp swp mem LoadSched LoadStop SLURMnodes=2 Examine the the running jobs information $ hostname n16 $ srun hostname n5 $ bjobs124 Lsfad Examine the the finished jobs information Submitting a Simple Job Script with LSF-HPCShow the environment Display the script Submitting an Interactive Job with LSF-HPC Run some commands from the pseudo-terminalSubmit the job Show the job allocation Exit the pseudo-terminal Submitting an HP-MPI Job with LSF-HPCView the interactive jobs View the finished jobs View the running job View the finished jobLsfhost.localdomai Using a Resource Requirements String in an LSF-HPC Command States by date and time$ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob 108 Glossary First-come See Fcfs First-served Global storage To the queueAs local storage Are not appropriate for replication Login requests and directs them to a node with a login role Single commandLinux Virtual See LVS Server Load file LSF master host Remotely. PXE booting is configured at the Bios level Network See NIS Information ServicesNotably to install and remove software packages Slurm backup Power available per unit of space SshSymmetric See SMP Multiprocessing 114 Index Index PGI Utilities, 63 Slurm commands