Control the parallel job | HP XC System 2.x Software guide

Table 7-2: LSF Equivalents of SLURM srun Options (cont.)

srun Option	Description	LSF Equivalent
-r, --relative=n	Run a job step relative to node n of	Use when launching parallel tasks.
	the current allocation. It is about
	placing tasks within allocation.

-D, --chdir=path

Specify working directory of job.

Job will be started in job submission directory by default.

-k , –no-kill	Do not automatically terminate a
	job if one of the nodes it has been
	allocated fails.
-T, --threads=nthreads	srun uses nthreads to initiate and
	control the parallel job.

When creating allocation, LSF does not use the –k option. SLURM always terminates a job if one of allocated nodes fails.

Use when launching parallel tasks.

-l , --label	Prepend task number to lines of
	stdout/stderr.
-u , --unbuffered	Do not line buffer stdout from
	remote tasks.

Use when launching parallel tasks.

Use when launching parallel tasks.

-m, --distribu-	Distribution method for remote
tion=blockcyclic	processes.
--mpi=mpi_type	Identify the type of MPI to be used.
--jobid=id	Initiate a job step under an already
	allocated job id.
-v	Verbose operation.
-d, --slurm-debug=level	A debug level for slurmd.
-W, --wait=seconds	How long to wait after the first task
	terminates before terminating all
	remaining tasks.

Use when launching parallel tasks.

Use when launching parallel tasks.

Do not use it. SLURM_JOBID is set in job environment to associate job with slurm allocation.

Use when launching parallel tasks.

Use when launching parallel tasks.

Use when launching parallel tasks.

-q, --quit-on-interrupt	Quit immediately on single SIGINT.	Meaningless under LSF.
-Q, --quiet	Suppress informational message.	Use it when launching parallel tasks.
--core=type	Adjust corefile format for parallel	Use it when launching parallel tasks.
	job.
-a, --attach=id	Attach srun to a running job.	Meaningless under LSF.
-j, --join	Join with running job.	Meaningless under LSF.
-s, --steal	Steal connection to running job.	Meaningless under LSF.

Using LSF 7-25

Image 109

HP XC System 2.x Software manual Control the parallel job

Contents

Hewlett-Packard Company Palo Alto, California June Product VersionPage Contents 5.4 Developing Applications Using Slurm Tuning ApplicationsDebugging Applications Using HP-MPI Using LSF 6.4 Using HP Mlib Examples Advanced TopicsGlossary Index Examples Tables FiguresPage Intended Audience About This DocumentDocument Organization HP XC Information Linux Administration HandbookQuickSpecs for HP XC System Software HP XC Program Development Environment Supplementary Information For More InformationHP Message Passing Interface HP Mathematical Library Http//supermon.sourceforge.net ManpagesHttp//systemimager.org Http//sourceforge.net/projects/modules Http//linuxvirtualserver.org Related InformationHttp//www-unix.mcs.anl.gov/mpi Typographical Conventions # cd /opt/hptc/config/sbinBold text Ctrl/x HP Encourages Your CommentsPage System Architecture Overview of the User EnvironmentOperating System Node Specialization File System Storage and I/OSAN Storage Local Storage File System Layout System Interconnect Network User Environment CommandsNetwork Address Translation NAT 1 LVS Parallel Applications Application Development EnvironmentSerial Applications Slurm Run-Time EnvironmentLoad Sharing Facility LSF-HPC How LSF-HPC and Slurm Interact HP-MPI Components, Tools, Compilers, Libraries, and Debuggers 8Overview of the User Environment Configuring Your Environment with Modulefiles Using the SystemLVS Login Routing Using ssh to Log 2Using the System Supplied Modulefiles Supplied ModulefilesModulefile Sets the HP XC User Environment Viewing Available Modulefiles Modulefiles Automatically Loaded on the SystemViewing Loaded Modulefiles Loading a Modulefile Unloading a Modulefile Automatically Loading a Modulefile at LoginModulefile Conflicts Loading a Modulefile for the Current Session Viewing Modulefile-Specific Help Creating a Modulefile$ module load modules $ man modulefile $ module help totalview Introduction Launching and Managing Jobs Quick StartGetting Information About Queues Getting Information About Resources Launching Jobs Getting Information About the System’s PartitionsSubmitting a Serial Job Example 2-1 Submitting a Serial Job Submitting a Non-MPI Parallel Job Using Slurm Options with the LSF External SchedulerExample 2-2 Submitting a Non-MPI Parallel Job $ bsub -n4 -I srun hostname Example 2-4 Running an MPI Job with LSF Submitting an MPI Job$ bsub -n4 Mpirun -srun ./helloworld Example 2-6 Submitting a Job Script Submitting a Batch Job or Job Script$ bsub -I -n4 myjobscript.sh Getting System Help and Information Performing Other Common User Tasks $ man -k keyword $ man sinfoPage Overview Developing Applications Standard Linux Compilers Using CompilersIntel Compilers PGI Compilers Checking Nodes and Partitions Before Running Jobs Setting Debugging OptionsInterrupting a Job Developing Serial Applications Developing Parallel Applications Using Mlib in Serial ApplicationsSerial Application Build Environment Building Serial Applications Modulefiles Parallel Application Build EnvironmentHP-MPI OpenMP Mlib Math Library Quadrics ShmemMPI Library $ mpicc object1.o ... -pthread -o myapp.exe PGI Fortran and C/C++ Compilers Intel Fortran and C/C++CompilersGNU C and C++ Compilers GNU Parallel Make Building Parallel Applications Reserved Symbols and NamesCompiling and Linking Non-MPI Applications Compiling and Linking HP-MPI Applications Designing Libraries for XC4000 Developing Libraries$ mpicc -c -g foo.c Advanced Topics Using the GNU Parallel Make CapabilityExample 3-1 Directory Structure Example 3-2 Recommended Directory Structure $ cd subdir srun -n1 -N1 $MAKE -j4 $ cd subdir srun $MAKE Example Procedure $ make PREFIX=’srun -n1 -N1’ MAKEJ=’-j4’ $ make PREFIX=’srun -n1 -N1 MAKEJ=’-j4’ 3 I/O Performance Considerations Local Disks on Compute NodesShared File View Private File View Communication Between Nodes Page Debugging Serial Applications Debugging ApplicationsDebugging Parallel Applications TotalView Debugging with TotalView Setting Up TotalViewSSH and TotalView Using TotalView with LSF-HPC Using TotalView with Slurm$ srun -Nx-A $ mpirun -tv -srun application $ bsub -nx-ext SLURMnodes=x \ -Is /usr/bin/xterm $ totalview Starting TotalView for the First Time TotalView Preferences Window Preferences window, click on the Launch Strings tab Debugging Applications $ mpicc -g -o Psimple simple.c -lm Debugging an Application$ mpirun -tv -srun -n2 ./Psimple TotalView Process Window Example $ mpirun -srun -n2 Psimple Debugging Running Applications Exiting TotalView $ scancel --user username$ squeue Page Using the Intel Trace Collector/Analyzer Tuning ApplicationsBuilding a Program Intel Trace Collector and HP-MPI Example Visualizing Data Intel Trace Analyzer and HP-MPI Running a Program Intel Trace Collector and HP-MPIExample Running the vtjacobic Example Program Slurm Commands Using SlurmCommand Function Introduction Launching Jobs with the srun Command Accessing the Slurm ManpagesSrun Roles and Modes Example 6-1 Simple Launch of a Serial Program Srun Modes Srun Roles Srun Run-Mode Options Srun Signal HandlingBatch Allocate Srun Resource-Allocation Options Part --partition=part Cpt --cpus-per-task=cptMinutes --time=minutes Nthreads --threads=nthreads Srun Control Options Srun I/O Options Mode --input=mode Mode --output=modeMode --error=mode Label Clist --constraint=clist Srun Constraint Options Mem=size Contiguous=yesnoMincpus=n Vmem=size Using srun with HP-MPI Monitoring Jobs with the squeue CommandUsing srun with LSF Srun Environment Variables Getting System Information with the sinfo Command Killing Jobs with the scancel Command Job Accounting Fault ToleranceSecurity $ sinfo -R Introduction to LSF in the HP XC Environment Using LSFOverview of LSF Nodelist= list-of-nodes Exclude= list-of-nodes Topology Support $ bsub -n 10 -ext SLURMnodes=10exclude=n16 srun myapp $ bsub -n 10 -ext SLURMnodes=10 srun myapp $ bqueues -l normal grep Jobstarter How LSF and Slurm Launch and Manage a Job$ bsub -Is hostname $ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript How LSF-HPC and Slurm Launch and Manage a Job Differences Between LSF on HP XC and Standard LSF Determining Available System Resources Determining Execution HostGetting Status of LSF Job Startup and Job Control Getting Host Load Information Getting Information About LSF-HPC Execution Host Node Checking LSF System Queues Submitting JobsGetting Information About the lsf Partition $ sinfo -p lsf Bsub bsuboptions jobname joboptions Summary of the LSF bsub Command Format Slurm Arguments Function LSF-SLURM External Scheduler Starting on lsfhost.localdomain n6 Submitting a Job in Parallel Submitting a Serial JobSubmitting an HP-MPI Job Example 7-5 Submitting an Interactive Serial Job Example 7-6 Submitting an HP-MPI Job Submitting a Batch Job or Job Script$ bsub -n4 -I mpirun -srun ./helloworld Example 7-8 Submitting a Batch Job Script Examples$ bsub -n4 -I ./myscript.sh $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh $ bsub -n4 -I ./myscript.sh -n8 -O Submitting a Job from a Non-HP XC HostType=SLINUX64 Getting Job Allocation Information Getting Information About JobsJob Allocation Information for a Running Job $ bsub -R type=SLINUX64 -n4 -I srun hostname Checking Status of a Job Example 7-13 Using the bjobs Command Short OutputJob Allocation Information for a Finished Job Example 7-15 Using the bhist Command Short Output Example 7-14 Using the bjobs Command Long OutputOutput Provided by the bhist Command Viewing a Job’s Historical Information Working Interactively Within an LSF-HPC Allocation Example 7-16 Using the bhist Command Long OutputSubmitting an Interactive Job to Launch the xterm Program Example 7-18 View Your Allocation in Slurm Example 7-17 View Your EnvironmentExample 7-19 View Your Running Job in LSF Example 7-20 View Job Details in LSF Example 7-21 Running Jobs from an xterm Window Submitting an Interactive Job to Launch a ShellExample 7-22 Submitting an Interactive Shell Program $ hostname $ srun hostname $ srun -n2 hostname LSF Equivalents of Slurm srun Options LSF Equivalents of Slurm srun Options$ srun hostname n1 $ exit Srun Option Description LSF Equivalent 24Using LSF Control the parallel job Page Using HP-MPI HP-MPI Directory Structure Setting Environment VariablesCompiling and Running Applications Building and Running an Example Application Building and Running helloworld Example Application helloworld$ mpicc -o helloworld $MPIROOT/help/helloworld.c $ $MPIROOT/bin/mpirun -srun -n4 helloworld HP-MPI options allowed with -srun Launching MPI Jobs System Interconnect Selection Creating Subshells and Launching Jobsteps$ mpirun -srun -n6 -O -N2 -m cyclic ./a.out host1 rank1 $ mpirun -srun -n4 -N2 -O -m cyclic ./a.out host1 rank1 $ mpirun -subnet 192.168.1.1 -prot -srun -n4 ./a.out Using LSF and HP-MPI System Interconnect Support $ /usr/sbin/ifconfig -aMPI Versioning Example 8-5 Allocating 12 Processors on 6 Nodes Truncated Messages 32-Bit Builds on XC4000Allowing Windows to Use Exclusive Locks $ mpirun -TCP -srun -N8 ./a.out Mpirun Command Options $MPIROOT/bin/mpirun -v -prot -np 2 /path/to/program.x Environment Variables Export MPIPINPERCENTAGE=30 Export MPIPHYSICALMEMORY=1048576Export MPIPAGEALIGNMEM=1 Export MPIMAXWINDOW=10 Export MPIUSELIBELAN=0 Mpich Object CompatibilityExport MPIUSELIBELANUSE=5 $MPIROOT/bin/mpirun.mpich -np 2 ./prog.x At http//docs.hp.com HP-MPI Documentation and ManpagesHP-MPI Manpage Categories Category Manpages Description Additional Information, Known Problems, and Work-arounds Intel Compiler Notes Using HP MlibVersion 8 Fortran Compiler Version 7 Fortran Compiler Mlib and Module Files HP Mlib for the HP XC6000 PlatformPlatform Support Library Support Modulefiles and Mlib Using Intel Compilers with HP MlibCompiling and Linking Licensing HP Mlib for the HP XC4000 PlatformMlib Manpages Linking SuperLUDIST $ pgcc options file 6Using HP Mlib Enabling Remote Execution with OpenSSH Advanced TopicsRunning an X Terminal Session from a Remote Node Determining IP Address of Your Local Machine Logging in to HP XC System Running an X terminal Session Using Slurm$ hostname $ host mymachine $ bsub -n4 -Ip srun -n1 xterm -display Running an X terminal Session Using LSF$ bjobs Page Building and Running a Serial Application ExamplesLaunching a Serial Interactive Shell Through LSF Example 1. Two Processors on Any Two Nodes Running LSF Jobs with a Slurm Allocation Request Example 2. Four Processors on Two Specific Nodes Launching a Parallel Interactive Shell Through LSFView the job $ bsub -Is -n4 -ext SLURMnodes=4 /bin/bash Check the running job’s information Check the finished job’s information Submitting a Simple Job Script with LSFShow the environment Display the script Show the job allocation Submitting an Interactive Job with LSFShow the Slurm job ID $ bsub -n8 -Ip /bin/sh Submitting an HP-MPI Job with LSF View the finished job View the running job$ bjobs -l $ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob Using a Resource Requirements String in an LSF CommandPage Glossary External network node Extensible firmware interfaceFairshare First come first served Integrated Lights Out Image serverInterconnect Internet address LSF master host Network Information ServicesManagement Processor Master host Parallel application Root Administration SwitchResource manager role Role Glossary-6 Symmetric multiprocessing Index-1 Index Index-2 Gdb, 4-1 GNU Index-3 Modulefile Index-4 Resource manager, 7-1role Serial applications