Using LSF, Using HP-MPI | HP XC System 2.x Software instruction

6.4.6.1	I/O Commands	6-8
6.4.6.2	I/O Redirection Alternatives	6-9
6.4.7	srun Constraint Options	6-10
6.4.8	srun Environment Variables	6-12
6.4.9	Using srun with HP-MPI	6-12
6.4.10	Using srun with LSF	6-12
6.5	Monitoring Jobs with the squeue Command	6-12
6.6	Killing Jobs with the scancel Command	6-13
6.7	Getting System Information with the sinfo Command	6-13
6.8	Job Accounting	6-14
6.9	Fault Tolerance	6-14
6.10	Security	6-14

7 Using LSF

7.1	Introduction to LSF in the HP XC Environment	7-1
7.1.1	Overview of LSF	7-1
7.1.2	Topology Support	7-2
7.1.3	Notes on LSF-HPC	7-3
7.1.4	How LSF and SLURM Launch and Manage a Job	7-4
7.1.5	Differences Between LSF on HP XC and Standard LSF	7-6
7.1.6	Notes About Using LSF in the HP XC Environment	7-7
7.1.6.1	Job Startup and Job Control	7-7
7.1.6.2	Preemption Support	7-7
7.2	Determining Execution Host	7-7
7.3	Determining Available System Resources	7-7
7.3.1	Getting Status of LSF	7-7
7.3.2	Getting Information About LSF-HPC Execution Host Node	7-8
7.3.3	Getting Host Load Information	7-8
7.3.4	Checking LSF System Queues	7-9
7.3.5	Getting Information About the lsf Partition	7-9
7.4	Submitting Jobs	7-9
7.4.1	Summary of the LSF bsub Command Format	7-10
7.4.2	LSF-SLURM External Scheduler	7-11
7.4.3	Submitting a Serial Job	7-13
7.4.4	Submitting a Job in Parallel	7-13
7.4.5	Submitting an HP-MPI Job	7-13
7.4.6	Submitting a Batch Job or Job Script	7-14
7.4.6.1	Examples	7-15
7.4.7	Submitting a Job from a Non-HP XC Host	7-16
7.5	Getting Information About Jobs	7-17
7.5.1	Getting Job Allocation Information	7-17
7.5.1.1	Job Allocation Information for a Running Job	7-17
7.5.1.2	Job Allocation Information for a Finished Job	7-18
7.5.2	Checking Status of a Job	7-18
7.5.3	Viewing a Job’s Historical Information	7-19
7.6	Working Interactively Within an LSF-HPC Allocation	7-20
7.6.1	Submitting an Interactive Job to Launch the xterm Program	7-20
7.6.2	Submitting an Interactive Job to Launch a Shell	7-22
7.7	LSF Equivalents of SLURM srun Options	7-23

8 Using HP-MPI

8.1

Overview

8-1

vi Contents

Image 6

HP XC System 2.x Software manual Using LSF, Using HP-MPI

Contents

June Product Version Hewlett-Packard Company Palo Alto, CaliforniaPage Contents Developing Applications 5.4 Tuning Applications Using SlurmDebugging Applications Using LSF Using HP-MPI Using HP Mlib 6.4 Advanced Topics ExamplesGlossary Index Examples Figures TablesPage About This Document Intended AudienceDocument Organization QuickSpecs for HP XC System Software Linux Administration HandbookHP XC Information HP XC Program Development Environment HP Message Passing Interface For More InformationSupplementary Information HP Mathematical Library Http//systemimager.org ManpagesHttp//supermon.sourceforge.net Http//sourceforge.net/projects/modules Related Information Http//linuxvirtualserver.orgHttp//www-unix.mcs.anl.gov/mpi # cd /opt/hptc/config/sbin Typographical ConventionsBold text HP Encourages Your Comments Ctrl/xPage Operating System Overview of the User EnvironmentSystem Architecture Node Specialization SAN Storage Storage and I/OFile System Local Storage System Interconnect Network File System Layout Network Address Translation NAT CommandsUser Environment 1 LVS Application Development Environment Parallel ApplicationsSerial Applications Load Sharing Facility LSF-HPC Run-Time EnvironmentSlurm How LSF-HPC and Slurm Interact Components, Tools, Compilers, Libraries, and Debuggers HP-MPI 8Overview of the User Environment LVS Login Routing Using the SystemConfiguring Your Environment with Modulefiles Using ssh to Log 2Using the System Supplied Modulefiles Supplied ModulefilesModulefile Sets the HP XC User Environment Viewing Loaded Modulefiles Modulefiles Automatically Loaded on the SystemViewing Available Modulefiles Loading a Modulefile Modulefile Conflicts Automatically Loading a Modulefile at LoginUnloading a Modulefile Loading a Modulefile for the Current Session $ module load modules $ man modulefile Creating a ModulefileViewing Modulefile-Specific Help $ module help totalview Getting Information About Queues Launching and Managing Jobs Quick StartIntroduction Getting Information About Resources Submitting a Serial Job Getting Information About the System’s PartitionsLaunching Jobs Example 2-1 Submitting a Serial Job Example 2-2 Submitting a Non-MPI Parallel Job Using Slurm Options with the LSF External SchedulerSubmitting a Non-MPI Parallel Job $ bsub -n4 -I srun hostname Submitting an MPI Job Example 2-4 Running an MPI Job with LSF$ bsub -n4 Mpirun -srun ./helloworld Submitting a Batch Job or Job Script Example 2-6 Submitting a Job Script$ bsub -I -n4 myjobscript.sh Performing Other Common User Tasks Getting System Help and Information $ man sinfo $ man -k keywordPage Developing Applications Overview Intel Compilers Using CompilersStandard Linux Compilers PGI Compilers Interrupting a Job Setting Debugging OptionsChecking Nodes and Partitions Before Running Jobs Developing Serial Applications Serial Application Build Environment Using Mlib in Serial ApplicationsDeveloping Parallel Applications Building Serial Applications HP-MPI Parallel Application Build EnvironmentModulefiles OpenMP MPI Library Quadrics ShmemMlib Math Library $ mpicc object1.o ... -pthread -o myapp.exe GNU C and C++ Compilers Intel Fortran and C/C++CompilersPGI Fortran and C/C++ Compilers GNU Parallel Make Compiling and Linking Non-MPI Applications Reserved Symbols and NamesBuilding Parallel Applications Compiling and Linking HP-MPI Applications Developing Libraries Designing Libraries for XC4000$ mpicc -c -g foo.c Example 3-1 Directory Structure Using the GNU Parallel Make CapabilityAdvanced Topics Example 3-2 Recommended Directory Structure $ cd subdir srun $MAKE $ cd subdir srun -n1 -N1 $MAKE -j4 Example Procedure $ make PREFIX=’srun -n1 -N1 MAKEJ=’-j4’ $ make PREFIX=’srun -n1 -N1’ MAKEJ=’-j4’ Shared File View Local Disks on Compute Nodes3 I/O Performance Considerations Private File View Communication Between Nodes Page Debugging Parallel Applications Debugging ApplicationsDebugging Serial Applications TotalView Setting Up TotalView Debugging with TotalViewSSH and TotalView $ srun -Nx-A $ mpirun -tv -srun application Using TotalView with SlurmUsing TotalView with LSF-HPC $ bsub -nx-ext SLURMnodes=x \ -Is /usr/bin/xterm Starting TotalView for the First Time $ totalview TotalView Preferences Window Preferences window, click on the Launch Strings tab Debugging Applications Debugging an Application $ mpicc -g -o Psimple simple.c -lm$ mpirun -tv -srun -n2 ./Psimple TotalView Process Window Example Debugging Running Applications $ mpirun -srun -n2 Psimple $ scancel --user username Exiting TotalView$ squeue Page Building a Program Intel Trace Collector and HP-MPI Tuning ApplicationsUsing the Intel Trace Collector/Analyzer Example Running a Program Intel Trace Collector and HP-MPI Visualizing Data Intel Trace Analyzer and HP-MPIExample Running the vtjacobic Example Program Command Function Using SlurmSlurm Commands Introduction Srun Roles and Modes Accessing the Slurm ManpagesLaunching Jobs with the srun Command Example 6-1 Simple Launch of a Serial Program Srun Roles Srun Modes Batch Srun Signal HandlingSrun Run-Mode Options Allocate Srun Resource-Allocation Options Minutes --time=minutes Cpt --cpus-per-task=cptPart --partition=part Nthreads --threads=nthreads Srun Control Options Srun I/O Options Mode --error=mode Mode --output=modeMode --input=mode Label Srun Constraint Options Clist --constraint=clist Mincpus=n Contiguous=yesnoMem=size Vmem=size Using srun with LSF Monitoring Jobs with the squeue CommandUsing srun with HP-MPI Srun Environment Variables Killing Jobs with the scancel Command Getting System Information with the sinfo Command Security Fault ToleranceJob Accounting $ sinfo -R Using LSF Introduction to LSF in the HP XC EnvironmentOverview of LSF Topology Support Nodelist= list-of-nodes Exclude= list-of-nodes $ bsub -n 10 -ext SLURMnodes=10 srun myapp $ bsub -n 10 -ext SLURMnodes=10exclude=n16 srun myapp How LSF and Slurm Launch and Manage a Job $ bqueues -l normal grep Jobstarter$ bsub -Is hostname How LSF-HPC and Slurm Launch and Manage a Job $ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript Differences Between LSF on HP XC and Standard LSF Getting Status of LSF Determining Execution HostDetermining Available System Resources Job Startup and Job Control Getting Information About LSF-HPC Execution Host Node Getting Host Load Information Getting Information About the lsf Partition Submitting JobsChecking LSF System Queues $ sinfo -p lsf Summary of the LSF bsub Command Format Bsub bsuboptions jobname joboptions LSF-SLURM External Scheduler Slurm Arguments Function Starting on lsfhost.localdomain n6 Submitting an HP-MPI Job Submitting a Serial JobSubmitting a Job in Parallel Example 7-5 Submitting an Interactive Serial Job Submitting a Batch Job or Job Script Example 7-6 Submitting an HP-MPI Job$ bsub -n4 -I mpirun -srun ./helloworld $ bsub -n4 -I ./myscript.sh ExamplesExample 7-8 Submitting a Batch Job Script $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh Submitting a Job from a Non-HP XC Host $ bsub -n4 -I ./myscript.sh -n8 -OType=SLINUX64 Job Allocation Information for a Running Job Getting Information About JobsGetting Job Allocation Information $ bsub -R type=SLINUX64 -n4 -I srun hostname Example 7-13 Using the bjobs Command Short Output Checking Status of a JobJob Allocation Information for a Finished Job Output Provided by the bhist Command Example 7-14 Using the bjobs Command Long OutputExample 7-15 Using the bhist Command Short Output Viewing a Job’s Historical Information Example 7-16 Using the bhist Command Long Output Working Interactively Within an LSF-HPC AllocationSubmitting an Interactive Job to Launch the xterm Program Example 7-19 View Your Running Job in LSF Example 7-17 View Your EnvironmentExample 7-18 View Your Allocation in Slurm Example 7-20 View Job Details in LSF Example 7-22 Submitting an Interactive Shell Program Submitting an Interactive Job to Launch a ShellExample 7-21 Running Jobs from an xterm Window $ hostname $ srun hostname $ srun -n2 hostname $ srun hostname n1 $ exit LSF Equivalents of Slurm srun OptionsLSF Equivalents of Slurm srun Options Srun Option Description LSF Equivalent 24Using LSF Control the parallel job Page Using HP-MPI Compiling and Running Applications Setting Environment VariablesHP-MPI Directory Structure Building and Running an Example Application $ mpicc -o helloworld $MPIROOT/help/helloworld.c Example Application helloworldBuilding and Running helloworld $ $MPIROOT/bin/mpirun -srun -n4 helloworld Launching MPI Jobs HP-MPI options allowed with -srun $ mpirun -srun -n6 -O -N2 -m cyclic ./a.out host1 rank1 Creating Subshells and Launching JobstepsSystem Interconnect Selection $ mpirun -srun -n4 -N2 -O -m cyclic ./a.out host1 rank1 Using LSF and HP-MPI $ mpirun -subnet 192.168.1.1 -prot -srun -n4 ./a.out MPI Versioning $ /usr/sbin/ifconfig -aSystem Interconnect Support Example 8-5 Allocating 12 Processors on 6 Nodes 32-Bit Builds on XC4000 Truncated MessagesAllowing Windows to Use Exclusive Locks Mpirun Command Options $ mpirun -TCP -srun -N8 ./a.out Environment Variables $MPIROOT/bin/mpirun -v -prot -np 2 /path/to/program.x Export MPIPAGEALIGNMEM=1 Export MPIPHYSICALMEMORY=1048576Export MPIPINPERCENTAGE=30 Export MPIMAXWINDOW=10 Export MPIUSELIBELANUSE=5 Mpich Object CompatibilityExport MPIUSELIBELAN=0 $MPIROOT/bin/mpirun.mpich -np 2 ./prog.x HP-MPI Manpage Categories HP-MPI Documentation and ManpagesAt http//docs.hp.com Category Manpages Description Additional Information, Known Problems, and Work-arounds Version 8 Fortran Compiler Using HP MlibIntel Compiler Notes Version 7 Fortran Compiler Platform Support HP Mlib for the HP XC6000 PlatformMlib and Module Files Library Support Using Intel Compilers with HP Mlib Modulefiles and MlibCompiling and Linking Mlib Manpages HP Mlib for the HP XC4000 PlatformLicensing Linking SuperLUDIST $ pgcc options file 6Using HP Mlib Running an X Terminal Session from a Remote Node Advanced TopicsEnabling Remote Execution with OpenSSH Determining IP Address of Your Local Machine $ hostname Running an X terminal Session Using SlurmLogging in to HP XC System $ host mymachine Running an X terminal Session Using LSF $ bsub -n4 -Ip srun -n1 xterm -display$ bjobs Page Examples Building and Running a Serial ApplicationLaunching a Serial Interactive Shell Through LSF Running LSF Jobs with a Slurm Allocation Request Example 1. Two Processors on Any Two Nodes Launching a Parallel Interactive Shell Through LSF Example 2. Four Processors on Two Specific NodesView the job Check the running job’s information $ bsub -Is -n4 -ext SLURMnodes=4 /bin/bash Show the environment Submitting a Simple Job Script with LSFCheck the finished job’s information Display the script Show the Slurm job ID Submitting an Interactive Job with LSFShow the job allocation $ bsub -n8 -Ip /bin/sh Submitting an HP-MPI Job with LSF View the running job View the finished job$ bjobs -l Using a Resource Requirements String in an LSF Command $ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjobPage Glossary Fairshare Extensible firmware interfaceExternal network node First come first served Interconnect Image serverIntegrated Lights Out Internet address Management Processor Network Information ServicesLSF master host Master host Resource manager role Root Administration SwitchParallel application Role Symmetric multiprocessing Glossary-6 Index Index-1 Gdb, 4-1 GNU Index-2 Modulefile Index-3 Resource manager, 7-1role Serial applications Index-4