125, 131 | HP XC System 4.x Software specification

A.4 Launching a Parallel Interactive Shell Through LSF	117
A.5 Submitting a Simple Job Script with LSF	119
A.6 Submitting an Interactive Job with LSF	120
A.7 Submitting an HP-MPI Job with LSF	122
A.8 Using a Resource Requirements String in an LSF Command	123
Glossary	125
Index	131

Table of Contents

Page 7

Image 7

HP XC System 4.x Software manual 125, 131

Contents

HP XC System Software XC User Guide Page Table of Contents Compiling and Linking Serial Applications Configuring Your Environment with Modulefiles Srun Roles Srun Modes Debugging Applications 115 107 131 125 10-1 List of Figures11-1 List of Tables List of Examples Typographic Conventions About This DocumentIntended Audience New and Changed Information in This Edition HP XC and Related HP Products Information HP XC System Software Users Guide Related Information Https//computing.llnl.gov/linux/slurm/documentation.html Http//systemimager.org Http//www-unix.mcs.anl.gov/mpi Manpages $ man discover $ man 8 discover HP Encourages Your Comments$ man -k keyword Page Page /proc/cpuinfo file is dynamic Only to the administrator of the HP XC systemNode Specialization Determining the Node Platform Local Storage Storage and I/OFile System SAN Storage File System Layout System Interconnect NetworkHP XC System Interconnects 1 LVS Determining System Configuration InformationUser Environment Network Address Translation NAT Application Development Environment CommandsParallel Applications Load Sharing Facility LSF Run-Time EnvironmentSerial Applications Slurm Nodes for the job Requested by the HP-MPI mpirun commandHow LSF and Slurm Interact HP-MPI Components, Tools, Compilers, Libraries, and Debuggers Page Logging In to the System Using the SystemLVS Login Routing Using the Secure Shell to Log $ bqueues IntroductionGetting Information About Queues Getting Information About Resources Performing Other Common User Tasks $ man sinfo Getting System Help and InformationDetermining the LSF Cluster Name and the LSF Execution Host $ lsid Overview of Modules Configuring Your Environment with Modulefiles Supplied Modulefiles Supplied Modulefiles HP-MPI Loading a Modulefile Modulefiles Automatically Loaded on the SystemViewing Available Modulefiles Viewing Loaded Modulefiles Viewing Modulefile-Specific Help Unloading a ModulefileModulefile Conflicts $ module load modules $ man modulefile Creating a Modulefile Application Development Environment Overview Developing Applications Compilers Compiler CommandsMPI Compiler Developing Serial Applications Setting Debugging OptionsExamining Nodes and Partitions Before Running Jobs Interrupting a Job Parallel Application Build Environment Developing Parallel ApplicationsSerial Application Build Environment Building Serial Applications Pthreads ModulefilesHP-MPI OpenMP GNU C and C++ Compilers MPI LibraryIntel Fortran and C/C++Compilers PGI Fortran and C/C++ Compilers Building Parallel Applications Developing Libraries To compile and link a C application using the mpicc commandDesigning Libraries for the CP4000 Platform Linkcommand 64-bit -L/opt/mypackage/lib/x8664 -lmystuff Linkcommand 32-bit -L/opt/mypackage/lib/i686 -lmystuffTo build a 64-bit application, you might enter Developing Applications Overview of Job Submission Submitting a Serial Job Using LSFSubmitting a Serial Job with the LSF bsub Command Submitting Jobs $ bsub -I srun hostname Example 5-2 Submitting a Serial Job Using LSFSubmitting a Serial Job Through Slurm Only Example 5-1 Submitting a Job from the Standard Input To submit a parallel job Following is the command line used to compile this programSubmitting a Parallel Job Submitting a Non-MPI Parallel Job $ bsub -n4 -I srun hostname Example 5-5 Submitting a Non-MPI Parallel Job $ bsub -n4 -I mpirun -srun ./helloworld Example 5-7 Submitting an MPI Job Nodelist=list of nodes Arguments for the Slurm External Scheduler $ bsub -n 10 -ext SLURMnodelist=n1-10 srun hostname $ bqueues -l dualcore grep Slurm Submitting a Batch Job or Job ScriptExample 5-14 Submitting a Job Script $ bsub -n 10 -ext SLURMconstraint=dualcore -I srun hostname $ bsub -n4 -I ./myscript.sh $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh Submitting Multiple MPI Jobs Across the Same Set of Nodes Using a Script to Submit Multiple JobsUsing a Makefile to Submit Multiple Jobs Use the squeue command to acquire information on the jobs $ cat mymake Following command line makes the program and executes it$ tail 113.out $ cat 117.out $ bsub -R type=SLINUX64 -n4 -I srun hostname Submitting a Job from a Host Other Than an HP XC HostRunning Preexecution Programs Type=SLINUX64Page TotalView Debugging ApplicationsDebugging Serial Applications Debugging Parallel Applications Module load mpi module load totalview Setting Up TotalViewDebugging with TotalView SSH and TotalView $ srun -Nx-A $ mpirun -tv -srun application Using TotalView with SlurmUsing TotalView with LSF Setting TotalView Preferences $ mpirun -tv -srun -n2 ./Psimple Debugging an Application Run the application $ scancel --user usernameDebugging Running Applications Exiting TotalViewPage Xtools Utilities Monitoring Node Activity Where Running Performance Health Tests $ ovp --verbose --verify=perfhealth/cpuusage You can list the available tests with the ovp -lcommand$ ovp -l $ ovp --verify=perfhealth/cpuusage HOMEDIRECTORY/ovpn16mmddyy.log HOMEDIRECTORY/ovpn16mmddyyr1.log Page Using the Intel Trace Collector and Intel Trace Analyzer Tuning ApplicationsBuilding a Program Intel Trace Collector and HP-MPI Example 8-2 C Example Running the vtjacobic Example Program Running a Program Intel Trace Collector and HP-MPIExample 8-1 The vtjacobic Example Program For more information, see the following Web site Intel Trace Collector and Analyzer with HP-MPI on HP XC Installation KitHP-MPI and the Intel Trace Collector Following is a Fortran example called vtjacobif Running a Program Visualizing Data Intel Trace Analyzer and HP-MPI Running a Program Across Nodes Using LSF# bsub -n4 -I mpirun.mpich -np 2 ./vtjacobic Page Slurm Utilities Using SlurmLaunching Jobs with the srun Command Introduction to Slurm Srun Roles and Modes Monitoring Jobs with the squeue CommandUsing the srun Command with HP-MPI Using the srun Command with LSF Getting System Information with the sinfo Command Terminating Jobs with the scancel CommandExample 9-5cancels all pending jobs $ sinfo -R Fault ToleranceJob Accounting Security Information for LSF Using LSF $ squeue --jobs $SLURMJOBID Overview of LSF Integrated with Slurm Example 10-1 Examples of LSF Job Launch Lsfadmin@n16 ~$ bsub -n4 -I srun hostname Differences Between LSF and LSF Integrated with Slurm Lshosts and lsload commands display for each of these items $ ssh n15 lshosts Job TerminologyConnection back to the terminal from which the job was Submitted. This job may run immediately, or it may run Batch system scheduling policies Batch jobs are submitted with the srun -bcommand. ByDefault, the output is written to Are submitted with the bsub -Icommand Job Startup and Job Control Using LSF Integrated with Slurm in the HP XC EnvironmentUseful Commands Submitting Jobs How LSF and Slurm Launch and Manage a Job LSF-SLURM External SchedulerLSF with Slurm Job Launch Exit Codes How LSF and Slurm Launch and Manage a Job User logs in to login node n16 Determining Available System Resources Determining the LSF Execution Host Examining System Core Status Following example shows the output from the lshosts commandGetting Information About the LSF Execution Host Node Getting Information About the lsf Partition Getting Information About JobsGetting Host Load Information Examining System Queues Than what the job requests Getting Job Allocation InformationThis allocation string has the following values LSF job with Slurm allocated resources Examining the Status of a Job Example 10-5 Using the bjobs Command Short OutputExample 10-4 Job Allocation Information for a Finished Job Viewing the Historical Information for a Job Example 10-6 Using the bjobs Command Long OutputExample 10-7 Using the bhist Command Short Output Output Provided by the bhist Command Use the bjobs command to view the Slurm Jobid Example 10-8 Using the bhist Command Long OutputTranslating Slurm and LSF JOBIDs $ bjobs -l 124 grep slurm Working Interactively Within an Allocation$ sacct -j $ bsub -I -n4 -ext SLURMnodes=4 /bin/bash This example assumes 2 cores per node Example 10-9 Launching an Interactive MPI JobAlternatively, you can use the following $ unset Slurmjobid $ unset Slurmnprocs LSF Equivalents of Slurm srun Options3describes the srun options and lists their LSF equivalents LSF Equivalents of Slurm srun Options Requests a specific list of nodes. Slurm Suppress informational message Determining IP Address of Your Local Machine Advanced TopicsEnabling Remote Execution with OpenSSH Running an X Terminal Session from a Remote Node Running an X terminal Session Using LSF Running an X terminal Session Using SlurmLogging in to HP XC System Monitors display server address Using the GNU Parallel Make CapabilityOptions used in this command are Allocate 4 cores Run the job on 1 core $ cd subdir srun -n1 -N1 $MAKE -j4 $ cd subdir srun $MAKE $ make PREFIX=’srun -n1 -N1 MAKEJ=-j4 Example Procedure Modified Makefile is invoked as follows Local Disks on Compute Nodes$ make PREFIX=srun -n1 -N1 MAKEJ=-j4 Shared File View Using Mpich on the HP XC System11.5 I/O Performance Considerations Communication Between Nodes Using Mpich with LSF Allocation Using Mpich with Slurm AllocationBsub command launches the wrapper script Examine the LSF execution host information ExamplesBuilding and Running a Serial Application Launching a Serial Interactive Shell Through LSF Running LSF Jobs with a Slurm Allocation Request Example 2. Four Cores on Two Specific Nodes Launching a Parallel Interactive Shell Through LSF $ hostname n16 $ srun hostname n5 $ bjobs Examine the running jobs information Display the script Submitting a Simple Job Script with LSFExamine the finished jobs information Show the environment Show the job allocation Submitting an Interactive Job with LSFRun the job Submit the job View the interactive jobs Run some commands from the pseudo-terminalShow the Slurm job ID Exit the pseudo-terminal View the node state Submitting an HP-MPI Job with LSFView the running job View the finished job Using a Resource Requirements String in an LSF Command$ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob 124 Network Availability set Glossary Fcfs Ipmi LVS PXE SVA Index Index LVS PGI 135