HP
XC System 3.x Software
manual
114
Fault Tolerance
Login
Setting Up TotalView
$ man lsfcommandname
Example Procedure
Tuning Applications
Slurm backup
Setting Debugging Options
$ scancel --user username
Using TotalView with LSF-HPC
Page 114
114
Page 113
Page 115
Image 114
Page 113
Page 115
Contents
HP XC System Software Users Guide
Page
Table of Contents
Configuring Your Environment with Modulefiles
Developing Applications
Submitting Jobs
Using LSF
Tuning Applications
Using Slurm
Debugging Applications
Advanced Topics
Examples
Glossary 109 Index 115
List of Figures
Page
List of Tables
Determining the Node Platform
Page
List of Examples
Submitting a Job Script
Page
Document Organization
About This Document
Intended Audience
This document is organized as follows
HP XC Information
$ man lsfcommandname
For More Information
Supplementary Information
$ man discover $ man 8 discover
Related Information
Manpages
$ man -k keyword
Related Compiler Web Sites
Related Linux Web Sites
Related MPI Web Sites
Additional Publications
Environment Variable
Typographic Conventions
HP Encourages Your Comments
User input
HP XC System Software
Overview of the User Environment
System Architecture
Operating System
Node Specialization
Storage and I/O
Local Storage
File System
SAN Storage
File System Layout
Determining System Configuration Information
System Interconnect Network
Network Address Translation NAT
Commands
User Environment
Modules
Parallel Applications
Application Development Environment
Run-Time Environment
Serial Applications
Load Sharing Facility LSF-HPC
Standard LSF
How LSF-HPC and Slurm Interact
Mpirun command
Components, Tools, Compilers, Libraries, and Debuggers
Using the Secure Shell to Log
Using the System
LVS Login Routing
Logging In to the System
Introduction
Getting Information About Queues
Getting Information About Resources
Performing Other Common User Tasks
Getting System Help and Information
$ man sinfo
Configuring Your Environment with Modulefiles
Overview of Modules
Supplied Modulefiles
Viewing Loaded Modulefiles
Modulefiles Automatically Loaded on the System
Viewing Available Modulefiles
Loading a Modulefile
Modulefile Conflicts
Automatically Loading a Modulefile at Login
Unloading a Modulefile
Loading a Modulefile for the Current Session
$ module load modules $ man modulefile
Creating a Modulefile
Viewing Modulefile-Specific Help
$ module help totalview
Page
Developing Applications
Application Development Environment Overview
Compilers
MPI Compiler
Examining Nodes and Partitions Before Running Jobs
Interrupting a Job
Partition Avail Timelimit Nodes State Nodelist
Serial Application Build Environment
Setting Debugging Options
Developing Serial Applications
Building Serial Applications
Modulefiles
Developing Parallel Applications
Parallel Application Build Environment
OpenMP
MPI Library
Pthreads
Quadrics Shmem
Intel Fortran and C/C++Compilers
Building Parallel Applications
Developing Libraries
Designing Libraries for the CP4000 Platform
Examples of Compiling and Linking HP-MPI Applications
Linkcommand 32-bit -L/opt/mypackage/lib/i686 -lmystuff
Linkcommand 64-bit -L/opt/mypackage/lib/x8664 -lmystuff
To build a 64-bit application, you might enter
Submitting Jobs
Overview of Job Submission
ExtSLURMslurm-arguments
Submitting a Serial Job with the LSF bsub Command
Submitting a Serial Job Using Standard LSF
Submitting a Serial Job Using LSF-HPC
$ bsub hostname
Submitting a Serial Job Through Slurm only
Bsub -nnum-procsbsub-optionsmpijob
Submitting a Non-MPI Parallel Job
$ bsub -n4 -I srun hostname
Mpirun mpirun--options-srunsrun-optionsmpi-jobname
Submitting a Batch Job or Job Script
$ bsub -n4 -I mpirun -srun ./helloworld
Bsub -nnum-procs bsub-optionsscript-name
$ bsub -I -n4 Myscript.sh
$ cat myscript.sh #!/bin/sh
Srun hostname mpirun -srun hellompi
$ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh
Running Preexecution Programs
$ bsub -n4 -I ./myscript.sh
$ cat ./envscript.sh #!/bin/sh name=`hostname`
Opt/hptc/bin/srun Mypreexec
Debugging Parallel Applications
Debugging Applications
Debugging Serial Applications
TotalView
SSH and TotalView
Setting Up TotalView
Using TotalView with Slurm
Module load mpimodule load totalview
Using TotalView with LSF-HPC
Setting TotalView Preferences
Debugging an Application
Directories in File ⇒ Search Path
Sourcefile initfdte.f was not found, using assembler mode
Debugging Running Applications
$ mpirun -srun -n2 Psimple
$ scancel --user username
Exiting TotalView
$ squeue
Page
Tuning Applications
Using the Intel Trace Collector and Intel Trace Analyzer
Building a Program Intel Trace Collector and HP-MPI
Libs
Running a Program Intel Trace Collector and HP-MPI
Visualizing Data Intel Trace Analyzer and HP-MPI
Cldflags
Using the Intel Trace Collector and Intel Trace Analyzer
Page
Srun Squeue Scancel Sinfo Scontrol
Using Slurm
Launching Jobs with the srun Command
Introduction to Slurm
Using the srun Command with LSF-HPC
Monitoring Jobs with the squeue Command
Using the srun Command with HP-MPI
Srun Roles and Modes
Terminating Jobs with the scancel Command
Getting System Information with the sinfo Command
Job Accounting
Fault Tolerance
Security
# chmod a+r /hptccluster/slurm/job/jobacct.log
Using LSF
Using Standard LSF on an HP XC System
Using LSF-HPC
Introduction to LSF-HPC in the HP XC Environment
Overview of LSF-HPC
Resources
Differences Between LSF-HPC and Standard LSF
Hostname
Hostname Status JL/U MAX Njobs RUN Ssusp Ususp RSV
Job Terminology
$ ssh n15 lshosts
Unknown Unknown
HP XCCompute Node Resource Support
SLURMnodelist =nodelist if specified
$ bsub -n 10 -ext SLURMnodes=10exclude=n16 -I srun hostname
$ bsub -n 10 -I srun hostname
$ bsub -n 10 -ext SLURMnodes=10 -I srun hostname
$ bsub -n 10 -ext SLURMconstraint=dualcore -I srun hostname
How LSF-HPC and Slurm Launch and Manage a Job
$ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript
Job Startup and Job Control
#!/bin/sh hostname srun hostname Mpirun -srun ./hellompi
Getting the Status of LSF-HPC
Determining the LSF Execution Host
Determining Available LSF-HPC System Resources
Getting Information About LSF Execution Host Node
Getting Information About the lsf Partition
Getting Host Load Information
Examining LSF-HPC System Queues
SLINUX6
Summary of the LSF bsub Command Format
$ sinfo -p lsf
$ sinfo -p lsf -lNe
Bsub -n num-procs-ext SLURMslurm-arguments \
LSF-SLURM External Scheduler
For information about running scripts
Bsub-options srun srun-optionsjobname job-options
Waiting for dispatch ... Starting on lsfhost.localdomain n1
Submitting a Job from a Non-HP XC Host
Starting on lsfhost.localdomain n6
Type=SLINUX64
Slurmid=slurmjobidncpus=slurmnprocsslurmalloc=nodelist
Getting Information About Jobs
Getting Job Allocation Information
$ bjobs -l
Time stamp
Examining the Status of a Job
$ bhist -l
$ bjobs
Summary of time in seconds spent Various States
Viewing the Historical Information for a Job
$ bhist
Jobid User Jobname Pend Psusp RUN Ususp Ssusp Unkwn Total
Translating Slurm and LSF-HPC JOBIDs
$ bjobs -l 124 grep slurm
Working Interactively Within an LSF-HPC Allocation
$ bsub -I -n4 -ext SLURMnodes=4 /bin/bash
$ srun --jobid=150 hostname
$ export SLURMJOBID=150 $ export SLURMNPROCS=4
$ unset Slurmjobid
Alternatively, you can use the following
$ unset Slurmjobid $ unset Slurmnprocs
$ srun --jobid=250 uptime
Job 125 is submitted to the default queue normal
LSF-HPC Equivalents of Slurm srun Options
$ bsub -n4 -ext SLURMnodes=4 -o %J.out sleep
Bsub -iinputfile
Mpi=mpitype
Quit-on-interrupt
Page
Running an X Terminal Session from a Remote Node
Advanced Topics
Enabling Remote Execution with OpenSSH
Determining IP Address of Your Local Machine
Running an X terminal Session Using Slurm
Running an X terminal Session Using LSF-HPC
Logging in to HP XC System
$ srun -n4 hostname n46
Using the GNU Parallel Make Capability
$ bsub -n4 -Ip srun -n1 xterm -display
$ srun -n2 hostname n46
$ cd subdir srun -n1 -N1 $MAKE -j4
Example Procedure
$ make PREFIX=’srun -n1 -N1 MAKEJ=-j4
Modified Makefile is invoked as follows
Local Disks on Compute Nodes
Performance Considerations
$ make PREFIX=srun -n1 -N1 MAKEJ=-j4
Private File View
Communication Between Nodes
Shared File View
Fp = fopen myfile, a+
Page
Launching a Serial Interactive Shell Through LSF-HPC
Appendix a Examples
Building and Running a Serial Application
Examine the LSF execution host information
Running LSF-HPC Jobs with a Slurm Allocation Request
R15s r1m r15m It tmp swp mem LoadSched LoadStop
Launching a Parallel Interactive Shell Through LSF-HPC
Example 2. Four cores on Two Specific Nodes
SLURMnodes=2
Examine the the running jobs information
$ hostname n16 $ srun hostname n5 $ bjobs
124 Lsfad
Show the environment
Submitting a Simple Job Script with LSF-HPC
Examine the the finished jobs information
Display the script
Submit the job
Run some commands from the pseudo-terminal
Submitting an Interactive Job with LSF-HPC
Show the job allocation
View the interactive jobs
Submitting an HP-MPI Job with LSF-HPC
Exit the pseudo-terminal
View the finished jobs
View the running job
View the finished job
Lsfhost.localdomai
Using a Resource Requirements String in an LSF-HPC Command
States by date and time
$ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob
108
Glossary
As local storage
To the queue
First-come See Fcfs First-served Global storage
Are not appropriate for replication
Linux Virtual See LVS Server Load file
Single command
Login requests and directs them to a node with a login role
LSF master host
Notably to install and remove software packages
Network See NIS Information Services
Remotely. PXE booting is configured at the Bios level
Slurm backup
Power available per unit of space
Ssh
Symmetric See SMP Multiprocessing
114
Index
Index
PGI
Utilities, 63 Slurm commands
Top
Page
Image
Contents