HP
XC System 3.x Software
manual
Fault Tolerance
Login
Setting Up TotalView
$ man lsfcommandname
Example Procedure
Tuning Applications
Slurm backup
Setting Debugging Options
$ scancel --user username
Using TotalView with LSF-HPC
Page 62
62
Page 61
Page 63
Image 62
Page 61
Page 63
Contents
HP XC System Software Users Guide
Page
Table of Contents
Submitting Jobs
Configuring Your Environment with Modulefiles
Developing Applications
Using LSF
Tuning Applications
Using Slurm
Debugging Applications
Glossary 109 Index 115
Advanced Topics
Examples
List of Figures
Page
List of Tables
Determining the Node Platform
Page
List of Examples
Submitting a Job Script
Page
Document Organization
About This Document
Intended Audience
This document is organized as follows
HP XC Information
Supplementary Information
$ man lsfcommandname
For More Information
$ man discover $ man 8 discover
Related Information
Manpages
$ man -k keyword
Related Compiler Web Sites
Related Linux Web Sites
Related MPI Web Sites
Additional Publications
Environment Variable
Typographic Conventions
HP Encourages Your Comments
User input
HP XC System Software
Overview of the User Environment
System Architecture
Operating System
Node Specialization
Storage and I/O
Local Storage
File System
SAN Storage
File System Layout
Network Address Translation NAT
Determining System Configuration Information
System Interconnect Network
Modules
Commands
User Environment
Parallel Applications
Application Development Environment
Run-Time Environment
Serial Applications
How LSF-HPC and Slurm Interact
Load Sharing Facility LSF-HPC
Standard LSF
Mpirun command
Components, Tools, Compilers, Libraries, and Debuggers
Using the Secure Shell to Log
Using the System
LVS Login Routing
Logging In to the System
Getting Information About Resources
Introduction
Getting Information About Queues
Performing Other Common User Tasks
Getting System Help and Information
$ man sinfo
Configuring Your Environment with Modulefiles
Overview of Modules
Supplied Modulefiles
Viewing Loaded Modulefiles
Modulefiles Automatically Loaded on the System
Viewing Available Modulefiles
Loading a Modulefile
Modulefile Conflicts
Automatically Loading a Modulefile at Login
Unloading a Modulefile
Loading a Modulefile for the Current Session
$ module load modules $ man modulefile
Creating a Modulefile
Viewing Modulefile-Specific Help
$ module help totalview
Page
Compilers
Developing Applications
Application Development Environment Overview
MPI Compiler
Examining Nodes and Partitions Before Running Jobs
Interrupting a Job
Partition Avail Timelimit Nodes State Nodelist
Serial Application Build Environment
Setting Debugging Options
Developing Serial Applications
Building Serial Applications
Modulefiles
Developing Parallel Applications
Parallel Application Build Environment
OpenMP
MPI Library
Pthreads
Quadrics Shmem
Intel Fortran and C/C++Compilers
Building Parallel Applications
Examples of Compiling and Linking HP-MPI Applications
Developing Libraries
Designing Libraries for the CP4000 Platform
To build a 64-bit application, you might enter
Linkcommand 32-bit -L/opt/mypackage/lib/i686 -lmystuff
Linkcommand 64-bit -L/opt/mypackage/lib/x8664 -lmystuff
ExtSLURMslurm-arguments
Submitting Jobs
Overview of Job Submission
Submitting a Serial Job with the LSF bsub Command
Submitting a Serial Job Using Standard LSF
Submitting a Serial Job Using LSF-HPC
$ bsub hostname
Submitting a Serial Job Through Slurm only
Bsub -nnum-procsbsub-optionsmpijob
Submitting a Non-MPI Parallel Job
$ bsub -n4 -I srun hostname
Mpirun mpirun--options-srunsrun-optionsmpi-jobname
Bsub -nnum-procs bsub-optionsscript-name
Submitting a Batch Job or Job Script
$ bsub -n4 -I mpirun -srun ./helloworld
$ bsub -I -n4 Myscript.sh
$ cat myscript.sh #!/bin/sh
Srun hostname mpirun -srun hellompi
$ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh
$ cat ./envscript.sh #!/bin/sh name=`hostname`
Running Preexecution Programs
$ bsub -n4 -I ./myscript.sh
Opt/hptc/bin/srun Mypreexec
Debugging Parallel Applications
Debugging Applications
Debugging Serial Applications
TotalView
SSH and TotalView
Setting Up TotalView
Using TotalView with Slurm
Module load mpimodule load totalview
Debugging an Application
Using TotalView with LSF-HPC
Setting TotalView Preferences
Directories in File ⇒ Search Path
Sourcefile initfdte.f was not found, using assembler mode
Debugging Running Applications
$ mpirun -srun -n2 Psimple
$ squeue
$ scancel --user username
Exiting TotalView
Page
Building a Program Intel Trace Collector and HP-MPI
Tuning Applications
Using the Intel Trace Collector and Intel Trace Analyzer
Libs
Running a Program Intel Trace Collector and HP-MPI
Visualizing Data Intel Trace Analyzer and HP-MPI
Cldflags
Using the Intel Trace Collector and Intel Trace Analyzer
Page
Srun Squeue Scancel Sinfo Scontrol
Using Slurm
Launching Jobs with the srun Command
Introduction to Slurm
Using the srun Command with LSF-HPC
Monitoring Jobs with the squeue Command
Using the srun Command with HP-MPI
Srun Roles and Modes
Job Accounting
Terminating Jobs with the scancel Command
Getting System Information with the sinfo Command
# chmod a+r /hptccluster/slurm/job/jobacct.log
Fault Tolerance
Security
Using LSF-HPC
Using LSF
Using Standard LSF on an HP XC System
Introduction to LSF-HPC in the HP XC Environment
Overview of LSF-HPC
Resources
Differences Between LSF-HPC and Standard LSF
Hostname
Hostname Status JL/U MAX Njobs RUN Ssusp Ususp RSV
Unknown Unknown
Job Terminology
$ ssh n15 lshosts
HP XCCompute Node Resource Support
SLURMnodelist =nodelist if specified
$ bsub -n 10 -ext SLURMnodes=10exclude=n16 -I srun hostname
$ bsub -n 10 -I srun hostname
$ bsub -n 10 -ext SLURMnodes=10 -I srun hostname
$ bsub -n 10 -ext SLURMconstraint=dualcore -I srun hostname
How LSF-HPC and Slurm Launch and Manage a Job
$ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript
Job Startup and Job Control
#!/bin/sh hostname srun hostname Mpirun -srun ./hellompi
Getting the Status of LSF-HPC
Determining the LSF Execution Host
Determining Available LSF-HPC System Resources
Getting Information About LSF Execution Host Node
Getting Information About the lsf Partition
Getting Host Load Information
Examining LSF-HPC System Queues
SLINUX6
$ sinfo -p lsf -lNe
Summary of the LSF bsub Command Format
$ sinfo -p lsf
Bsub -n num-procs-ext SLURMslurm-arguments \
LSF-SLURM External Scheduler
For information about running scripts
Bsub-options srun srun-optionsjobname job-options
Waiting for dispatch ... Starting on lsfhost.localdomain n1
Submitting a Job from a Non-HP XC Host
Starting on lsfhost.localdomain n6
Type=SLINUX64
Slurmid=slurmjobidncpus=slurmnprocsslurmalloc=nodelist
Getting Information About Jobs
Getting Job Allocation Information
$ bjobs -l
Time stamp
Examining the Status of a Job
$ bhist -l
$ bjobs
Summary of time in seconds spent Various States
Viewing the Historical Information for a Job
$ bhist
Jobid User Jobname Pend Psusp RUN Ususp Ssusp Unkwn Total
Translating Slurm and LSF-HPC JOBIDs
$ bjobs -l 124 grep slurm
Working Interactively Within an LSF-HPC Allocation
$ bsub -I -n4 -ext SLURMnodes=4 /bin/bash
$ srun --jobid=150 hostname
$ export SLURMJOBID=150 $ export SLURMNPROCS=4
$ unset Slurmjobid
Alternatively, you can use the following
$ unset Slurmjobid $ unset Slurmnprocs
$ srun --jobid=250 uptime
Job 125 is submitted to the default queue normal
LSF-HPC Equivalents of Slurm srun Options
$ bsub -n4 -ext SLURMnodes=4 -o %J.out sleep
Bsub -iinputfile
Mpi=mpitype
Quit-on-interrupt
Page
Running an X Terminal Session from a Remote Node
Advanced Topics
Enabling Remote Execution with OpenSSH
Determining IP Address of Your Local Machine
Logging in to HP XC System
Running an X terminal Session Using Slurm
Running an X terminal Session Using LSF-HPC
$ srun -n4 hostname n46
Using the GNU Parallel Make Capability
$ bsub -n4 -Ip srun -n1 xterm -display
$ srun -n2 hostname n46
$ cd subdir srun -n1 -N1 $MAKE -j4
Example Procedure
$ make PREFIX=’srun -n1 -N1 MAKEJ=-j4
Modified Makefile is invoked as follows
Local Disks on Compute Nodes
Performance Considerations
$ make PREFIX=srun -n1 -N1 MAKEJ=-j4
Private File View
Communication Between Nodes
Shared File View
Fp = fopen myfile, a+
Page
Launching a Serial Interactive Shell Through LSF-HPC
Appendix a Examples
Building and Running a Serial Application
Examine the LSF execution host information
Running LSF-HPC Jobs with a Slurm Allocation Request
R15s r1m r15m It tmp swp mem LoadSched LoadStop
Launching a Parallel Interactive Shell Through LSF-HPC
Example 2. Four cores on Two Specific Nodes
SLURMnodes=2
124 Lsfad
Examine the the running jobs information
$ hostname n16 $ srun hostname n5 $ bjobs
Show the environment
Submitting a Simple Job Script with LSF-HPC
Examine the the finished jobs information
Display the script
Submit the job
Run some commands from the pseudo-terminal
Submitting an Interactive Job with LSF-HPC
Show the job allocation
View the interactive jobs
Submitting an HP-MPI Job with LSF-HPC
Exit the pseudo-terminal
View the finished jobs
Lsfhost.localdomai
View the running job
View the finished job
$ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob
Using a Resource Requirements String in an LSF-HPC Command
States by date and time
108
Glossary
As local storage
To the queue
First-come See Fcfs First-served Global storage
Are not appropriate for replication
Linux Virtual See LVS Server Load file
Single command
Login requests and directs them to a node with a login role
LSF master host
Notably to install and remove software packages
Network See NIS Information Services
Remotely. PXE booting is configured at the Bios level
Slurm backup
Symmetric See SMP Multiprocessing
Power available per unit of space
Ssh
114
Index
Index
PGI
Utilities, 63 Slurm commands
Related pages
Where can I find
Zanussi parts
for my refrigerator?
Top
Page
Image
Contents