Advanced Topics, Examples | HP XC System 3.x Software instruction

Getting Information About the lsf Partition	76
Submitting Jobs	77
Summary of the LSF bsub Command Format	77
LSF-SLURM External Scheduler	78
Submitting a Job from a Non-HP XC Host	79
Getting Information About Jobs	80
Getting Job Allocation Information	80
Job Allocation Information for a Running Job	80
Job Allocation Information for a Finished Job	81
Examining the Status of a Job	81
Viewing the Historical Information for a Job	82
Translating SLURM and LSF-HPC JOBIDs	83
Working Interactively Within an LSF-HPC Allocation	84
LSF-HPC Equivalents of SLURM srun Options	86
10 Advanced Topics
Enabling Remote Execution with OpenSSH	91
Running an X Terminal Session from a Remote Node	91
Using the GNU Parallel Make Capability	93
Example Procedure 1	95
Example Procedure 2	95
Example Procedure 3	96
Local Disks on Compute Nodes	96
I/O Performance Considerations	96
Shared File View	97
Private File View	97
Communication Between Nodes	97
A Examples	99
Building and Running a Serial Application	99
Launching a Serial Interactive Shell Through LSF-HPC	99
Running LSF-HPC Jobs with a SLURM Allocation Request	100
Example 1. Two cores on Any Two Nodes	100
Example 2. Four cores on Two Specific Nodes	101
Launching a Parallel Interactive Shell Through LSF-HPC	101
Submitting a Simple Job Script with LSF-HPC	103
Submitting an Interactive Job with LSF-HPC	104
Submitting an HP-MPI Job with LSF-HPC	105
Using a Resource Requirements String in an LSF-HPC Command	107
Glossary	109
Index	115

6Table of Contents

Image 6

HP XC System 3.x Software manual Advanced Topics, Examples, Glossary 109 Index 115

Contents

HP XC System Software Users Guide Page Table of Contents Configuring Your Environment with Modulefiles Developing ApplicationsSubmitting Jobs Using LSF Tuning ApplicationsUsing Slurm Debugging Applications Advanced Topics ExamplesGlossary 109 Index 115 List of Figures Page List of Tables Determining the Node PlatformPage List of Examples Submitting a Job ScriptPage Document Organization About This DocumentIntended Audience This document is organized as follows HP XC Information $ man lsfcommandname For More InformationSupplementary Information $ man discover $ man 8 discover Related InformationManpages $ man -k keyword Related Compiler Web Sites Related Linux Web SitesRelated MPI Web Sites Additional Publications Environment Variable Typographic ConventionsHP Encourages Your Comments User input HP XC System Software Overview of the User EnvironmentSystem Architecture Operating System Node Specialization Storage and I/O Local Storage File SystemSAN Storage File System Layout Determining System Configuration Information System Interconnect NetworkNetwork Address Translation NAT Commands User EnvironmentModules Parallel Applications Application Development EnvironmentRun-Time Environment Serial Applications Load Sharing Facility LSF-HPC Standard LSFHow LSF-HPC and Slurm Interact Mpirun command Components, Tools, Compilers, Libraries, and Debuggers Using the Secure Shell to Log Using the SystemLVS Login Routing Logging In to the System Introduction Getting Information About QueuesGetting Information About Resources Performing Other Common User Tasks Getting System Help and Information $ man sinfo Configuring Your Environment with Modulefiles Overview of Modules Supplied Modulefiles Viewing Loaded Modulefiles Modulefiles Automatically Loaded on the SystemViewing Available Modulefiles Loading a Modulefile Modulefile Conflicts Automatically Loading a Modulefile at LoginUnloading a Modulefile Loading a Modulefile for the Current Session $ module load modules $ man modulefile Creating a ModulefileViewing Modulefile-Specific Help $ module help totalviewPage Developing Applications Application Development Environment OverviewCompilers MPI Compiler Examining Nodes and Partitions Before Running JobsInterrupting a Job Partition Avail Timelimit Nodes State Nodelist Serial Application Build Environment Setting Debugging OptionsDeveloping Serial Applications Building Serial Applications Modulefiles Developing Parallel ApplicationsParallel Application Build Environment OpenMP MPI Library PthreadsQuadrics Shmem Intel Fortran and C/C++Compilers Building Parallel Applications Developing Libraries Designing Libraries for the CP4000 PlatformExamples of Compiling and Linking HP-MPI Applications Linkcommand 32-bit -L/opt/mypackage/lib/i686 -lmystuff Linkcommand 64-bit -L/opt/mypackage/lib/x8664 -lmystuffTo build a 64-bit application, you might enter Submitting Jobs Overview of Job SubmissionExtSLURMslurm-arguments Submitting a Serial Job with the LSF bsub Command Submitting a Serial Job Using Standard LSFSubmitting a Serial Job Using LSF-HPC $ bsub hostname Submitting a Serial Job Through Slurm only Bsub -nnum-procsbsub-optionsmpijob Submitting a Non-MPI Parallel Job$ bsub -n4 -I srun hostname Mpirun mpirun--options-srunsrun-optionsmpi-jobname Submitting a Batch Job or Job Script $ bsub -n4 -I mpirun -srun ./helloworldBsub -nnum-procs bsub-optionsscript-name $ bsub -I -n4 Myscript.sh $ cat myscript.sh #!/bin/shSrun hostname mpirun -srun hellompi $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh Running Preexecution Programs $ bsub -n4 -I ./myscript.sh$ cat ./envscript.sh #!/bin/sh name=`hostname` Opt/hptc/bin/srun Mypreexec Debugging Parallel Applications Debugging ApplicationsDebugging Serial Applications TotalView SSH and TotalView Setting Up TotalViewUsing TotalView with Slurm Module load mpimodule load totalview Using TotalView with LSF-HPC Setting TotalView PreferencesDebugging an Application Directories in File ⇒ Search Path Sourcefile initfdte.f was not found, using assembler modeDebugging Running Applications $ mpirun -srun -n2 Psimple $ scancel --user username Exiting TotalView$ squeue Page Tuning Applications Using the Intel Trace Collector and Intel Trace AnalyzerBuilding a Program Intel Trace Collector and HP-MPI Libs Running a Program Intel Trace Collector and HP-MPIVisualizing Data Intel Trace Analyzer and HP-MPI Cldflags Using the Intel Trace Collector and Intel Trace Analyzer Page Srun Squeue Scancel Sinfo Scontrol Using SlurmLaunching Jobs with the srun Command Introduction to Slurm Using the srun Command with LSF-HPC Monitoring Jobs with the squeue CommandUsing the srun Command with HP-MPI Srun Roles and Modes Terminating Jobs with the scancel Command Getting System Information with the sinfo CommandJob Accounting Fault Tolerance Security# chmod a+r /hptccluster/slurm/job/jobacct.log Using LSF Using Standard LSF on an HP XC SystemUsing LSF-HPC Introduction to LSF-HPC in the HP XC Environment Overview of LSF-HPC Resources Differences Between LSF-HPC and Standard LSFHostname Hostname Status JL/U MAX Njobs RUN Ssusp Ususp RSV Job Terminology $ ssh n15 lshostsUnknown Unknown HP XCCompute Node Resource Support SLURMnodelist =nodelist if specified $ bsub -n 10 -ext SLURMnodes=10exclude=n16 -I srun hostname $ bsub -n 10 -I srun hostname$ bsub -n 10 -ext SLURMnodes=10 -I srun hostname $ bsub -n 10 -ext SLURMconstraint=dualcore -I srun hostname How LSF-HPC and Slurm Launch and Manage a Job $ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript Job Startup and Job Control #!/bin/sh hostname srun hostname Mpirun -srun ./hellompi Getting the Status of LSF-HPC Determining the LSF Execution HostDetermining Available LSF-HPC System Resources Getting Information About LSF Execution Host Node Getting Information About the lsf Partition Getting Host Load InformationExamining LSF-HPC System Queues SLINUX6 Summary of the LSF bsub Command Format $ sinfo -p lsf$ sinfo -p lsf -lNe Bsub -n num-procs-ext SLURMslurm-arguments \ LSF-SLURM External SchedulerFor information about running scripts Bsub-options srun srun-optionsjobname job-options Waiting for dispatch ... Starting on lsfhost.localdomain n1 Submitting a Job from a Non-HP XC HostStarting on lsfhost.localdomain n6 Type=SLINUX64 Slurmid=slurmjobidncpus=slurmnprocsslurmalloc=nodelist Getting Information About JobsGetting Job Allocation Information $ bjobs -l Time stamp Examining the Status of a Job$ bhist -l $ bjobs Summary of time in seconds spent Various States Viewing the Historical Information for a Job$ bhist Jobid User Jobname Pend Psusp RUN Ususp Ssusp Unkwn Total Translating Slurm and LSF-HPC JOBIDs $ bjobs -l 124 grep slurm Working Interactively Within an LSF-HPC Allocation$ bsub -I -n4 -ext SLURMnodes=4 /bin/bash $ srun --jobid=150 hostname $ export SLURMJOBID=150 $ export SLURMNPROCS=4 $ unset SlurmjobidAlternatively, you can use the following $ unset Slurmjobid $ unset Slurmnprocs $ srun --jobid=250 uptime Job 125 is submitted to the default queue normalLSF-HPC Equivalents of Slurm srun Options $ bsub -n4 -ext SLURMnodes=4 -o %J.out sleep Bsub -iinputfile Mpi=mpitype Quit-on-interrupt Page Running an X Terminal Session from a Remote Node Advanced TopicsEnabling Remote Execution with OpenSSH Determining IP Address of Your Local Machine Running an X terminal Session Using Slurm Running an X terminal Session Using LSF-HPCLogging in to HP XC System $ srun -n4 hostname n46 Using the GNU Parallel Make Capability$ bsub -n4 -Ip srun -n1 xterm -display $ srun -n2 hostname n46 $ cd subdir srun -n1 -N1 $MAKE -j4 Example Procedure $ make PREFIX=’srun -n1 -N1 MAKEJ=-j4 Modified Makefile is invoked as follows Local Disks on Compute NodesPerformance Considerations $ make PREFIX=srun -n1 -N1 MAKEJ=-j4 Private File View Communication Between NodesShared File View Fp = fopen myfile, a+Page Launching a Serial Interactive Shell Through LSF-HPC Appendix a ExamplesBuilding and Running a Serial Application Examine the LSF execution host information Running LSF-HPC Jobs with a Slurm Allocation Request R15s r1m r15m It tmp swp mem LoadSched LoadStop Launching a Parallel Interactive Shell Through LSF-HPCExample 2. Four cores on Two Specific Nodes SLURMnodes=2 Examine the the running jobs information $ hostname n16 $ srun hostname n5 $ bjobs124 Lsfad Show the environment Submitting a Simple Job Script with LSF-HPCExamine the the finished jobs information Display the script Submit the job Run some commands from the pseudo-terminalSubmitting an Interactive Job with LSF-HPC Show the job allocation View the interactive jobs Submitting an HP-MPI Job with LSF-HPCExit the pseudo-terminal View the finished jobs View the running job View the finished jobLsfhost.localdomai Using a Resource Requirements String in an LSF-HPC Command States by date and time$ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob 108 Glossary As local storage To the queueFirst-come See Fcfs First-served Global storage Are not appropriate for replication Linux Virtual See LVS Server Load file Single commandLogin requests and directs them to a node with a login role LSF master host Notably to install and remove software packages Network See NIS Information ServicesRemotely. PXE booting is configured at the Bios level Slurm backup Power available per unit of space SshSymmetric See SMP Multiprocessing 114 Index Index PGI Utilities, 63 Slurm commands