HP XC System 2.x Software guide

Image 144

HP XC System 2.x Software manual

Contents

June Product Version Hewlett-Packard Company Palo Alto, CaliforniaPage Contents Developing Applications 5.4 Tuning Applications Using SlurmDebugging Applications Using LSF Using HP-MPI Using HP Mlib 6.4 Advanced Topics ExamplesGlossary Index Examples Figures TablesPage About This Document Intended AudienceDocument Organization Linux Administration Handbook HP XC InformationQuickSpecs for HP XC System Software HP XC Program Development Environment For More Information Supplementary InformationHP Message Passing Interface HP Mathematical Library Manpages Http//supermon.sourceforge.netHttp//systemimager.org Http//sourceforge.net/projects/modules Related Information Http//linuxvirtualserver.orgHttp//www-unix.mcs.anl.gov/mpi # cd /opt/hptc/config/sbin Typographical ConventionsBold text HP Encourages Your Comments Ctrl/xPage Overview of the User Environment System ArchitectureOperating System Node Specialization Storage and I/O File SystemSAN Storage Local Storage System Interconnect Network File System Layout Commands User EnvironmentNetwork Address Translation NAT 1 LVS Application Development Environment Parallel ApplicationsSerial Applications Run-Time Environment SlurmLoad Sharing Facility LSF-HPC How LSF-HPC and Slurm Interact Components, Tools, Compilers, Libraries, and Debuggers HP-MPI 8Overview of the User Environment Using the System Configuring Your Environment with ModulefilesLVS Login Routing Using ssh to Log 2Using the System Supplied Modulefiles Supplied ModulefilesModulefile Sets the HP XC User Environment Modulefiles Automatically Loaded on the System Viewing Available ModulefilesViewing Loaded Modulefiles Loading a Modulefile Automatically Loading a Modulefile at Login Unloading a ModulefileModulefile Conflicts Loading a Modulefile for the Current Session Creating a Modulefile Viewing Modulefile-Specific Help$ module load modules $ man modulefile $ module help totalview Launching and Managing Jobs Quick Start IntroductionGetting Information About Queues Getting Information About Resources Getting Information About the System’s Partitions Launching JobsSubmitting a Serial Job Example 2-1 Submitting a Serial Job Using Slurm Options with the LSF External Scheduler Submitting a Non-MPI Parallel JobExample 2-2 Submitting a Non-MPI Parallel Job $ bsub -n4 -I srun hostname Submitting an MPI Job Example 2-4 Running an MPI Job with LSF$ bsub -n4 Mpirun -srun ./helloworld Submitting a Batch Job or Job Script Example 2-6 Submitting a Job Script$ bsub -I -n4 myjobscript.sh Performing Other Common User Tasks Getting System Help and Information $ man sinfo $ man -k keywordPage Developing Applications Overview Using Compilers Standard Linux CompilersIntel Compilers PGI Compilers Setting Debugging Options Checking Nodes and Partitions Before Running JobsInterrupting a Job Developing Serial Applications Using Mlib in Serial Applications Developing Parallel ApplicationsSerial Application Build Environment Building Serial Applications Parallel Application Build Environment ModulefilesHP-MPI OpenMP Quadrics Shmem Mlib Math LibraryMPI Library $ mpicc object1.o ... -pthread -o myapp.exe Intel Fortran and C/C++Compilers PGI Fortran and C/C++ CompilersGNU C and C++ Compilers GNU Parallel Make Reserved Symbols and Names Building Parallel ApplicationsCompiling and Linking Non-MPI Applications Compiling and Linking HP-MPI Applications Developing Libraries Designing Libraries for XC4000$ mpicc -c -g foo.c Using the GNU Parallel Make Capability Advanced TopicsExample 3-1 Directory Structure Example 3-2 Recommended Directory Structure $ cd subdir srun $MAKE $ cd subdir srun -n1 -N1 $MAKE -j4 Example Procedure $ make PREFIX=’srun -n1 -N1 MAKEJ=’-j4’ $ make PREFIX=’srun -n1 -N1’ MAKEJ=’-j4’ Local Disks on Compute Nodes 3 I/O Performance ConsiderationsShared File View Private File View Communication Between Nodes Page Debugging Applications Debugging Serial ApplicationsDebugging Parallel Applications TotalView Setting Up TotalView Debugging with TotalViewSSH and TotalView Using TotalView with Slurm Using TotalView with LSF-HPC$ srun -Nx-A $ mpirun -tv -srun application $ bsub -nx-ext SLURMnodes=x \ -Is /usr/bin/xterm Starting TotalView for the First Time $ totalview TotalView Preferences Window Preferences window, click on the Launch Strings tab Debugging Applications Debugging an Application $ mpicc -g -o Psimple simple.c -lm$ mpirun -tv -srun -n2 ./Psimple TotalView Process Window Example Debugging Running Applications $ mpirun -srun -n2 Psimple $ scancel --user username Exiting TotalView$ squeue Page Tuning Applications Using the Intel Trace Collector/AnalyzerBuilding a Program Intel Trace Collector and HP-MPI Example Running a Program Intel Trace Collector and HP-MPI Visualizing Data Intel Trace Analyzer and HP-MPIExample Running the vtjacobic Example Program Using Slurm Slurm CommandsCommand Function Introduction Accessing the Slurm Manpages Launching Jobs with the srun CommandSrun Roles and Modes Example 6-1 Simple Launch of a Serial Program Srun Roles Srun Modes Srun Signal Handling Srun Run-Mode OptionsBatch Allocate Srun Resource-Allocation Options Cpt --cpus-per-task=cpt Part --partition=partMinutes --time=minutes Nthreads --threads=nthreads Srun Control Options Srun I/O Options Mode --output=mode Mode --input=modeMode --error=mode Label Srun Constraint Options Clist --constraint=clist Contiguous=yesno Mem=sizeMincpus=n Vmem=size Monitoring Jobs with the squeue Command Using srun with HP-MPIUsing srun with LSF Srun Environment Variables Killing Jobs with the scancel Command Getting System Information with the sinfo Command Fault Tolerance Job AccountingSecurity $ sinfo -R Using LSF Introduction to LSF in the HP XC EnvironmentOverview of LSF Topology Support Nodelist= list-of-nodes Exclude= list-of-nodes $ bsub -n 10 -ext SLURMnodes=10 srun myapp $ bsub -n 10 -ext SLURMnodes=10exclude=n16 srun myapp How LSF and Slurm Launch and Manage a Job $ bqueues -l normal grep Jobstarter$ bsub -Is hostname How LSF-HPC and Slurm Launch and Manage a Job $ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript Differences Between LSF on HP XC and Standard LSF Determining Execution Host Determining Available System ResourcesGetting Status of LSF Job Startup and Job Control Getting Information About LSF-HPC Execution Host Node Getting Host Load Information Submitting Jobs Checking LSF System QueuesGetting Information About the lsf Partition $ sinfo -p lsf Summary of the LSF bsub Command Format Bsub bsuboptions jobname joboptions LSF-SLURM External Scheduler Slurm Arguments Function Starting on lsfhost.localdomain n6 Submitting a Serial Job Submitting a Job in ParallelSubmitting an HP-MPI Job Example 7-5 Submitting an Interactive Serial Job Submitting a Batch Job or Job Script Example 7-6 Submitting an HP-MPI Job$ bsub -n4 -I mpirun -srun ./helloworld Examples Example 7-8 Submitting a Batch Job Script$ bsub -n4 -I ./myscript.sh $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh Submitting a Job from a Non-HP XC Host $ bsub -n4 -I ./myscript.sh -n8 -OType=SLINUX64 Getting Information About Jobs Getting Job Allocation InformationJob Allocation Information for a Running Job $ bsub -R type=SLINUX64 -n4 -I srun hostname Example 7-13 Using the bjobs Command Short Output Checking Status of a JobJob Allocation Information for a Finished Job Example 7-14 Using the bjobs Command Long Output Example 7-15 Using the bhist Command Short OutputOutput Provided by the bhist Command Viewing a Job’s Historical Information Example 7-16 Using the bhist Command Long Output Working Interactively Within an LSF-HPC AllocationSubmitting an Interactive Job to Launch the xterm Program Example 7-17 View Your Environment Example 7-18 View Your Allocation in SlurmExample 7-19 View Your Running Job in LSF Example 7-20 View Job Details in LSF Submitting an Interactive Job to Launch a Shell Example 7-21 Running Jobs from an xterm WindowExample 7-22 Submitting an Interactive Shell Program $ hostname $ srun hostname $ srun -n2 hostname LSF Equivalents of Slurm srun Options LSF Equivalents of Slurm srun Options$ srun hostname n1 $ exit Srun Option Description LSF Equivalent 24Using LSF Control the parallel job Page Using HP-MPI Setting Environment Variables HP-MPI Directory StructureCompiling and Running Applications Building and Running an Example Application Example Application helloworld Building and Running helloworld$ mpicc -o helloworld $MPIROOT/help/helloworld.c $ $MPIROOT/bin/mpirun -srun -n4 helloworld Launching MPI Jobs HP-MPI options allowed with -srun Creating Subshells and Launching Jobsteps System Interconnect Selection$ mpirun -srun -n6 -O -N2 -m cyclic ./a.out host1 rank1 $ mpirun -srun -n4 -N2 -O -m cyclic ./a.out host1 rank1 Using LSF and HP-MPI $ mpirun -subnet 192.168.1.1 -prot -srun -n4 ./a.out $ /usr/sbin/ifconfig -a System Interconnect SupportMPI Versioning Example 8-5 Allocating 12 Processors on 6 Nodes 32-Bit Builds on XC4000 Truncated MessagesAllowing Windows to Use Exclusive Locks Mpirun Command Options $ mpirun -TCP -srun -N8 ./a.out Environment Variables $MPIROOT/bin/mpirun -v -prot -np 2 /path/to/program.x Export MPIPHYSICALMEMORY=1048576 Export MPIPINPERCENTAGE=30Export MPIPAGEALIGNMEM=1 Export MPIMAXWINDOW=10 Mpich Object Compatibility Export MPIUSELIBELAN=0Export MPIUSELIBELANUSE=5 $MPIROOT/bin/mpirun.mpich -np 2 ./prog.x HP-MPI Documentation and Manpages At http//docs.hp.comHP-MPI Manpage Categories Category Manpages Description Additional Information, Known Problems, and Work-arounds Using HP Mlib Intel Compiler NotesVersion 8 Fortran Compiler Version 7 Fortran Compiler HP Mlib for the HP XC6000 Platform Mlib and Module FilesPlatform Support Library Support Using Intel Compilers with HP Mlib Modulefiles and MlibCompiling and Linking HP Mlib for the HP XC4000 Platform LicensingMlib Manpages Linking SuperLUDIST $ pgcc options file 6Using HP Mlib Advanced Topics Enabling Remote Execution with OpenSSHRunning an X Terminal Session from a Remote Node Determining IP Address of Your Local Machine Running an X terminal Session Using Slurm Logging in to HP XC System$ hostname $ host mymachine Running an X terminal Session Using LSF $ bsub -n4 -Ip srun -n1 xterm -display$ bjobs Page Examples Building and Running a Serial ApplicationLaunching a Serial Interactive Shell Through LSF Running LSF Jobs with a Slurm Allocation Request Example 1. Two Processors on Any Two Nodes Launching a Parallel Interactive Shell Through LSF Example 2. Four Processors on Two Specific NodesView the job Check the running job’s information $ bsub -Is -n4 -ext SLURMnodes=4 /bin/bash Submitting a Simple Job Script with LSF Check the finished job’s informationShow the environment Display the script Submitting an Interactive Job with LSF Show the job allocationShow the Slurm job ID $ bsub -n8 -Ip /bin/sh Submitting an HP-MPI Job with LSF View the running job View the finished job$ bjobs -l Using a Resource Requirements String in an LSF Command $ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjobPage Glossary Extensible firmware interface External network nodeFairshare First come first served Image server Integrated Lights OutInterconnect Internet address Network Information Services LSF master hostManagement Processor Master host Root Administration Switch Parallel applicationResource manager role Role Symmetric multiprocessing Glossary-6 Index Index-1 Gdb, 4-1 GNU Index-2 Modulefile Index-3 Resource manager, 7-1role Serial applications Index-4