HP XC System 2.x Software specification

© Copyright 2003–2005 Hewlett-Packard Development Company, L.P.

UNIX® is a registered trademark of The Open Group.

Linux® is a U.S. registered trademark of Linus Torvalds.

LSF, Platform Computing, and the LSF and Platform Computing logos are trademarks or registered trademarks of Platform Computing Corporation.

Intel®, the Intel logo, Itanium®, Xeon™, and Pentium® are trademarks or registered trademarks of Intel Corporation in the United States and other countries.

TotalView® is a registered trademark of Etnus, Inc.

Quadrics® is a registered trademark of Quadrics, Ltd.

Myrinet® and Myricom® are registered trademarks of Myricom, Inc.

Red Hat ® is a registered trademark of Red Hat Inc.

Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license.

The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Image 2

HP XC System 2.x Software manual

Contents

June Product Version Hewlett-Packard Company Palo Alto, CaliforniaPage Contents Developing Applications 5.4 Debugging Applications Tuning ApplicationsUsing Slurm Using LSF Using HP-MPI Using HP Mlib 6.4 Glossary Index Examples Advanced TopicsExamples Figures TablesPage Document Organization About This DocumentIntended Audience QuickSpecs for HP XC System Software Linux Administration HandbookHP XC Information HP XC Program Development Environment HP Message Passing Interface For More InformationSupplementary Information HP Mathematical Library Http//systemimager.org ManpagesHttp//supermon.sourceforge.net Http//sourceforge.net/projects/modules Http//www-unix.mcs.anl.gov/mpi Related InformationHttp//linuxvirtualserver.org Bold text # cd /opt/hptc/config/sbinTypographical Conventions HP Encourages Your Comments Ctrl/xPage Operating System Overview of the User EnvironmentSystem Architecture Node Specialization SAN Storage Storage and I/OFile System Local Storage System Interconnect Network File System Layout Network Address Translation NAT CommandsUser Environment 1 LVS Serial Applications Application Development EnvironmentParallel Applications Load Sharing Facility LSF-HPC Run-Time EnvironmentSlurm How LSF-HPC and Slurm Interact Components, Tools, Compilers, Libraries, and Debuggers HP-MPI 8Overview of the User Environment LVS Login Routing Using the SystemConfiguring Your Environment with Modulefiles Using ssh to Log 2Using the System Modulefile Sets the HP XC User Environment Supplied ModulefilesSupplied Modulefiles Viewing Loaded Modulefiles Modulefiles Automatically Loaded on the SystemViewing Available Modulefiles Loading a Modulefile Modulefile Conflicts Automatically Loading a Modulefile at LoginUnloading a Modulefile Loading a Modulefile for the Current Session $ module load modules $ man modulefile Creating a ModulefileViewing Modulefile-Specific Help $ module help totalview Getting Information About Queues Launching and Managing Jobs Quick StartIntroduction Getting Information About Resources Submitting a Serial Job Getting Information About the System’s PartitionsLaunching Jobs Example 2-1 Submitting a Serial Job Example 2-2 Submitting a Non-MPI Parallel Job Using Slurm Options with the LSF External SchedulerSubmitting a Non-MPI Parallel Job $ bsub -n4 -I srun hostname $ bsub -n4 Mpirun -srun ./helloworld Submitting an MPI JobExample 2-4 Running an MPI Job with LSF $ bsub -I -n4 myjobscript.sh Submitting a Batch Job or Job ScriptExample 2-6 Submitting a Job Script Performing Other Common User Tasks Getting System Help and Information $ man sinfo $ man -k keywordPage Developing Applications Overview Intel Compilers Using CompilersStandard Linux Compilers PGI Compilers Interrupting a Job Setting Debugging OptionsChecking Nodes and Partitions Before Running Jobs Developing Serial Applications Serial Application Build Environment Using Mlib in Serial ApplicationsDeveloping Parallel Applications Building Serial Applications HP-MPI Parallel Application Build EnvironmentModulefiles OpenMP MPI Library Quadrics ShmemMlib Math Library $ mpicc object1.o ... -pthread -o myapp.exe GNU C and C++ Compilers Intel Fortran and C/C++CompilersPGI Fortran and C/C++ Compilers GNU Parallel Make Compiling and Linking Non-MPI Applications Reserved Symbols and NamesBuilding Parallel Applications Compiling and Linking HP-MPI Applications $ mpicc -c -g foo.c Developing LibrariesDesigning Libraries for XC4000 Example 3-1 Directory Structure Using the GNU Parallel Make CapabilityAdvanced Topics Example 3-2 Recommended Directory Structure $ cd subdir srun $MAKE $ cd subdir srun -n1 -N1 $MAKE -j4 Example Procedure $ make PREFIX=’srun -n1 -N1 MAKEJ=’-j4’ $ make PREFIX=’srun -n1 -N1’ MAKEJ=’-j4’ Shared File View Local Disks on Compute Nodes3 I/O Performance Considerations Private File View Communication Between Nodes Page Debugging Parallel Applications Debugging ApplicationsDebugging Serial Applications TotalView SSH and TotalView Setting Up TotalViewDebugging with TotalView $ srun -Nx-A $ mpirun -tv -srun application Using TotalView with SlurmUsing TotalView with LSF-HPC $ bsub -nx-ext SLURMnodes=x \ -Is /usr/bin/xterm Starting TotalView for the First Time $ totalview TotalView Preferences Window Preferences window, click on the Launch Strings tab Debugging Applications $ mpirun -tv -srun -n2 ./Psimple Debugging an Application$ mpicc -g -o Psimple simple.c -lm TotalView Process Window Example Debugging Running Applications $ mpirun -srun -n2 Psimple $ squeue $ scancel --user usernameExiting TotalView Page Building a Program Intel Trace Collector and HP-MPI Tuning ApplicationsUsing the Intel Trace Collector/Analyzer Example Example Running the vtjacobic Example Program Running a Program Intel Trace Collector and HP-MPIVisualizing Data Intel Trace Analyzer and HP-MPI Command Function Using SlurmSlurm Commands Introduction Srun Roles and Modes Accessing the Slurm ManpagesLaunching Jobs with the srun Command Example 6-1 Simple Launch of a Serial Program Srun Roles Srun Modes Batch Srun Signal HandlingSrun Run-Mode Options Allocate Srun Resource-Allocation Options Minutes --time=minutes Cpt --cpus-per-task=cptPart --partition=part Nthreads --threads=nthreads Srun Control Options Srun I/O Options Mode --error=mode Mode --output=modeMode --input=mode Label Srun Constraint Options Clist --constraint=clist Mincpus=n Contiguous=yesnoMem=size Vmem=size Using srun with LSF Monitoring Jobs with the squeue CommandUsing srun with HP-MPI Srun Environment Variables Killing Jobs with the scancel Command Getting System Information with the sinfo Command Security Fault ToleranceJob Accounting $ sinfo -R Overview of LSF Using LSFIntroduction to LSF in the HP XC Environment Topology Support Nodelist= list-of-nodes Exclude= list-of-nodes $ bsub -n 10 -ext SLURMnodes=10 srun myapp $ bsub -n 10 -ext SLURMnodes=10exclude=n16 srun myapp $ bsub -Is hostname How LSF and Slurm Launch and Manage a Job$ bqueues -l normal grep Jobstarter How LSF-HPC and Slurm Launch and Manage a Job $ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript Differences Between LSF on HP XC and Standard LSF Getting Status of LSF Determining Execution HostDetermining Available System Resources Job Startup and Job Control Getting Information About LSF-HPC Execution Host Node Getting Host Load Information Getting Information About the lsf Partition Submitting JobsChecking LSF System Queues $ sinfo -p lsf Summary of the LSF bsub Command Format Bsub bsuboptions jobname joboptions LSF-SLURM External Scheduler Slurm Arguments Function Starting on lsfhost.localdomain n6 Submitting an HP-MPI Job Submitting a Serial JobSubmitting a Job in Parallel Example 7-5 Submitting an Interactive Serial Job $ bsub -n4 -I mpirun -srun ./helloworld Submitting a Batch Job or Job ScriptExample 7-6 Submitting an HP-MPI Job $ bsub -n4 -I ./myscript.sh ExamplesExample 7-8 Submitting a Batch Job Script $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh Type=SLINUX64 Submitting a Job from a Non-HP XC Host$ bsub -n4 -I ./myscript.sh -n8 -O Job Allocation Information for a Running Job Getting Information About JobsGetting Job Allocation Information $ bsub -R type=SLINUX64 -n4 -I srun hostname Job Allocation Information for a Finished Job Example 7-13 Using the bjobs Command Short OutputChecking Status of a Job Output Provided by the bhist Command Example 7-14 Using the bjobs Command Long OutputExample 7-15 Using the bhist Command Short Output Viewing a Job’s Historical Information Submitting an Interactive Job to Launch the xterm Program Example 7-16 Using the bhist Command Long OutputWorking Interactively Within an LSF-HPC Allocation Example 7-19 View Your Running Job in LSF Example 7-17 View Your EnvironmentExample 7-18 View Your Allocation in Slurm Example 7-20 View Job Details in LSF Example 7-22 Submitting an Interactive Shell Program Submitting an Interactive Job to Launch a ShellExample 7-21 Running Jobs from an xterm Window $ hostname $ srun hostname $ srun -n2 hostname $ srun hostname n1 $ exit LSF Equivalents of Slurm srun OptionsLSF Equivalents of Slurm srun Options Srun Option Description LSF Equivalent 24Using LSF Control the parallel job Page Using HP-MPI Compiling and Running Applications Setting Environment VariablesHP-MPI Directory Structure Building and Running an Example Application $ mpicc -o helloworld $MPIROOT/help/helloworld.c Example Application helloworldBuilding and Running helloworld $ $MPIROOT/bin/mpirun -srun -n4 helloworld Launching MPI Jobs HP-MPI options allowed with -srun $ mpirun -srun -n6 -O -N2 -m cyclic ./a.out host1 rank1 Creating Subshells and Launching JobstepsSystem Interconnect Selection $ mpirun -srun -n4 -N2 -O -m cyclic ./a.out host1 rank1 Using LSF and HP-MPI $ mpirun -subnet 192.168.1.1 -prot -srun -n4 ./a.out MPI Versioning $ /usr/sbin/ifconfig -aSystem Interconnect Support Example 8-5 Allocating 12 Processors on 6 Nodes Allowing Windows to Use Exclusive Locks 32-Bit Builds on XC4000Truncated Messages Mpirun Command Options $ mpirun -TCP -srun -N8 ./a.out Environment Variables $MPIROOT/bin/mpirun -v -prot -np 2 /path/to/program.x Export MPIPAGEALIGNMEM=1 Export MPIPHYSICALMEMORY=1048576Export MPIPINPERCENTAGE=30 Export MPIMAXWINDOW=10 Export MPIUSELIBELANUSE=5 Mpich Object CompatibilityExport MPIUSELIBELAN=0 $MPIROOT/bin/mpirun.mpich -np 2 ./prog.x HP-MPI Manpage Categories HP-MPI Documentation and ManpagesAt http//docs.hp.com Category Manpages Description Additional Information, Known Problems, and Work-arounds Version 8 Fortran Compiler Using HP MlibIntel Compiler Notes Version 7 Fortran Compiler Platform Support HP Mlib for the HP XC6000 PlatformMlib and Module Files Library Support Compiling and Linking Using Intel Compilers with HP MlibModulefiles and Mlib Mlib Manpages HP Mlib for the HP XC4000 PlatformLicensing Linking SuperLUDIST $ pgcc options file 6Using HP Mlib Running an X Terminal Session from a Remote Node Advanced TopicsEnabling Remote Execution with OpenSSH Determining IP Address of Your Local Machine $ hostname Running an X terminal Session Using SlurmLogging in to HP XC System $ host mymachine $ bjobs Running an X terminal Session Using LSF$ bsub -n4 -Ip srun -n1 xterm -display Page Launching a Serial Interactive Shell Through LSF ExamplesBuilding and Running a Serial Application Running LSF Jobs with a Slurm Allocation Request Example 1. Two Processors on Any Two Nodes View the job Launching a Parallel Interactive Shell Through LSFExample 2. Four Processors on Two Specific Nodes Check the running job’s information $ bsub -Is -n4 -ext SLURMnodes=4 /bin/bash Show the environment Submitting a Simple Job Script with LSFCheck the finished job’s information Display the script Show the Slurm job ID Submitting an Interactive Job with LSFShow the job allocation $ bsub -n8 -Ip /bin/sh Submitting an HP-MPI Job with LSF $ bjobs -l View the running jobView the finished job Using a Resource Requirements String in an LSF Command $ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjobPage Glossary Fairshare Extensible firmware interfaceExternal network node First come first served Interconnect Image serverIntegrated Lights Out Internet address Management Processor Network Information ServicesLSF master host Master host Resource manager role Root Administration SwitchParallel application Role Symmetric multiprocessing Glossary-6 Index Index-1 Gdb, 4-1 GNU Index-2 Modulefile Index-3 Resource manager, 7-1role Serial applications Index-4