Contents | HP XC System 2.x Software specs

		Contents
About This Document
1 Overview of the User Environment
1.1	System Architecture	. ... ... .. 1-1
1.1.1	Operating System	. ... ... .. 1-1
1.1.2	Node Specialization	. ... ... .. 1-1
1.1.3	Storage and I/O	. ... ... .. 1-2
1.1.4	File System	. ... ... .. 1-2
1.1.5	System Interconnect Network	. ... ... .. 1-3
1.1.6	Network Address Translation (NAT)	. ... ... .. 1-4
1.2	User Environment	. ... ... .. 1-4
1.2.1	LVS	. ... ... .. 1-4
1.2.2	Modules	. ... ... .. 1-4
1.2.3	Commands	. ... ... .. 1-4
1.2.3.1	Linux Commands	. ... ... .. 1-5
1.2.3.2	LSF Commands	. ... ... .. 1-5
1.2.3.3	SLURM Commands	. ... ... .. 1-5
1.2.3.4	HP-MPI Commands	. ... ... .. 1-5
1.2.3.5	Modules Commands	. ... ... .. 1-5
1.3	Application Development Environment	. ... ... .. 1-5
1.3.1	Parallel Applications	. ... ... .. 1-5
1.3.2	Serial Applications	. ... ... .. 1-5
1.4	Run-Time Environment	. ... ... .. 1-6
1.4.1	SLURM	. ... ... .. 1-6
1.4.2	Load Sharing Facility (LSF-HPC)	. ... ... .. 1-6
1.4.3	How LSF-HPC and SLURM Interact	. ... ... .. 1-6
1.4.4	HP-MPI	. ... ... .. 1-7
1.5	Components, Tools, Compilers, Libraries, and Debuggers	. ... ... .. 1-7
2 Using the System
2.1	Logging in to the System	. ... ... .. 2-1
2.1.1	LVS Login Routing	. ... ... .. 2-1
2.1.2	Using ssh to Log In	. ... ... .. 2-1
2.2	Configuring Your Environment with Modulefiles	. ... ... .. 2-1
2.2.1	Notes on Modulefiles	. ... ... .. 2-2
2.2.2	Supplied Modulefiles	. ... ... .. 2-3
2.2.3	Modulefiles Automatically Loaded on the System	. ... ... .. 2-4
2.2.4	Viewing Available Modulefiles	. ... ... .. 2-4
2.2.5	Viewing Loaded Modulefiles	. ... ... .. 2-4
2.2.6	Loading a Modulefile	. ... ... .. 2-4
2.2.6.1	Loading a Modulefile for the Current Session	. ... ... .. 2-5
2.2.6.2	Automatically Loading a Modulefile at Login	. ... ... .. 2-5
2.2.7	Unloading a Modulefile	. ... ... .. 2-5
2.2.8	Modulefile Conflicts	. ... ... .. 2-5
2.2.9	Creating a Modulefile	. ... ... .. 2-6
2.2.10	Viewing Modulefile-Specific Help	. ... ... .. 2-6

Contents iii

Image 3

HP XC System 2.x Software manual Contents

Contents

Hewlett-Packard Company Palo Alto, California June Product VersionPage Contents 5.4 Developing Applications Tuning Applications Using SlurmDebugging Applications Using HP-MPI Using LSF 6.4 Using HP Mlib Advanced Topics ExamplesGlossary Index Examples Tables FiguresPage About This Document Intended AudienceDocument Organization HP XC Program Development Environment Linux Administration HandbookHP XC Information QuickSpecs for HP XC System Software HP Mathematical Library For More InformationSupplementary Information HP Message Passing Interface Http//sourceforge.net/projects/modules ManpagesHttp//supermon.sourceforge.net Http//systemimager.org Related Information Http//linuxvirtualserver.orgHttp//www-unix.mcs.anl.gov/mpi # cd /opt/hptc/config/sbin Typographical ConventionsBold text Ctrl/x HP Encourages Your CommentsPage Node Specialization Overview of the User EnvironmentSystem Architecture Operating System Local Storage Storage and I/OFile System SAN Storage File System Layout System Interconnect Network 1 LVS CommandsUser Environment Network Address Translation NAT Application Development Environment Parallel ApplicationsSerial Applications How LSF-HPC and Slurm Interact Run-Time EnvironmentSlurm Load Sharing Facility LSF-HPC HP-MPI Components, Tools, Compilers, Libraries, and Debuggers 8Overview of the User Environment Using ssh to Log Using the SystemConfiguring Your Environment with Modulefiles LVS Login Routing 2Using the System Supplied Modulefiles Supplied ModulefilesModulefile Sets the HP XC User Environment Loading a Modulefile Modulefiles Automatically Loaded on the SystemViewing Available Modulefiles Viewing Loaded Modulefiles Loading a Modulefile for the Current Session Automatically Loading a Modulefile at LoginUnloading a Modulefile Modulefile Conflicts $ module help totalview Creating a ModulefileViewing Modulefile-Specific Help $ module load modules $ man modulefile Getting Information About Resources Launching and Managing Jobs Quick StartIntroduction Getting Information About Queues Example 2-1 Submitting a Serial Job Getting Information About the System’s PartitionsLaunching Jobs Submitting a Serial Job $ bsub -n4 -I srun hostname Using Slurm Options with the LSF External SchedulerSubmitting a Non-MPI Parallel Job Example 2-2 Submitting a Non-MPI Parallel Job Submitting an MPI Job Example 2-4 Running an MPI Job with LSF$ bsub -n4 Mpirun -srun ./helloworld Submitting a Batch Job or Job Script Example 2-6 Submitting a Job Script$ bsub -I -n4 myjobscript.sh Getting System Help and Information Performing Other Common User Tasks $ man -k keyword $ man sinfoPage Overview Developing Applications PGI Compilers Using CompilersStandard Linux Compilers Intel Compilers Developing Serial Applications Setting Debugging OptionsChecking Nodes and Partitions Before Running Jobs Interrupting a Job Building Serial Applications Using Mlib in Serial ApplicationsDeveloping Parallel Applications Serial Application Build Environment OpenMP Parallel Application Build EnvironmentModulefiles HP-MPI $ mpicc object1.o ... -pthread -o myapp.exe Quadrics ShmemMlib Math Library MPI Library GNU Parallel Make Intel Fortran and C/C++CompilersPGI Fortran and C/C++ Compilers GNU C and C++ Compilers Compiling and Linking HP-MPI Applications Reserved Symbols and NamesBuilding Parallel Applications Compiling and Linking Non-MPI Applications Developing Libraries Designing Libraries for XC4000$ mpicc -c -g foo.c Example 3-2 Recommended Directory Structure Using the GNU Parallel Make CapabilityAdvanced Topics Example 3-1 Directory Structure $ cd subdir srun -n1 -N1 $MAKE -j4 $ cd subdir srun $MAKE Example Procedure $ make PREFIX=’srun -n1 -N1’ MAKEJ=’-j4’ $ make PREFIX=’srun -n1 -N1 MAKEJ=’-j4’ Private File View Local Disks on Compute Nodes3 I/O Performance Considerations Shared File View Communication Between Nodes Page TotalView Debugging ApplicationsDebugging Serial Applications Debugging Parallel Applications Setting Up TotalView Debugging with TotalViewSSH and TotalView $ bsub -nx-ext SLURMnodes=x \ -Is /usr/bin/xterm Using TotalView with SlurmUsing TotalView with LSF-HPC $ srun -Nx-A $ mpirun -tv -srun application $ totalview Starting TotalView for the First Time TotalView Preferences Window Preferences window, click on the Launch Strings tab Debugging Applications Debugging an Application $ mpicc -g -o Psimple simple.c -lm$ mpirun -tv -srun -n2 ./Psimple TotalView Process Window Example $ mpirun -srun -n2 Psimple Debugging Running Applications $ scancel --user username Exiting TotalView$ squeue Page Example Tuning ApplicationsUsing the Intel Trace Collector/Analyzer Building a Program Intel Trace Collector and HP-MPI Running a Program Intel Trace Collector and HP-MPI Visualizing Data Intel Trace Analyzer and HP-MPIExample Running the vtjacobic Example Program Introduction Using SlurmSlurm Commands Command Function Example 6-1 Simple Launch of a Serial Program Accessing the Slurm ManpagesLaunching Jobs with the srun Command Srun Roles and Modes Srun Modes Srun Roles Allocate Srun Signal HandlingSrun Run-Mode Options Batch Srun Resource-Allocation Options Nthreads --threads=nthreads Cpt --cpus-per-task=cptPart --partition=part Minutes --time=minutes Srun Control Options Srun I/O Options Label Mode --output=modeMode --input=mode Mode --error=mode Clist --constraint=clist Srun Constraint Options Vmem=size Contiguous=yesnoMem=size Mincpus=n Srun Environment Variables Monitoring Jobs with the squeue CommandUsing srun with HP-MPI Using srun with LSF Getting System Information with the sinfo Command Killing Jobs with the scancel Command $ sinfo -R Fault ToleranceJob Accounting Security Using LSF Introduction to LSF in the HP XC EnvironmentOverview of LSF Nodelist= list-of-nodes Exclude= list-of-nodes Topology Support $ bsub -n 10 -ext SLURMnodes=10exclude=n16 srun myapp $ bsub -n 10 -ext SLURMnodes=10 srun myapp How LSF and Slurm Launch and Manage a Job $ bqueues -l normal grep Jobstarter$ bsub -Is hostname $ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript How LSF-HPC and Slurm Launch and Manage a Job Differences Between LSF on HP XC and Standard LSF Job Startup and Job Control Determining Execution HostDetermining Available System Resources Getting Status of LSF Getting Host Load Information Getting Information About LSF-HPC Execution Host Node $ sinfo -p lsf Submitting JobsChecking LSF System Queues Getting Information About the lsf Partition Bsub bsuboptions jobname joboptions Summary of the LSF bsub Command Format Slurm Arguments Function LSF-SLURM External Scheduler Starting on lsfhost.localdomain n6 Example 7-5 Submitting an Interactive Serial Job Submitting a Serial JobSubmitting a Job in Parallel Submitting an HP-MPI Job Submitting a Batch Job or Job Script Example 7-6 Submitting an HP-MPI Job$ bsub -n4 -I mpirun -srun ./helloworld $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh ExamplesExample 7-8 Submitting a Batch Job Script $ bsub -n4 -I ./myscript.sh Submitting a Job from a Non-HP XC Host $ bsub -n4 -I ./myscript.sh -n8 -OType=SLINUX64 $ bsub -R type=SLINUX64 -n4 -I srun hostname Getting Information About JobsGetting Job Allocation Information Job Allocation Information for a Running Job Example 7-13 Using the bjobs Command Short Output Checking Status of a JobJob Allocation Information for a Finished Job Viewing a Job’s Historical Information Example 7-14 Using the bjobs Command Long OutputExample 7-15 Using the bhist Command Short Output Output Provided by the bhist Command Example 7-16 Using the bhist Command Long Output Working Interactively Within an LSF-HPC AllocationSubmitting an Interactive Job to Launch the xterm Program Example 7-20 View Job Details in LSF Example 7-17 View Your EnvironmentExample 7-18 View Your Allocation in Slurm Example 7-19 View Your Running Job in LSF $ hostname $ srun hostname $ srun -n2 hostname Submitting an Interactive Job to Launch a ShellExample 7-21 Running Jobs from an xterm Window Example 7-22 Submitting an Interactive Shell Program Srun Option Description LSF Equivalent LSF Equivalents of Slurm srun OptionsLSF Equivalents of Slurm srun Options $ srun hostname n1 $ exit 24Using LSF Control the parallel job Page Using HP-MPI Building and Running an Example Application Setting Environment VariablesHP-MPI Directory Structure Compiling and Running Applications $ $MPIROOT/bin/mpirun -srun -n4 helloworld Example Application helloworldBuilding and Running helloworld $ mpicc -o helloworld $MPIROOT/help/helloworld.c HP-MPI options allowed with -srun Launching MPI Jobs $ mpirun -srun -n4 -N2 -O -m cyclic ./a.out host1 rank1 Creating Subshells and Launching JobstepsSystem Interconnect Selection $ mpirun -srun -n6 -O -N2 -m cyclic ./a.out host1 rank1 $ mpirun -subnet 192.168.1.1 -prot -srun -n4 ./a.out Using LSF and HP-MPI Example 8-5 Allocating 12 Processors on 6 Nodes $ /usr/sbin/ifconfig -aSystem Interconnect Support MPI Versioning 32-Bit Builds on XC4000 Truncated MessagesAllowing Windows to Use Exclusive Locks $ mpirun -TCP -srun -N8 ./a.out Mpirun Command Options $MPIROOT/bin/mpirun -v -prot -np 2 /path/to/program.x Environment Variables Export MPIMAXWINDOW=10 Export MPIPHYSICALMEMORY=1048576Export MPIPINPERCENTAGE=30 Export MPIPAGEALIGNMEM=1 $MPIROOT/bin/mpirun.mpich -np 2 ./prog.x Mpich Object CompatibilityExport MPIUSELIBELAN=0 Export MPIUSELIBELANUSE=5 Category Manpages Description HP-MPI Documentation and ManpagesAt http//docs.hp.com HP-MPI Manpage Categories Additional Information, Known Problems, and Work-arounds Version 7 Fortran Compiler Using HP MlibIntel Compiler Notes Version 8 Fortran Compiler Library Support HP Mlib for the HP XC6000 PlatformMlib and Module Files Platform Support Using Intel Compilers with HP Mlib Modulefiles and MlibCompiling and Linking Linking SuperLUDIST HP Mlib for the HP XC4000 PlatformLicensing Mlib Manpages $ pgcc options file 6Using HP Mlib Determining IP Address of Your Local Machine Advanced TopicsEnabling Remote Execution with OpenSSH Running an X Terminal Session from a Remote Node $ host mymachine Running an X terminal Session Using SlurmLogging in to HP XC System $ hostname Running an X terminal Session Using LSF $ bsub -n4 -Ip srun -n1 xterm -display$ bjobs Page Examples Building and Running a Serial ApplicationLaunching a Serial Interactive Shell Through LSF Example 1. Two Processors on Any Two Nodes Running LSF Jobs with a Slurm Allocation Request Launching a Parallel Interactive Shell Through LSF Example 2. Four Processors on Two Specific NodesView the job $ bsub -Is -n4 -ext SLURMnodes=4 /bin/bash Check the running job’s information Display the script Submitting a Simple Job Script with LSFCheck the finished job’s information Show the environment $ bsub -n8 -Ip /bin/sh Submitting an Interactive Job with LSFShow the job allocation Show the Slurm job ID Submitting an HP-MPI Job with LSF View the running job View the finished job$ bjobs -l $ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob Using a Resource Requirements String in an LSF CommandPage Glossary First come first served Extensible firmware interfaceExternal network node Fairshare Internet address Image serverIntegrated Lights Out Interconnect Master host Network Information ServicesLSF master host Management Processor Role Root Administration SwitchParallel application Resource manager role Glossary-6 Symmetric multiprocessing Index-1 Index Index-2 Gdb, 4-1 GNU Index-3 Modulefile Index-4 Resource manager, 7-1role Serial applications