HP XC System 2.x Software specs

date and time stamp: Submitted from host <lsfhost.localdomain>, to Queue <normal>,CWD <$HOME>,

6 Processors Requested;

date and time stamp: Dispatched to 6 Hosts/Processors <6*lsfhost.localdomain>;

date and time stamp:			slurm_id=22;ncpus=6;slurm_alloc=n[13-15];
date and time stamp:			Starting (Pid 11216);
date and time stamp:			Done	successfully.
The CPU	time used is		0.0 seconds;
date and time stamp: Post				job process done successfully;
Summary	of time in seconds			spent in various states by date and time
PEND	PSUSP	RUN	USUSP	SSUSP	UNKWN	TOTAL
11	0	7	0	0	0	18

A.8 Using a Resource Requirements String in an LSF Command

The following examples show how to use a resource requirement string in an LSF command.

•Run myjob on an Alpha/AXP host or an HP XC host if one is available, but not both:

$ bsub -n 8 -R "ALPHA5 SLINUX64" \ -ext "SLURM[nodes=4-4]" myjob

If myjob runs on an HP XC host, the SLURM[nodes=4-4]allocation option is applied. If it runs on an Alpha/AXP host, the SLURM option is ignored.

•Run myjob on any host type, and apply allocation options appropriately:

$ bsub-n 8 -R "type==any" \

-ext "SLURM[nodes=4-4];RMS[ptile=2]" myjob

If myjob runs on an HP XC host, the SLURM[nodes=4-4]option is applied. If myjob runs on an HP AlphaServer SC host, the RMS ptile option is applied. If it runs on any other host type, the SLURM and RMS options are ignored.

Examples A-9

Image 143

HP XC System 2.x Software manual Using a Resource Requirements String in an LSF Command

Contents

Hewlett-Packard Company Palo Alto, California June Product VersionPage Contents 5.4 Developing Applications Debugging Applications Tuning ApplicationsUsing Slurm Using HP-MPI Using LSF 6.4 Using HP Mlib Glossary Index Examples Advanced TopicsExamples Tables FiguresPage Document Organization About This DocumentIntended Audience HP XC Program Development Environment Linux Administration HandbookHP XC Information QuickSpecs for HP XC System Software HP Mathematical Library For More InformationSupplementary Information HP Message Passing Interface Http//sourceforge.net/projects/modules ManpagesHttp//supermon.sourceforge.net Http//systemimager.org Http//www-unix.mcs.anl.gov/mpi Related InformationHttp//linuxvirtualserver.org Bold text # cd /opt/hptc/config/sbinTypographical Conventions Ctrl/x HP Encourages Your CommentsPage Node Specialization Overview of the User EnvironmentSystem Architecture Operating System Local Storage Storage and I/OFile System SAN Storage File System Layout System Interconnect Network 1 LVS CommandsUser Environment Network Address Translation NAT Serial Applications Application Development EnvironmentParallel Applications How LSF-HPC and Slurm Interact Run-Time EnvironmentSlurm Load Sharing Facility LSF-HPC HP-MPI Components, Tools, Compilers, Libraries, and Debuggers 8Overview of the User Environment Using ssh to Log Using the SystemConfiguring Your Environment with Modulefiles LVS Login Routing 2Using the System Modulefile Sets the HP XC User Environment Supplied ModulefilesSupplied Modulefiles Loading a Modulefile Modulefiles Automatically Loaded on the SystemViewing Available Modulefiles Viewing Loaded Modulefiles Loading a Modulefile for the Current Session Automatically Loading a Modulefile at LoginUnloading a Modulefile Modulefile Conflicts $ module help totalview Creating a ModulefileViewing Modulefile-Specific Help $ module load modules $ man modulefile Getting Information About Resources Launching and Managing Jobs Quick StartIntroduction Getting Information About Queues Example 2-1 Submitting a Serial Job Getting Information About the System’s PartitionsLaunching Jobs Submitting a Serial Job $ bsub -n4 -I srun hostname Using Slurm Options with the LSF External SchedulerSubmitting a Non-MPI Parallel Job Example 2-2 Submitting a Non-MPI Parallel Job $ bsub -n4 Mpirun -srun ./helloworld Submitting an MPI JobExample 2-4 Running an MPI Job with LSF $ bsub -I -n4 myjobscript.sh Submitting a Batch Job or Job ScriptExample 2-6 Submitting a Job Script Getting System Help and Information Performing Other Common User Tasks $ man -k keyword $ man sinfoPage Overview Developing Applications PGI Compilers Using CompilersStandard Linux Compilers Intel Compilers Developing Serial Applications Setting Debugging OptionsChecking Nodes and Partitions Before Running Jobs Interrupting a Job Building Serial Applications Using Mlib in Serial ApplicationsDeveloping Parallel Applications Serial Application Build Environment OpenMP Parallel Application Build EnvironmentModulefiles HP-MPI $ mpicc object1.o ... -pthread -o myapp.exe Quadrics ShmemMlib Math Library MPI Library GNU Parallel Make Intel Fortran and C/C++CompilersPGI Fortran and C/C++ Compilers GNU C and C++ Compilers Compiling and Linking HP-MPI Applications Reserved Symbols and NamesBuilding Parallel Applications Compiling and Linking Non-MPI Applications $ mpicc -c -g foo.c Developing LibrariesDesigning Libraries for XC4000 Example 3-2 Recommended Directory Structure Using the GNU Parallel Make CapabilityAdvanced Topics Example 3-1 Directory Structure $ cd subdir srun -n1 -N1 $MAKE -j4 $ cd subdir srun $MAKE Example Procedure $ make PREFIX=’srun -n1 -N1’ MAKEJ=’-j4’ $ make PREFIX=’srun -n1 -N1 MAKEJ=’-j4’ Private File View Local Disks on Compute Nodes3 I/O Performance Considerations Shared File View Communication Between Nodes Page TotalView Debugging ApplicationsDebugging Serial Applications Debugging Parallel Applications SSH and TotalView Setting Up TotalViewDebugging with TotalView $ bsub -nx-ext SLURMnodes=x \ -Is /usr/bin/xterm Using TotalView with SlurmUsing TotalView with LSF-HPC $ srun -Nx-A $ mpirun -tv -srun application $ totalview Starting TotalView for the First Time TotalView Preferences Window Preferences window, click on the Launch Strings tab Debugging Applications $ mpirun -tv -srun -n2 ./Psimple Debugging an Application$ mpicc -g -o Psimple simple.c -lm TotalView Process Window Example $ mpirun -srun -n2 Psimple Debugging Running Applications $ squeue $ scancel --user usernameExiting TotalView Page Example Tuning ApplicationsUsing the Intel Trace Collector/Analyzer Building a Program Intel Trace Collector and HP-MPI Example Running the vtjacobic Example Program Running a Program Intel Trace Collector and HP-MPIVisualizing Data Intel Trace Analyzer and HP-MPI Introduction Using SlurmSlurm Commands Command Function Example 6-1 Simple Launch of a Serial Program Accessing the Slurm ManpagesLaunching Jobs with the srun Command Srun Roles and Modes Srun Modes Srun Roles Allocate Srun Signal HandlingSrun Run-Mode Options Batch Srun Resource-Allocation Options Nthreads --threads=nthreads Cpt --cpus-per-task=cptPart --partition=part Minutes --time=minutes Srun Control Options Srun I/O Options Label Mode --output=modeMode --input=mode Mode --error=mode Clist --constraint=clist Srun Constraint Options Vmem=size Contiguous=yesnoMem=size Mincpus=n Srun Environment Variables Monitoring Jobs with the squeue CommandUsing srun with HP-MPI Using srun with LSF Getting System Information with the sinfo Command Killing Jobs with the scancel Command $ sinfo -R Fault ToleranceJob Accounting Security Overview of LSF Using LSFIntroduction to LSF in the HP XC Environment Nodelist= list-of-nodes Exclude= list-of-nodes Topology Support $ bsub -n 10 -ext SLURMnodes=10exclude=n16 srun myapp $ bsub -n 10 -ext SLURMnodes=10 srun myapp $ bsub -Is hostname How LSF and Slurm Launch and Manage a Job$ bqueues -l normal grep Jobstarter $ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript How LSF-HPC and Slurm Launch and Manage a Job Differences Between LSF on HP XC and Standard LSF Job Startup and Job Control Determining Execution HostDetermining Available System Resources Getting Status of LSF Getting Host Load Information Getting Information About LSF-HPC Execution Host Node $ sinfo -p lsf Submitting JobsChecking LSF System Queues Getting Information About the lsf Partition Bsub bsuboptions jobname joboptions Summary of the LSF bsub Command Format Slurm Arguments Function LSF-SLURM External Scheduler Starting on lsfhost.localdomain n6 Example 7-5 Submitting an Interactive Serial Job Submitting a Serial JobSubmitting a Job in Parallel Submitting an HP-MPI Job $ bsub -n4 -I mpirun -srun ./helloworld Submitting a Batch Job or Job ScriptExample 7-6 Submitting an HP-MPI Job $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh ExamplesExample 7-8 Submitting a Batch Job Script $ bsub -n4 -I ./myscript.sh Type=SLINUX64 Submitting a Job from a Non-HP XC Host$ bsub -n4 -I ./myscript.sh -n8 -O $ bsub -R type=SLINUX64 -n4 -I srun hostname Getting Information About JobsGetting Job Allocation Information Job Allocation Information for a Running Job Job Allocation Information for a Finished Job Example 7-13 Using the bjobs Command Short OutputChecking Status of a Job Viewing a Job’s Historical Information Example 7-14 Using the bjobs Command Long OutputExample 7-15 Using the bhist Command Short Output Output Provided by the bhist Command Submitting an Interactive Job to Launch the xterm Program Example 7-16 Using the bhist Command Long OutputWorking Interactively Within an LSF-HPC Allocation Example 7-20 View Job Details in LSF Example 7-17 View Your EnvironmentExample 7-18 View Your Allocation in Slurm Example 7-19 View Your Running Job in LSF $ hostname $ srun hostname $ srun -n2 hostname Submitting an Interactive Job to Launch a ShellExample 7-21 Running Jobs from an xterm Window Example 7-22 Submitting an Interactive Shell Program Srun Option Description LSF Equivalent LSF Equivalents of Slurm srun OptionsLSF Equivalents of Slurm srun Options $ srun hostname n1 $ exit 24Using LSF Control the parallel job Page Using HP-MPI Building and Running an Example Application Setting Environment VariablesHP-MPI Directory Structure Compiling and Running Applications $ $MPIROOT/bin/mpirun -srun -n4 helloworld Example Application helloworldBuilding and Running helloworld $ mpicc -o helloworld $MPIROOT/help/helloworld.c HP-MPI options allowed with -srun Launching MPI Jobs $ mpirun -srun -n4 -N2 -O -m cyclic ./a.out host1 rank1 Creating Subshells and Launching JobstepsSystem Interconnect Selection $ mpirun -srun -n6 -O -N2 -m cyclic ./a.out host1 rank1 $ mpirun -subnet 192.168.1.1 -prot -srun -n4 ./a.out Using LSF and HP-MPI Example 8-5 Allocating 12 Processors on 6 Nodes $ /usr/sbin/ifconfig -aSystem Interconnect Support MPI Versioning Allowing Windows to Use Exclusive Locks 32-Bit Builds on XC4000Truncated Messages $ mpirun -TCP -srun -N8 ./a.out Mpirun Command Options $MPIROOT/bin/mpirun -v -prot -np 2 /path/to/program.x Environment Variables Export MPIMAXWINDOW=10 Export MPIPHYSICALMEMORY=1048576Export MPIPINPERCENTAGE=30 Export MPIPAGEALIGNMEM=1 $MPIROOT/bin/mpirun.mpich -np 2 ./prog.x Mpich Object CompatibilityExport MPIUSELIBELAN=0 Export MPIUSELIBELANUSE=5 Category Manpages Description HP-MPI Documentation and ManpagesAt http//docs.hp.com HP-MPI Manpage Categories Additional Information, Known Problems, and Work-arounds Version 7 Fortran Compiler Using HP MlibIntel Compiler Notes Version 8 Fortran Compiler Library Support HP Mlib for the HP XC6000 PlatformMlib and Module Files Platform Support Compiling and Linking Using Intel Compilers with HP MlibModulefiles and Mlib Linking SuperLUDIST HP Mlib for the HP XC4000 PlatformLicensing Mlib Manpages $ pgcc options file 6Using HP Mlib Determining IP Address of Your Local Machine Advanced TopicsEnabling Remote Execution with OpenSSH Running an X Terminal Session from a Remote Node $ host mymachine Running an X terminal Session Using SlurmLogging in to HP XC System $ hostname $ bjobs Running an X terminal Session Using LSF$ bsub -n4 -Ip srun -n1 xterm -display Page Launching a Serial Interactive Shell Through LSF ExamplesBuilding and Running a Serial Application Example 1. Two Processors on Any Two Nodes Running LSF Jobs with a Slurm Allocation Request View the job Launching a Parallel Interactive Shell Through LSFExample 2. Four Processors on Two Specific Nodes $ bsub -Is -n4 -ext SLURMnodes=4 /bin/bash Check the running job’s information Display the script Submitting a Simple Job Script with LSFCheck the finished job’s information Show the environment $ bsub -n8 -Ip /bin/sh Submitting an Interactive Job with LSFShow the job allocation Show the Slurm job ID Submitting an HP-MPI Job with LSF $ bjobs -l View the running jobView the finished job $ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob Using a Resource Requirements String in an LSF CommandPage Glossary First come first served Extensible firmware interfaceExternal network node Fairshare Internet address Image serverIntegrated Lights Out Interconnect Master host Network Information ServicesLSF master host Management Processor Role Root Administration SwitchParallel application Resource manager role Glossary-6 Symmetric multiprocessing Index-1 Index Index-2 Gdb, 4-1 GNU Index-3 Modulefile Index-4 Resource manager, 7-1role Serial applications