View the running job, $ bjobs -l | HP XC System 2.x Software

Show the environment:

$ lsid

Platform LSF HPC 6.0 for SLURM, Sep 23 2004 Copyright 1992-2004 Platform Computing Corporation

My cluster name is penguin

My master name is lsfhost.localdomain

$ sinfo
PARTITION	AVAIL	TIMELIMIT NODES			STATE	NODELIST
lsf	up	infinite	4		alloc	n[13-16]
$ lshosts
HOST_NAME	type	model	cpuf ncpus maxmem maxswp				server	RESOURCES
lsfhost.loc	SLINUX6	DEFAULT 1.0		8	1M	-	Yes	(slurm)
$ bhosts
HOST_NAME		STATUS	JL/U	MAX NJOBS RUN SSUSP USUSP RSV
lsfhost.localdomai		ok	-	8	0	0	0	0	0

Run the job:

$ bsub -I-n6-ext "SLURM[nodes=3]" mpirun -srun /usr/share/helloJob <1009> is submitted to default queue <normal>.

<<Waiting for dispatch ...>> <<Starting on lsfhost.localdomain>> I’m process 0! from ( n13 pid 27222)

Greetings from process 1! from ( n13 pid 27223) Greetings from process 2! from ( n14 pid 14011) Greetings from process 3! from ( n14 pid 14012) Greetings from process 4! from ( n15 pid 18227) Greetings from process 5! from ( n15 pid 18228) mpirun exits with status: 0

View the running job:

$ bjobs -l 1009

Job <1009>, User <smith>, Project <default>, Status <DONE>, Queue <normal>, Interactive mode, Extsched <SLURM[nodes=3]>,

Command </opt/hpmpi/bin/mpirun -srun /usr/share/hello>

date and time stamp: Submitted from host <lsfhost.localdomain>, CWD <$HOME>, 6 Processors Requested;

date and time stamp: Started on 6 Hosts/Processors <6*lsfhost.localdomain>;

date and time stamp: slurm_id=22;ncpus=6;slurm_alloc=n[13-15];

date and time stamp: Done successfully. The CPU time used is 0.0 seconds.

SCHEDULING PARAMETERS:
	r15s r1m r15m			ut	pg	io	ls	it tmp swp mem
loadSched	-	-	-	-	-	-	-	-	-	-	-
loadStop	-	-	-	-	-	-	-	-	-	-	-

EXTERNAL	MESSAGES:
MSG_ID	FROM	POST_TIME	MESSAGE	ATTACHMENT
0	-	-	-	-
1	lsfadmin	date and time	SLURM[nodes=2]	N

View the finished job:

$ bhist -l 1009

Job <1009>, User <smith>, Project <default>,

Interactive mode, Extsched <SLURM[nodes=3]>,

Command </opt/hpmpi/bin/mpirun -srun /usr/share/hello>

A-8Examples

Image 142

HP XC System 2.x Software manual View the running job, View the finished job, $ bjobs -l

Contents

June Product Version Hewlett-Packard Company Palo Alto, CaliforniaPage Contents Developing Applications 5.4 Using Slurm Tuning ApplicationsDebugging Applications Using LSF Using HP-MPI Using HP Mlib 6.4 Examples Advanced TopicsGlossary Index Examples Figures TablesPage Intended Audience About This DocumentDocument Organization QuickSpecs for HP XC System Software Linux Administration HandbookHP XC Information HP XC Program Development Environment HP Message Passing Interface For More InformationSupplementary Information HP Mathematical Library Http//systemimager.org ManpagesHttp//supermon.sourceforge.net Http//sourceforge.net/projects/modules Http//linuxvirtualserver.org Related InformationHttp//www-unix.mcs.anl.gov/mpi Typographical Conventions # cd /opt/hptc/config/sbinBold text HP Encourages Your Comments Ctrl/xPage Operating System Overview of the User EnvironmentSystem Architecture Node Specialization SAN Storage Storage and I/OFile System Local Storage System Interconnect Network File System Layout Network Address Translation NAT CommandsUser Environment 1 LVS Parallel Applications Application Development EnvironmentSerial Applications Load Sharing Facility LSF-HPC Run-Time EnvironmentSlurm How LSF-HPC and Slurm Interact Components, Tools, Compilers, Libraries, and Debuggers HP-MPI 8Overview of the User Environment LVS Login Routing Using the SystemConfiguring Your Environment with Modulefiles Using ssh to Log 2Using the System Supplied Modulefiles Supplied ModulefilesModulefile Sets the HP XC User Environment Viewing Loaded Modulefiles Modulefiles Automatically Loaded on the SystemViewing Available Modulefiles Loading a Modulefile Modulefile Conflicts Automatically Loading a Modulefile at LoginUnloading a Modulefile Loading a Modulefile for the Current Session $ module load modules $ man modulefile Creating a ModulefileViewing Modulefile-Specific Help $ module help totalview Getting Information About Queues Launching and Managing Jobs Quick StartIntroduction Getting Information About Resources Submitting a Serial Job Getting Information About the System’s PartitionsLaunching Jobs Example 2-1 Submitting a Serial Job Example 2-2 Submitting a Non-MPI Parallel Job Using Slurm Options with the LSF External SchedulerSubmitting a Non-MPI Parallel Job $ bsub -n4 -I srun hostname Example 2-4 Running an MPI Job with LSF Submitting an MPI Job$ bsub -n4 Mpirun -srun ./helloworld Example 2-6 Submitting a Job Script Submitting a Batch Job or Job Script$ bsub -I -n4 myjobscript.sh Performing Other Common User Tasks Getting System Help and Information $ man sinfo $ man -k keywordPage Developing Applications Overview Intel Compilers Using CompilersStandard Linux Compilers PGI Compilers Interrupting a Job Setting Debugging OptionsChecking Nodes and Partitions Before Running Jobs Developing Serial Applications Serial Application Build Environment Using Mlib in Serial ApplicationsDeveloping Parallel Applications Building Serial Applications HP-MPI Parallel Application Build EnvironmentModulefiles OpenMP MPI Library Quadrics ShmemMlib Math Library $ mpicc object1.o ... -pthread -o myapp.exe GNU C and C++ Compilers Intel Fortran and C/C++CompilersPGI Fortran and C/C++ Compilers GNU Parallel Make Compiling and Linking Non-MPI Applications Reserved Symbols and NamesBuilding Parallel Applications Compiling and Linking HP-MPI Applications Designing Libraries for XC4000 Developing Libraries$ mpicc -c -g foo.c Example 3-1 Directory Structure Using the GNU Parallel Make CapabilityAdvanced Topics Example 3-2 Recommended Directory Structure $ cd subdir srun $MAKE $ cd subdir srun -n1 -N1 $MAKE -j4 Example Procedure $ make PREFIX=’srun -n1 -N1 MAKEJ=’-j4’ $ make PREFIX=’srun -n1 -N1’ MAKEJ=’-j4’ Shared File View Local Disks on Compute Nodes3 I/O Performance Considerations Private File View Communication Between Nodes Page Debugging Parallel Applications Debugging ApplicationsDebugging Serial Applications TotalView Debugging with TotalView Setting Up TotalViewSSH and TotalView $ srun -Nx-A $ mpirun -tv -srun application Using TotalView with SlurmUsing TotalView with LSF-HPC $ bsub -nx-ext SLURMnodes=x \ -Is /usr/bin/xterm Starting TotalView for the First Time $ totalview TotalView Preferences Window Preferences window, click on the Launch Strings tab Debugging Applications $ mpicc -g -o Psimple simple.c -lm Debugging an Application$ mpirun -tv -srun -n2 ./Psimple TotalView Process Window Example Debugging Running Applications $ mpirun -srun -n2 Psimple Exiting TotalView $ scancel --user username$ squeue Page Building a Program Intel Trace Collector and HP-MPI Tuning ApplicationsUsing the Intel Trace Collector/Analyzer Example Visualizing Data Intel Trace Analyzer and HP-MPI Running a Program Intel Trace Collector and HP-MPIExample Running the vtjacobic Example Program Command Function Using SlurmSlurm Commands Introduction Srun Roles and Modes Accessing the Slurm ManpagesLaunching Jobs with the srun Command Example 6-1 Simple Launch of a Serial Program Srun Roles Srun Modes Batch Srun Signal HandlingSrun Run-Mode Options Allocate Srun Resource-Allocation Options Minutes --time=minutes Cpt --cpus-per-task=cptPart --partition=part Nthreads --threads=nthreads Srun Control Options Srun I/O Options Mode --error=mode Mode --output=modeMode --input=mode Label Srun Constraint Options Clist --constraint=clist Mincpus=n Contiguous=yesnoMem=size Vmem=size Using srun with LSF Monitoring Jobs with the squeue CommandUsing srun with HP-MPI Srun Environment Variables Killing Jobs with the scancel Command Getting System Information with the sinfo Command Security Fault ToleranceJob Accounting $ sinfo -R Introduction to LSF in the HP XC Environment Using LSFOverview of LSF Topology Support Nodelist= list-of-nodes Exclude= list-of-nodes $ bsub -n 10 -ext SLURMnodes=10 srun myapp $ bsub -n 10 -ext SLURMnodes=10exclude=n16 srun myapp $ bqueues -l normal grep Jobstarter How LSF and Slurm Launch and Manage a Job$ bsub -Is hostname How LSF-HPC and Slurm Launch and Manage a Job $ bsub -n4 -ext SLURMnodes=4 -o output.out ./myscript Differences Between LSF on HP XC and Standard LSF Getting Status of LSF Determining Execution HostDetermining Available System Resources Job Startup and Job Control Getting Information About LSF-HPC Execution Host Node Getting Host Load Information Getting Information About the lsf Partition Submitting JobsChecking LSF System Queues $ sinfo -p lsf Summary of the LSF bsub Command Format Bsub bsuboptions jobname joboptions LSF-SLURM External Scheduler Slurm Arguments Function Starting on lsfhost.localdomain n6 Submitting an HP-MPI Job Submitting a Serial JobSubmitting a Job in Parallel Example 7-5 Submitting an Interactive Serial Job Example 7-6 Submitting an HP-MPI Job Submitting a Batch Job or Job Script$ bsub -n4 -I mpirun -srun ./helloworld $ bsub -n4 -I ./myscript.sh ExamplesExample 7-8 Submitting a Batch Job Script $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh $ bsub -n4 -I ./myscript.sh -n8 -O Submitting a Job from a Non-HP XC HostType=SLINUX64 Job Allocation Information for a Running Job Getting Information About JobsGetting Job Allocation Information $ bsub -R type=SLINUX64 -n4 -I srun hostname Checking Status of a Job Example 7-13 Using the bjobs Command Short OutputJob Allocation Information for a Finished Job Output Provided by the bhist Command Example 7-14 Using the bjobs Command Long OutputExample 7-15 Using the bhist Command Short Output Viewing a Job’s Historical Information Working Interactively Within an LSF-HPC Allocation Example 7-16 Using the bhist Command Long OutputSubmitting an Interactive Job to Launch the xterm Program Example 7-19 View Your Running Job in LSF Example 7-17 View Your EnvironmentExample 7-18 View Your Allocation in Slurm Example 7-20 View Job Details in LSF Example 7-22 Submitting an Interactive Shell Program Submitting an Interactive Job to Launch a ShellExample 7-21 Running Jobs from an xterm Window $ hostname $ srun hostname $ srun -n2 hostname $ srun hostname n1 $ exit LSF Equivalents of Slurm srun OptionsLSF Equivalents of Slurm srun Options Srun Option Description LSF Equivalent 24Using LSF Control the parallel job Page Using HP-MPI Compiling and Running Applications Setting Environment VariablesHP-MPI Directory Structure Building and Running an Example Application $ mpicc -o helloworld $MPIROOT/help/helloworld.c Example Application helloworldBuilding and Running helloworld $ $MPIROOT/bin/mpirun -srun -n4 helloworld Launching MPI Jobs HP-MPI options allowed with -srun $ mpirun -srun -n6 -O -N2 -m cyclic ./a.out host1 rank1 Creating Subshells and Launching JobstepsSystem Interconnect Selection $ mpirun -srun -n4 -N2 -O -m cyclic ./a.out host1 rank1 Using LSF and HP-MPI $ mpirun -subnet 192.168.1.1 -prot -srun -n4 ./a.out MPI Versioning $ /usr/sbin/ifconfig -aSystem Interconnect Support Example 8-5 Allocating 12 Processors on 6 Nodes Truncated Messages 32-Bit Builds on XC4000Allowing Windows to Use Exclusive Locks Mpirun Command Options $ mpirun -TCP -srun -N8 ./a.out Environment Variables $MPIROOT/bin/mpirun -v -prot -np 2 /path/to/program.x Export MPIPAGEALIGNMEM=1 Export MPIPHYSICALMEMORY=1048576Export MPIPINPERCENTAGE=30 Export MPIMAXWINDOW=10 Export MPIUSELIBELANUSE=5 Mpich Object CompatibilityExport MPIUSELIBELAN=0 $MPIROOT/bin/mpirun.mpich -np 2 ./prog.x HP-MPI Manpage Categories HP-MPI Documentation and ManpagesAt http//docs.hp.com Category Manpages Description Additional Information, Known Problems, and Work-arounds Version 8 Fortran Compiler Using HP MlibIntel Compiler Notes Version 7 Fortran Compiler Platform Support HP Mlib for the HP XC6000 PlatformMlib and Module Files Library Support Modulefiles and Mlib Using Intel Compilers with HP MlibCompiling and Linking Mlib Manpages HP Mlib for the HP XC4000 PlatformLicensing Linking SuperLUDIST $ pgcc options file 6Using HP Mlib Running an X Terminal Session from a Remote Node Advanced TopicsEnabling Remote Execution with OpenSSH Determining IP Address of Your Local Machine $ hostname Running an X terminal Session Using SlurmLogging in to HP XC System $ host mymachine $ bsub -n4 -Ip srun -n1 xterm -display Running an X terminal Session Using LSF$ bjobs Page Building and Running a Serial Application ExamplesLaunching a Serial Interactive Shell Through LSF Running LSF Jobs with a Slurm Allocation Request Example 1. Two Processors on Any Two Nodes Example 2. Four Processors on Two Specific Nodes Launching a Parallel Interactive Shell Through LSFView the job Check the running job’s information $ bsub -Is -n4 -ext SLURMnodes=4 /bin/bash Show the environment Submitting a Simple Job Script with LSFCheck the finished job’s information Display the script Show the Slurm job ID Submitting an Interactive Job with LSFShow the job allocation $ bsub -n8 -Ip /bin/sh Submitting an HP-MPI Job with LSF View the finished job View the running job$ bjobs -l Using a Resource Requirements String in an LSF Command $ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjobPage Glossary Fairshare Extensible firmware interfaceExternal network node First come first served Interconnect Image serverIntegrated Lights Out Internet address Management Processor Network Information ServicesLSF master host Master host Resource manager role Root Administration SwitchParallel application Role Symmetric multiprocessing Glossary-6 Index Index-1 Gdb, 4-1 GNU Index-2 Modulefile Index-3 Resource manager, 7-1role Serial applications Index-4