Example Procedure | HP XC System 4.x Software instruction

for i in ${HYPRE_DIRS}; \ do \

if [ -d $$i ]; \ then \

echo "Making $$i ..."; \ (cd $$i; make); \

echo ""; \ fi; \

done

clean:

@ \

for i in ${HYPRE_DIRS}; \ do \

if [ -d $$i ]; \ then \

echo "Cleaning $$i ..."; \ (cd $$i; make clean); \

fi; \ done

veryclean: @ \

for i in ${HYPRE_DIRS}; \ do \

if [ -d $$i ]; \ then \

echo "Very-cleaning $$i ..."; \ (cd $$i; make veryclean); \

fi; \ done

11.3.1 Example Procedure 1

Go through the directories serially and have the make procedure within each directory be parallel.

For the purpose of this exercise we are only parallelizing the “make all” component. The “clean” and “veryclean” components can be parallelized in a similar fashion.

Modified Makefile:

all:

@ \

for i ${HYPRE_DIRS}; \ do \

if [ -d $$i ]; \ then \

echo "Making $$i ..."; \

echo $(PREFIX) $(MAKE) $(MAKE_J) -C $$i; \ $(PREFIX) $(MAKE) $(MAKE_J) -C $$i; \

fi; \ done

By modifying the Makefile to reflect the changes illustrated above, we will now be processing each directory serially and parallelize the individual makes within each directory. The modified Makefile is invoked as follows:

$ make PREFIX=’srun –n1 –N1 MAKE_J='-j4'

11.3.2 Example Procedure 2

Go through the directories in parallel and have the make procedure within each directory be serial.

For the purpose of this exercise we are only parallelizing the “make all” component. The “clean” and “veryclean” components can be parallelized in a similar fashion.

11.3 Using the GNU Parallel Make Capability 111

Image 111

HP XC System 4.x Software manual Example Procedure, $ make PREFIX=’srun -n1 -N1 MAKEJ=-j4

Contents

HP XC System Software XC User Guide Page Table of Contents Compiling and Linking Serial Applications Configuring Your Environment with Modulefiles Srun Roles Srun Modes Debugging Applications 115 107 131 125 List of Figures 10-111-1 List of Tables List of Examples Typographic Conventions About This DocumentIntended Audience New and Changed Information in This Edition HP XC and Related HP Products Information HP XC System Software Users Guide Related Information Https//computing.llnl.gov/linux/slurm/documentation.html Http//systemimager.org Http//www-unix.mcs.anl.gov/mpi Manpages HP Encourages Your Comments $ man discover $ man 8 discover$ man -k keyword Page Page /proc/cpuinfo file is dynamic Only to the administrator of the HP XC systemNode Specialization Determining the Node Platform Local Storage Storage and I/OFile System SAN Storage System Interconnect Network File System LayoutHP XC System Interconnects 1 LVS Determining System Configuration InformationUser Environment Network Address Translation NAT Commands Application Development EnvironmentParallel Applications Load Sharing Facility LSF Run-Time EnvironmentSerial Applications Slurm Nodes for the job Requested by the HP-MPI mpirun commandHow LSF and Slurm Interact HP-MPI Components, Tools, Compilers, Libraries, and Debuggers Page Logging In to the System Using the SystemLVS Login Routing Using the Secure Shell to Log $ bqueues IntroductionGetting Information About Queues Getting Information About Resources Performing Other Common User Tasks $ man sinfo Getting System Help and InformationDetermining the LSF Cluster Name and the LSF Execution Host $ lsid Overview of Modules Configuring Your Environment with Modulefiles Supplied Modulefiles Supplied Modulefiles HP-MPI Loading a Modulefile Modulefiles Automatically Loaded on the SystemViewing Available Modulefiles Viewing Loaded Modulefiles Unloading a Modulefile Viewing Modulefile-Specific HelpModulefile Conflicts $ module load modules $ man modulefile Creating a Modulefile Application Development Environment Overview Developing Applications Compiler Commands CompilersMPI Compiler Developing Serial Applications Setting Debugging OptionsExamining Nodes and Partitions Before Running Jobs Interrupting a Job Parallel Application Build Environment Developing Parallel ApplicationsSerial Application Build Environment Building Serial Applications Pthreads ModulefilesHP-MPI OpenMP GNU C and C++ Compilers MPI LibraryIntel Fortran and C/C++Compilers PGI Fortran and C/C++ Compilers Building Parallel Applications To compile and link a C application using the mpicc command Developing LibrariesDesigning Libraries for the CP4000 Platform Linkcommand 32-bit -L/opt/mypackage/lib/i686 -lmystuff Linkcommand 64-bit -L/opt/mypackage/lib/x8664 -lmystuffTo build a 64-bit application, you might enter Developing Applications Overview of Job Submission Submitting a Serial Job Using LSFSubmitting a Serial Job with the LSF bsub Command Submitting Jobs $ bsub -I srun hostname Example 5-2 Submitting a Serial Job Using LSFSubmitting a Serial Job Through Slurm Only Example 5-1 Submitting a Job from the Standard Input To submit a parallel job Following is the command line used to compile this programSubmitting a Parallel Job Submitting a Non-MPI Parallel Job $ bsub -n4 -I srun hostname Example 5-5 Submitting a Non-MPI Parallel Job $ bsub -n4 -I mpirun -srun ./helloworld Example 5-7 Submitting an MPI Job Nodelist=list of nodes Arguments for the Slurm External Scheduler $ bsub -n 10 -ext SLURMnodelist=n1-10 srun hostname $ bqueues -l dualcore grep Slurm Submitting a Batch Job or Job ScriptExample 5-14 Submitting a Job Script $ bsub -n 10 -ext SLURMconstraint=dualcore -I srun hostname $ bsub -n4 -I ./myscript.sh $ bsub -n4 -ext SLURMnodes=4 -I ./myscript.sh Submitting Multiple MPI Jobs Across the Same Set of Nodes Using a Script to Submit Multiple JobsUsing a Makefile to Submit Multiple Jobs Use the squeue command to acquire information on the jobs Following command line makes the program and executes it $ cat mymake$ tail 113.out $ cat 117.out $ bsub -R type=SLINUX64 -n4 -I srun hostname Submitting a Job from a Host Other Than an HP XC HostRunning Preexecution Programs Type=SLINUX64Page TotalView Debugging ApplicationsDebugging Serial Applications Debugging Parallel Applications Module load mpi module load totalview Setting Up TotalViewDebugging with TotalView SSH and TotalView $ srun -Nx-A $ mpirun -tv -srun application Using TotalView with SlurmUsing TotalView with LSF Setting TotalView Preferences $ mpirun -tv -srun -n2 ./Psimple Debugging an Application Run the application $ scancel --user usernameDebugging Running Applications Exiting TotalViewPage Xtools Utilities Monitoring Node Activity Where Running Performance Health Tests $ ovp --verbose --verify=perfhealth/cpuusage You can list the available tests with the ovp -lcommand$ ovp -l $ ovp --verify=perfhealth/cpuusage HOMEDIRECTORY/ovpn16mmddyy.log HOMEDIRECTORY/ovpn16mmddyyr1.log Page Tuning Applications Using the Intel Trace Collector and Intel Trace AnalyzerBuilding a Program Intel Trace Collector and HP-MPI Example 8-2 C Example Running the vtjacobic Example Program Running a Program Intel Trace Collector and HP-MPIExample 8-1 The vtjacobic Example Program For more information, see the following Web site Installation Kit Intel Trace Collector and Analyzer with HP-MPI on HP XCHP-MPI and the Intel Trace Collector Following is a Fortran example called vtjacobif Running a Program Running a Program Across Nodes Using LSF Visualizing Data Intel Trace Analyzer and HP-MPI# bsub -n4 -I mpirun.mpich -np 2 ./vtjacobic Page Slurm Utilities Using SlurmLaunching Jobs with the srun Command Introduction to Slurm Srun Roles and Modes Monitoring Jobs with the squeue CommandUsing the srun Command with HP-MPI Using the srun Command with LSF Terminating Jobs with the scancel Command Getting System Information with the sinfo CommandExample 9-5cancels all pending jobs $ sinfo -R Fault ToleranceJob Accounting Security Information for LSF Using LSF $ squeue --jobs $SLURMJOBID Overview of LSF Integrated with Slurm Example 10-1 Examples of LSF Job Launch Lsfadmin@n16 ~$ bsub -n4 -I srun hostname Differences Between LSF and LSF Integrated with Slurm Lshosts and lsload commands display for each of these items $ ssh n15 lshosts Job TerminologyConnection back to the terminal from which the job was Submitted. This job may run immediately, or it may run Batch system scheduling policies Batch jobs are submitted with the srun -bcommand. ByDefault, the output is written to Are submitted with the bsub -Icommand Job Startup and Job Control Using LSF Integrated with Slurm in the HP XC EnvironmentUseful Commands Submitting Jobs LSF-SLURM External Scheduler How LSF and Slurm Launch and Manage a JobLSF with Slurm Job Launch Exit Codes How LSF and Slurm Launch and Manage a Job User logs in to login node n16 Determining Available System Resources Determining the LSF Execution Host Following example shows the output from the lshosts command Examining System Core StatusGetting Information About the LSF Execution Host Node Getting Information About the lsf Partition Getting Information About JobsGetting Host Load Information Examining System Queues Than what the job requests Getting Job Allocation InformationThis allocation string has the following values LSF job with Slurm allocated resources Example 10-5 Using the bjobs Command Short Output Examining the Status of a JobExample 10-4 Job Allocation Information for a Finished Job Viewing the Historical Information for a Job Example 10-6 Using the bjobs Command Long OutputExample 10-7 Using the bhist Command Short Output Output Provided by the bhist Command Example 10-8 Using the bhist Command Long Output Use the bjobs command to view the Slurm JobidTranslating Slurm and LSF JOBIDs $ bjobs -l 124 grep slurm Working Interactively Within an Allocation$ sacct -j $ bsub -I -n4 -ext SLURMnodes=4 /bin/bash Example 10-9 Launching an Interactive MPI Job This example assumes 2 cores per nodeAlternatively, you can use the following $ unset Slurmjobid $ unset Slurmnprocs LSF Equivalents of Slurm srun Options3describes the srun options and lists their LSF equivalents LSF Equivalents of Slurm srun Options Requests a specific list of nodes. Slurm Suppress informational message Determining IP Address of Your Local Machine Advanced TopicsEnabling Remote Execution with OpenSSH Running an X Terminal Session from a Remote Node Running an X terminal Session Using Slurm Running an X terminal Session Using LSFLogging in to HP XC System Monitors display server address Using the GNU Parallel Make CapabilityOptions used in this command are Allocate 4 cores Run the job on 1 core $ cd subdir srun -n1 -N1 $MAKE -j4 $ cd subdir srun $MAKE $ make PREFIX=’srun -n1 -N1 MAKEJ=-j4 Example Procedure Local Disks on Compute Nodes Modified Makefile is invoked as follows$ make PREFIX=srun -n1 -N1 MAKEJ=-j4 Shared File View Using Mpich on the HP XC System11.5 I/O Performance Considerations Communication Between Nodes Using Mpich with Slurm Allocation Using Mpich with LSF AllocationBsub command launches the wrapper script Examine the LSF execution host information ExamplesBuilding and Running a Serial Application Launching a Serial Interactive Shell Through LSF Running LSF Jobs with a Slurm Allocation Request Example 2. Four Cores on Two Specific Nodes Launching a Parallel Interactive Shell Through LSF $ hostname n16 $ srun hostname n5 $ bjobs Examine the running jobs information Display the script Submitting a Simple Job Script with LSFExamine the finished jobs information Show the environment Show the job allocation Submitting an Interactive Job with LSFRun the job Submit the job View the interactive jobs Run some commands from the pseudo-terminalShow the Slurm job ID Exit the pseudo-terminal Submitting an HP-MPI Job with LSF View the node stateView the running job Using a Resource Requirements String in an LSF Command View the finished job$ bsub -n 8 -R ALPHA5 SLINUX64 \ -ext SLURMnodes=4-4 myjob 124 Network Availability set Glossary Fcfs Ipmi LVS PXE SVA Index Index LVS PGI 135