InfiniPath User Guide
InfiniPath User Guide Version
Index
Page
Table of Contents
Section Using InfiniPath MPI
Appendix a Benchmark Programs
InfiniPath User Guide Version
Appendix D Recommended Reading
InfiniPath Software Structure
InfiniPath User Guide Version IB6054601-00 D
Who Should Read this Guide
How this Guide is Organized
Switches
Overview
Interoperability
Appendix E Glossary of technical terms
What’s New in this Release
PathScale-QLogic Adapter Model Numbers
Supported Distributions and Kernels
InfiniPath/OpenFabrics Supported Distributions and Kernels
Software Components
Conventions Used in this Document
Documentation and Technical Support
This Guide uses these typographical conventions
InfiniPath product documentation includes
Introduction Documentation and Technical Support
Page
Installed Layout
Introduction
Documentation can be found
Memory Footprint
MPI include files are
Adapter Required Component Optional Memory Footprint Comment
Configuration and Startup Bios Settings
Memory Footprint, 331 MB per Node
InfiniPath Driver Filesystem
InfiniPath Driver Software Configuration
InfiniPath Driver Startup
Subnet Management Agent
Layered Ethernet Driver
Create or edit the following file as root
Ipathether Configuration on Fedora and RHEL4
$ ifconfig -a $ ls /sys/class/net
Ipathether Configuration on Suse 9.3, Sles 9, and Sles
ONBOOT=YES
Determine the MAC address that will be used
Guid can also be returned by running
If this file does not exist, skip to Step
# lsmod grep -q ipathether modprobe ipathether
Add the following lines to the file
OpenFabrics Configuration and Startup
Configuring the IPoIB Network Interface
# ifup eth2 # ifconfig eth2
# ifconfig ib0 10.1.17.3 netmask 0xffffff00
OpenSM
Further Information on Configuring and Loading Drivers
To enable the driver, use the command as root
Starting and Stopping the InfiniPath Software
You can stop it again like this
# chkconfig infinipath off
# /sbin/chkconfig --list opensmd grep -w on
# /etc/init.d/infinipath start stop restart
$ /sbin/lsmod grep ipathether
Configuring ssh and sshd Using shosts.equiv
Software Status
Change the file to mode 600 when finished editing
Remove Unneeded Services
Process Limitation with ssh
Performance and Management Tips
# killall -HUP sshd
Disable Powersaving Features
Balanced Processor Power
For Suse 9.3 and 10.0 run this command as root
SDP Module Parameters for Best Performance
CPU Affinity
Hyper-Threading
Homogeneous Nodes
Run the shell script ipathcontrol as follows
Ipathbughelper
For non-QLogic RPMs, it will look like
Option -qwill query and -qawill query all
Non-QLogic built
Will produce output like this
Command strings can also be used. Here is a sample
Customer Acceptance Utility
Displays help messages giving defined usage
Page
Getting Started with MPI
InfiniPath MPI
Other MPI Implementations
An Example C Program
There is more information on the mpihosts file in section
$ cp /usr/share/mpich/examples/basic
$ mpicc -o cpi cpi.c
Examples Using Other Languages
Run it with
$ mpif77 -o fpi3 fpi3.f
$ mpirun -np 2 -m mpihosts ./fpi3
Configuring MPI Programs for InfiniPath MPI
Run it
Configuring for ssh Using ssh-agent
InfiniPath MPI Details
Edit ~/.ssh/config so that it reads like this
This tells ssh that your key pair should let you
Gets verbose output of all the commands in the script
Shows how to compile a program
Compiling and Linking
Provides help
To Use Another Compiler
To use gcc for compiling and linking C++ programs use
To use gcc for compiling and linking Fortran77 programs use
To use PGI for Fortran90/Fortran95 programs, use
Cross-compilation Issues
Compiler and Linker Variables
$ mpif90 -f90=ifort $ mpif95 -f95=ifort
$ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90
Running MPI Programs
Mpihosts File
Console I/O in MPI Programs
Environment for Node Programs
Environment for Multiple Versions of InfiniPath or MPI
$ mpirun -np n -m mpihosts -rcfile mpirunrc program
Multiprocessor Nodes
Number of processes to spawn
Create up to specified number of processes per node
$ mpirun -np n -m mpihosts -ppn p program-name
Print MPI version. Default Off
Print mpirun help message. Default Off
Startup script for setting environment on nodes Default
Run each process in an xterm window. Default Off
Disable-mpi-progress-check
Sets the working directory for the node program
# modprobe rdmacm # modprobe rdmaucm
Using Other MPI Implementations
MPI Over uDAPL
Using MPD
$ mpdboot -f hostsfile
MPD Description
File I/O in MPI
MPI-IO with Romio
InfiniPath MPI and Hybrid MPI/OpenMP Applications
Debugging MPI Programs
MPI Errors
Using Debuggers
InfiniPath MPI Limitations
No ports available on /dev/ipath
Benchmark Programs
Benchmark 1 Measuring MPI Latency Between Two Nodes
Benchmark 2 Measuring MPI Bandwidth Between Two Nodes
$ mpirun -np 2 -ppn 1 -m mpihosts osulatency
Benchmark 3 Messaging Rate Microbenchmarks
$ mpirun -np 8 ./mpimultibw
$ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100
Benchmark 4 Measuring MPI Latency in Host Rings
Might produce output like this
Page
Batch Queuing Script
Allocating Resources
Generating the mpihosts File
$ batchmpirun -np n mympiprogram
Clean Termination of MPI Processes
Simple Process Management
# /sbin/fuser -v /dev/ipath
# lsof /dev/ipath
# ulimit -l
Lock Enough Memory on Nodes When Using Slurm
# /sbin/fuser -k /dev/ipath
Troubleshooting InfiniPath Adapter Installation
Mechanical and Electrical Considerations
Table C-1. LED Link and Data Indicators
Some HTX Motherboards May Need 2 or More CPUs in Use
$ ipathpkttest -B
Mtrr Mapping and Write Combining
Incorrect Mtrr Mapping
Change Setting for Mapping Memory
Issue with SuperMicro H8DCE-HTe and QHT7040
Incorrect Mtrr Mapping Causes Unexpected Low Bandwidth
To check your bandwidth try
Software Installation Issues
Install Warning with RHEL4U2
Mpirun Installation Requires 32-bit Support
OpenFabrics Dependencies
Installing Newer Drivers from Other Distributions
Remove the InfiniPath kernel components with the command
Check your distribution for the exact RPM name
$ rpm -e infinipath-kernel --nodeps
Installing for Your Distribution
Kernel and Initialization Issues
Reload all modules by using this command as root
These are used for SLES, SUSE, and Fedora
Kernel Needs CONFIGPCIMSI=y
Pcimsiquirk
Driver Load Fails Due to Unsupported Kernel
InfiniPath Interrupts Not Working
Normal output will like similar to this
$ grep ibipath /proc/interrupts
OpenFabrics Load Errors If ibipath Driver Load Fails
$ grep -i acpi /proc/cmdline
MPI Job Failures Due to Initialization Problems
InfiniPath ibipath Initialization Failure
OpenFabrics Issues
Load and Configure IPoIB Before Loading SDP
System Administration Troubleshooting
Stop OpenSM Before Stopping/Restarting InfiniPath
Performance Issues
Mvapich Performance Issues
InfiniPath MPI Troubleshooting
Broken Intermediate Link
$ mpirun
$ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatency
Compiler/Linker Mismatch
Compiler Can’t Find Include, Module or Library Files
$ export MPICHCC=gcc $ mpicc mpiworld.c
$ mpicc myprogram.c
Compiling on Development Nodes
Specifying the Run-time Library Path
$ mpicc myprogram.c -I/path/to/devel/include
$ mpif90 myprogramf90.f90 -I/path/to/devel/include
Run Time Errors With Different MPI Implementations
# /etc/ldconfig
Troubleshooting InfiniPath MPI Troubleshooting
HP-MPI mpirun and executable used together
Using MPI.mod Files
When compiling, use descriptive names for the object files
Extending MPI Modules
Troubleshooting InfiniPath MPI Troubleshooting
$ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000
Error Messages Generated by mpirun
Messages from the InfiniPath Library
Troubleshooting InfiniPath MPI Troubleshooting
MPI Messages
$ mpirun -np 2 -m ~/tmp/q mpilatency 100
Following indicates an unknown host
There is no route to a valid host
$ mpirun -np 2 -m ~/tmp/q mpilatency 100000
There is no route to any host
One program on one node died
Driver and Link Error Messages Reported by MPI Programs
$ mpirun -np ~/tmp/q -q 60 mpilatency 1000000
MPI Stats
These two commands perform the same functions
Useful Programs and Files for Debugging
Check Cluster Homogeneity with ipathcheckout
Restarting InfiniPath
Summary of Useful Programs and Files
Shell script that gathers status
Example contents are
Modprobe
Ibstatus
Ipathcheckout
Ipathcontrol
Ipathbug-helper
To check the contents of an RPM, use these commands
Option-qwill query and --qawill query all
# lsmod egrep ’ipathibrdmafindex’
$ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1
Table C-3. statusstr File
Command strings can also be used. Its format is as follows
Example contents for QLogic-built drivers
References for MPI
Books for Learning MPI Programming
Reference and Source for Slurm
InfiniBand OpenFabrics
Clusters
Rocks
Glossary
GID
LID
Mtrr
SDP
Glossary IB6054601-00 D
Index
Configuration of on Suse and Sles
OpenFabrics Configuration