InfiniPath User Guide
 InfiniPath User Guide Version
 Index
Page
 Table of Contents
 Section Using InfiniPath MPI
 Appendix a Benchmark Programs
 InfiniPath User Guide Version
 InfiniPath Software Structure
Appendix D Recommended Reading
 InfiniPath User Guide Version IB6054601-00 D
 How this Guide is Organized
Who Should Read this Guide
 Appendix E Glossary of technical terms
Switches
Overview
Interoperability
 PathScale-QLogic Adapter Model Numbers
What’s New in this Release
 InfiniPath/OpenFabrics Supported Distributions and Kernels
Supported Distributions and Kernels
 Software Components
 InfiniPath product documentation includes
Conventions Used in this Document
Documentation and Technical Support
This Guide uses these typographical conventions
 Introduction Documentation and Technical Support
Page
 Introduction
Installed Layout
 Memory Footprint
MPI include files are
Documentation can be found
 Adapter Required Component Optional Memory Footprint Comment
 Memory Footprint, 331 MB per Node
Configuration and Startup Bios Settings
 InfiniPath Driver Software Configuration
InfiniPath Driver Startup
InfiniPath Driver Filesystem
 Layered Ethernet Driver
Subnet Management Agent
 Ipathether Configuration on Fedora and RHEL4
$ ifconfig -a $ ls /sys/class/net
Create or edit the following file as root
 ONBOOT=YES
Ipathether Configuration on Suse 9.3, Sles 9, and Sles
 # lsmod grep -q ipathether modprobe ipathether
Determine the MAC address that will be used
Guid can also be returned by running
If this file does not exist, skip to Step
 Add the following lines to the file
 # ifconfig ib0 10.1.17.3 netmask 0xffffff00
OpenFabrics Configuration and Startup
Configuring the IPoIB Network Interface
# ifup eth2 # ifconfig eth2
 OpenSM
 You can stop it again like this
Further Information on Configuring and Loading Drivers
To enable the driver, use the command as root
Starting and Stopping the InfiniPath Software
 $ /sbin/lsmod grep ipathether
# chkconfig infinipath off
# /sbin/chkconfig --list opensmd grep -w on
# /etc/init.d/infinipath start stop restart
 Software Status
Configuring ssh and sshd Using shosts.equiv
 Change the file to mode 600 when finished editing
 # killall -HUP sshd
Remove Unneeded Services
Process Limitation with ssh
Performance and Management Tips
 Disable Powersaving Features
 CPU Affinity
Balanced Processor Power
For Suse 9.3 and 10.0 run this command as root
SDP Module Parameters for Best Performance
 Ipathbughelper
Hyper-Threading
Homogeneous Nodes
Run the shell script ipathcontrol as follows
 Option -qwill query and -qawill query all
Non-QLogic built
For non-QLogic RPMs, it will look like
 Command strings can also be used. Here is a sample
Customer Acceptance Utility
Will produce output like this
 Displays help messages giving defined usage
Page
 InfiniPath MPI
Other MPI Implementations
Getting Started with MPI
 $ mpicc -o cpi cpi.c
An Example C Program
There is more information on the mpihosts file in section
$ cp /usr/share/mpich/examples/basic
 $ mpirun -np 2 -m mpihosts ./fpi3
Examples Using Other Languages
Run it with
$ mpif77 -o fpi3 fpi3.f
 Run it
Configuring MPI Programs for InfiniPath MPI
 InfiniPath MPI Details
Configuring for ssh Using ssh-agent
 This tells ssh that your key pair should let you
Edit ~/.ssh/config so that it reads like this
 Provides help
Gets verbose output of all the commands in the script
Shows how to compile a program
Compiling and Linking
 To use PGI for Fortran90/Fortran95 programs, use
To Use Another Compiler
To use gcc for compiling and linking C++ programs use
To use gcc for compiling and linking Fortran77 programs use
 $ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90
Cross-compilation Issues
Compiler and Linker Variables
$ mpif90 -f90=ifort $ mpif95 -f95=ifort
 Running MPI Programs
 Mpihosts File
 Environment for Node Programs
Console I/O in MPI Programs
 $ mpirun -np n -m mpihosts -rcfile mpirunrc program
Environment for Multiple Versions of InfiniPath or MPI
 $ mpirun -np n -m mpihosts -ppn p program-name
Multiprocessor Nodes
Number of processes to spawn
Create up to specified number of processes per node
 Run each process in an xterm window. Default Off
Print MPI version. Default Off
Print mpirun help message. Default Off
Startup script for setting environment on nodes Default
 Sets the working directory for the node program
Disable-mpi-progress-check
 Using Other MPI Implementations
MPI Over uDAPL
# modprobe rdmacm # modprobe rdmaucm
 File I/O in MPI
Using MPD
$ mpdboot -f hostsfile
MPD Description
 InfiniPath MPI and Hybrid MPI/OpenMP Applications
MPI-IO with Romio
 MPI Errors
Using Debuggers
Debugging MPI Programs
 InfiniPath MPI Limitations
 No ports available on /dev/ipath
 Benchmark 1 Measuring MPI Latency Between Two Nodes
Benchmark Programs
 $ mpirun -np 2 -ppn 1 -m mpihosts osulatency
Benchmark 2 Measuring MPI Bandwidth Between Two Nodes
 Benchmark 3 Messaging Rate Microbenchmarks
 $ mpirun -np 8 ./mpimultibw
 Benchmark 4 Measuring MPI Latency in Host Rings
Might produce output like this
$ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100
Page
 Allocating Resources
Batch Queuing Script
 $ batchmpirun -np n mympiprogram
Generating the mpihosts File
 # lsof /dev/ipath
Clean Termination of MPI Processes
Simple Process Management
# /sbin/fuser -v /dev/ipath
 Lock Enough Memory on Nodes When Using Slurm
# /sbin/fuser -k /dev/ipath
# ulimit -l
 Mechanical and Electrical Considerations
Troubleshooting InfiniPath Adapter Installation
 Some HTX Motherboards May Need 2 or More CPUs in Use
Table C-1. LED Link and Data Indicators
 Mtrr Mapping and Write Combining
Incorrect Mtrr Mapping
$ ipathpkttest -B
 To check your bandwidth try
Change Setting for Mapping Memory
Issue with SuperMicro H8DCE-HTe and QHT7040
Incorrect Mtrr Mapping Causes Unexpected Low Bandwidth
 OpenFabrics Dependencies
Software Installation Issues
Install Warning with RHEL4U2
Mpirun Installation Requires 32-bit Support
 $ rpm -e infinipath-kernel --nodeps
Installing Newer Drivers from Other Distributions
Remove the InfiniPath kernel components with the command
Check your distribution for the exact RPM name
 These are used for SLES, SUSE, and Fedora
Installing for Your Distribution
Kernel and Initialization Issues
Reload all modules by using this command as root
 Pcimsiquirk
Kernel Needs CONFIGPCIMSI=y
 $ grep ibipath /proc/interrupts
Driver Load Fails Due to Unsupported Kernel
InfiniPath Interrupts Not Working
Normal output will like similar to this
 $ grep -i acpi /proc/cmdline
OpenFabrics Load Errors If ibipath Driver Load Fails
 InfiniPath ibipath Initialization Failure
MPI Job Failures Due to Initialization Problems
 Stop OpenSM Before Stopping/Restarting InfiniPath
OpenFabrics Issues
Load and Configure IPoIB Before Loading SDP
System Administration Troubleshooting
 Broken Intermediate Link
Performance Issues
Mvapich Performance Issues
InfiniPath MPI Troubleshooting
 $ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatency
$ mpirun
 $ mpicc myprogram.c
Compiler/Linker Mismatch
Compiler Can’t Find Include, Module or Library Files
$ export MPICHCC=gcc $ mpicc mpiworld.c
 $ mpif90 myprogramf90.f90 -I/path/to/devel/include
Compiling on Development Nodes
Specifying the Run-time Library Path
$ mpicc myprogram.c -I/path/to/devel/include
 # /etc/ldconfig
Run Time Errors With Different MPI Implementations
 Troubleshooting InfiniPath MPI Troubleshooting
 Using MPI.mod Files
When compiling, use descriptive names for the object files
HP-MPI mpirun and executable used together
 Extending MPI Modules
 Troubleshooting InfiniPath MPI Troubleshooting
 Error Messages Generated by mpirun
Messages from the InfiniPath Library
$ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000
 Troubleshooting InfiniPath MPI Troubleshooting
 MPI Messages
 Following indicates an unknown host
There is no route to a valid host
$ mpirun -np 2 -m ~/tmp/q mpilatency 100
 There is no route to any host
One program on one node died
$ mpirun -np 2 -m ~/tmp/q mpilatency 100000
 $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000
Driver and Link Error Messages Reported by MPI Programs
 MPI Stats
 Restarting InfiniPath
These two commands perform the same functions
Useful Programs and Files for Debugging
Check Cluster Homogeneity with ipathcheckout
 Shell script that gathers status
Summary of Useful Programs and Files
 Modprobe
Example contents are
 Ibstatus
 Ipathcheckout
 Ipathcontrol
 Ipathbug-helper
 $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1
To check the contents of an RPM, use these commands
Option-qwill query and --qawill query all
# lsmod egrep ’ipathibrdmafindex’
 Table C-3. statusstr File
 Example contents for QLogic-built drivers
Command strings can also be used. Its format is as follows
 InfiniBand OpenFabrics
References for MPI
Books for Learning MPI Programming
Reference and Source for Slurm
 Rocks
Clusters
 Glossary
 GID
 LID
 Mtrr
 SDP
 Glossary IB6054601-00 D
 Index
 OpenFabrics Configuration
Configuration of on Suse and Sles