Manuals / Q-Logic / Computer Equipment / Switch

Q-Logic IB6054601-00 D manual Broken Intermediate Link, Mvapich Performance Issues

Models: IB6054601-00 D

1 122

Download 122 pages 48.66 Kb

Page 87

Q

C – Troubleshooting InfiniPath MPI Troubleshooting

C.6.1

Broken Intermediate Link

Sometimes message traffic passes through the fabric while other traffic appears to be blocked. In this case, MPI jobs fail to run.

In large cluster configurations, switches may be attached to other switches in order to supply the necessary inter-node connectivity. Problems with these inter-switch (or intermediate) links are sometime more difficult to diagnose than failure of the final link between a switch and a node. The failure of an intermediate link may allow some traffic to pass through the fabric while other traffic is blocked or degraded.

If you encounter such behavior in a multi-layer fabric, check that all switch cable connections are correct. Statistics for managed switches are available on a per-port basis, and may help with debugging. See your switch vendor for more information.

C.7

Performance Issues

Performance issues that are currently being addressed are covered in this section.

C.7.1

MVAPICH Performance Issues

MVAPICH over OpenFabrics over InfiniPath performance tuning has not yet been done. Improved performance will be delivered in future releases.

C.8

InfiniPath MPI Troubleshooting

Problems specific to compiling and running MPI programs are detailed below.

C.8.1

Mixed Releases of MPI RPMs

Make sure that all of the MPI RPMs are from the same release. When using mpirun, an error message will occur if different components of the MPI RPMs are from different releases. This is a sample message in the case where mpirun from release 1.3 is being used with a 2.0 library:

$ mpirun -np 2 -m ~/tmp/x2 osu_latency

MPI_runscript-xqa-14.0: ssh -x> Cannot detect InfiniPath interconnect.

MPI_runscript-xqa-14.0: ssh -x> Seek help on loading InfiniPath interconnect driver.

MPI_runscript-xqa-15.1: ssh -x> Cannot detect InfiniPath interconnect.

MPI_runscript-xqa-15.1: ssh -x> Seek help on loading InfiniPath interconnect driver.

MPIRUN: Node program(s) exited during connection setup

IB6054601-00 D

C-13

Image 87

Q-Logic IB6054601-00 D manual Broken Intermediate Link, Mvapich Performance Issues, InfiniPath MPI Troubleshooting

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version InfiniPath Software Structure Appendix D Recommended ReadingInfiniPath User Guide Version IB6054601-00 D How this Guide is Organized Who Should Read this GuideAppendix E Glossary of technical terms SwitchesOverview InteroperabilityPathScale-QLogic Adapter Model Numbers What’s New in this ReleaseInfiniPath/OpenFabrics Supported Distributions and Kernels Supported Distributions and KernelsSoftware Components InfiniPath product documentation includes Conventions Used in this DocumentDocumentation and Technical Support This Guide uses these typographical conventionsIntroduction Documentation and Technical Support Page Introduction Installed LayoutMemory Footprint MPI include files areDocumentation can be found Adapter Required Component Optional Memory Footprint Comment Memory Footprint, 331 MB per Node Configuration and Startup Bios SettingsInfiniPath Driver Software Configuration InfiniPath Driver StartupInfiniPath Driver Filesystem Layered Ethernet Driver Subnet Management AgentIpathether Configuration on Fedora and RHEL4 $ ifconfig -a $ ls /sys/class/netCreate or edit the following file as root ONBOOT=YES Ipathether Configuration on Suse 9.3, Sles 9, and Sles# lsmod grep -q ipathether modprobe ipathether Determine the MAC address that will be usedGuid can also be returned by running If this file does not exist, skip to StepAdd the following lines to the file # ifconfig ib0 10.1.17.3 netmask 0xffffff00 OpenFabrics Configuration and StartupConfiguring the IPoIB Network Interface # ifup eth2 # ifconfig eth2OpenSM You can stop it again like this Further Information on Configuring and Loading DriversTo enable the driver, use the command as root Starting and Stopping the InfiniPath Software$ /sbin/lsmod grep ipathether # chkconfig infinipath off# /sbin/chkconfig --list opensmd grep -w on # /etc/init.d/infinipath start stop restartSoftware Status Configuring ssh and sshd Using shosts.equivChange the file to mode 600 when finished editing # killall -HUP sshd Remove Unneeded ServicesProcess Limitation with ssh Performance and Management TipsDisable Powersaving Features CPU Affinity Balanced Processor PowerFor Suse 9.3 and 10.0 run this command as root SDP Module Parameters for Best PerformanceIpathbughelper Hyper-ThreadingHomogeneous Nodes Run the shell script ipathcontrol as followsOption -qwill query and -qawill query all Non-QLogic builtFor non-QLogic RPMs, it will look like Command strings can also be used. Here is a sample Customer Acceptance UtilityWill produce output like this Displays help messages giving defined usage Page InfiniPath MPI Other MPI ImplementationsGetting Started with MPI $ mpicc -o cpi cpi.c An Example C ProgramThere is more information on the mpihosts file in section $ cp /usr/share/mpich/examples/basic$ mpirun -np 2 -m mpihosts ./fpi3 Examples Using Other LanguagesRun it with $ mpif77 -o fpi3 fpi3.fRun it Configuring MPI Programs for InfiniPath MPIInfiniPath MPI Details Configuring for ssh Using ssh-agentThis tells ssh that your key pair should let you Edit ~/.ssh/config so that it reads like thisProvides help Gets verbose output of all the commands in the scriptShows how to compile a program Compiling and LinkingTo use PGI for Fortran90/Fortran95 programs, use To Use Another CompilerTo use gcc for compiling and linking C++ programs use To use gcc for compiling and linking Fortran77 programs use$ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90 Cross-compilation IssuesCompiler and Linker Variables $ mpif90 -f90=ifort $ mpif95 -f95=ifortRunning MPI Programs Mpihosts File Environment for Node Programs Console I/O in MPI Programs$ mpirun -np n -m mpihosts -rcfile mpirunrc program Environment for Multiple Versions of InfiniPath or MPI$ mpirun -np n -m mpihosts -ppn p program-name Multiprocessor NodesNumber of processes to spawn Create up to specified number of processes per nodeRun each process in an xterm window. Default Off Print MPI version. Default OffPrint mpirun help message. Default Off Startup script for setting environment on nodes DefaultSets the working directory for the node program Disable-mpi-progress-checkUsing Other MPI Implementations MPI Over uDAPL# modprobe rdmacm # modprobe rdmaucm File I/O in MPI Using MPD$ mpdboot -f hostsfile MPD DescriptionInfiniPath MPI and Hybrid MPI/OpenMP Applications MPI-IO with RomioMPI Errors Using DebuggersDebugging MPI Programs InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark 1 Measuring MPI Latency Between Two Nodes Benchmark Programs$ mpirun -np 2 -ppn 1 -m mpihosts osulatency Benchmark 2 Measuring MPI Bandwidth Between Two NodesBenchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw Benchmark 4 Measuring MPI Latency in Host Rings Might produce output like this$ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Page Allocating Resources Batch Queuing Script$ batchmpirun -np n mympiprogram Generating the mpihosts File# lsof /dev/ipath Clean Termination of MPI ProcessesSimple Process Management # /sbin/fuser -v /dev/ipathLock Enough Memory on Nodes When Using Slurm # /sbin/fuser -k /dev/ipath# ulimit -l Mechanical and Electrical Considerations Troubleshooting InfiniPath Adapter InstallationSome HTX Motherboards May Need 2 or More CPUs in Use Table C-1. LED Link and Data IndicatorsMtrr Mapping and Write Combining Incorrect Mtrr Mapping$ ipathpkttest -B To check your bandwidth try Change Setting for Mapping MemoryIssue with SuperMicro H8DCE-HTe and QHT7040 Incorrect Mtrr Mapping Causes Unexpected Low BandwidthOpenFabrics Dependencies Software Installation IssuesInstall Warning with RHEL4U2 Mpirun Installation Requires 32-bit Support$ rpm -e infinipath-kernel --nodeps Installing Newer Drivers from Other DistributionsRemove the InfiniPath kernel components with the command Check your distribution for the exact RPM nameThese are used for SLES, SUSE, and Fedora Installing for Your DistributionKernel and Initialization Issues Reload all modules by using this command as rootPcimsiquirk Kernel Needs CONFIGPCIMSI=y$ grep ibipath /proc/interrupts Driver Load Fails Due to Unsupported KernelInfiniPath Interrupts Not Working Normal output will like similar to this$ grep -i acpi /proc/cmdline OpenFabrics Load Errors If ibipath Driver Load FailsInfiniPath ibipath Initialization Failure MPI Job Failures Due to Initialization ProblemsStop OpenSM Before Stopping/Restarting InfiniPath OpenFabrics IssuesLoad and Configure IPoIB Before Loading SDP System Administration TroubleshootingBroken Intermediate Link Performance IssuesMvapich Performance Issues InfiniPath MPI Troubleshooting$ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatency $ mpirun$ mpicc myprogram.c Compiler/Linker MismatchCompiler Can’t Find Include, Module or Library Files $ export MPICHCC=gcc $ mpicc mpiworld.c$ mpif90 myprogramf90.f90 -I/path/to/devel/include Compiling on Development NodesSpecifying the Run-time Library Path $ mpicc myprogram.c -I/path/to/devel/include# /etc/ldconfig Run Time Errors With Different MPI ImplementationsTroubleshooting InfiniPath MPI Troubleshooting Using MPI.mod Files When compiling, use descriptive names for the object filesHP-MPI mpirun and executable used together Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting Error Messages Generated by mpirun Messages from the InfiniPath Library$ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Troubleshooting InfiniPath MPI Troubleshooting MPI Messages Following indicates an unknown host There is no route to a valid host$ mpirun -np 2 -m ~/tmp/q mpilatency 100 There is no route to any host One program on one node died$ mpirun -np 2 -m ~/tmp/q mpilatency 100000 $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000 Driver and Link Error Messages Reported by MPI ProgramsMPI Stats Restarting InfiniPath These two commands perform the same functionsUseful Programs and Files for Debugging Check Cluster Homogeneity with ipathcheckoutShell script that gathers status Summary of Useful Programs and FilesModprobe Example contents areIbstatus Ipathcheckout Ipathcontrol Ipathbug-helper $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1 To check the contents of an RPM, use these commandsOption-qwill query and --qawill query all # lsmod egrep ’ipathibrdmafindex’Table C-3. statusstr File Example contents for QLogic-built drivers Command strings can also be used. Its format is as followsInfiniBand OpenFabrics References for MPIBooks for Learning MPI Programming Reference and Source for SlurmRocks ClustersGlossary GID LID Mtrr SDP Glossary IB6054601-00 D Index OpenFabrics Configuration Configuration of on Suse and Sles