Manuals / Q-Logic / Computer Equipment / Switch

Q-Logic IB6054601-00 D manual Troubleshooting InfiniPath Adapter Installation

Models: IB6054601-00 D

1 122

Download 122 pages 48.66 Kb

Page 75

Appendix C

Troubleshooting

This Appendix describes some of the existing provisions for diagnosing and fixing problems. The sections are organized in the following order:

■C.1 “Troubleshooting InfiniPath adapter installation”

■C.2 “BIOS settings”

■C.3 “Software installation issues”

■C.4 “Kernel and initialization issues”

■C.5 “OpenFabrics issues”

■C.6 “System administration troubleshooting”

■C.7 “Performance issues”

■C.8 “InfiniPath MPI troubleshooting”

■C.9 “Useful programs and files for debugging”

C.1

Troubleshooting InfiniPath Adapter Installation

This section lists conditions you may encounter while installing the InfiniPath QLE7140 or QHT7140 adapter, and offers suggestions for working around them.

C.1.1

Mechanical and Electrical Considerations

The LEDs function as link and data indicators once the InfiniPath hardware and software has been installed, the driver has been loaded, and the fabric is being actively managed by a Subnet Manager. The following table shows the possible

IB6054601-00 D

C-1

Image 75

Q-Logic IB6054601-00 D manual Troubleshooting InfiniPath Adapter Installation, Mechanical and Electrical Considerations

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version InfiniPath Software Structure Appendix D Recommended ReadingInfiniPath User Guide Version IB6054601-00 D How this Guide is Organized Who Should Read this GuideAppendix E Glossary of technical terms SwitchesOverview InteroperabilityPathScale-QLogic Adapter Model Numbers What’s New in this ReleaseInfiniPath/OpenFabrics Supported Distributions and Kernels Supported Distributions and KernelsSoftware Components InfiniPath product documentation includes Conventions Used in this DocumentDocumentation and Technical Support This Guide uses these typographical conventionsIntroduction Documentation and Technical Support Page Introduction Installed LayoutMemory Footprint MPI include files areDocumentation can be found Adapter Required Component Optional Memory Footprint Comment Memory Footprint, 331 MB per Node Configuration and Startup Bios SettingsInfiniPath Driver Software Configuration InfiniPath Driver StartupInfiniPath Driver Filesystem Layered Ethernet Driver Subnet Management AgentIpathether Configuration on Fedora and RHEL4 $ ifconfig -a $ ls /sys/class/netCreate or edit the following file as root ONBOOT=YES Ipathether Configuration on Suse 9.3, Sles 9, and Sles# lsmod grep -q ipathether modprobe ipathether Determine the MAC address that will be usedGuid can also be returned by running If this file does not exist, skip to StepAdd the following lines to the file # ifconfig ib0 10.1.17.3 netmask 0xffffff00 OpenFabrics Configuration and StartupConfiguring the IPoIB Network Interface # ifup eth2 # ifconfig eth2OpenSM You can stop it again like this Further Information on Configuring and Loading DriversTo enable the driver, use the command as root Starting and Stopping the InfiniPath Software$ /sbin/lsmod grep ipathether # chkconfig infinipath off# /sbin/chkconfig --list opensmd grep -w on # /etc/init.d/infinipath start stop restartSoftware Status Configuring ssh and sshd Using shosts.equivChange the file to mode 600 when finished editing # killall -HUP sshd Remove Unneeded ServicesProcess Limitation with ssh Performance and Management TipsDisable Powersaving Features CPU Affinity Balanced Processor PowerFor Suse 9.3 and 10.0 run this command as root SDP Module Parameters for Best PerformanceIpathbughelper Hyper-ThreadingHomogeneous Nodes Run the shell script ipathcontrol as followsOption -qwill query and -qawill query all Non-QLogic builtFor non-QLogic RPMs, it will look like Command strings can also be used. Here is a sample Customer Acceptance UtilityWill produce output like this Displays help messages giving defined usage Page InfiniPath MPI Other MPI ImplementationsGetting Started with MPI $ mpicc -o cpi cpi.c An Example C ProgramThere is more information on the mpihosts file in section $ cp /usr/share/mpich/examples/basic$ mpirun -np 2 -m mpihosts ./fpi3 Examples Using Other LanguagesRun it with $ mpif77 -o fpi3 fpi3.fRun it Configuring MPI Programs for InfiniPath MPIInfiniPath MPI Details Configuring for ssh Using ssh-agentThis tells ssh that your key pair should let you Edit ~/.ssh/config so that it reads like thisProvides help Gets verbose output of all the commands in the scriptShows how to compile a program Compiling and LinkingTo use PGI for Fortran90/Fortran95 programs, use To Use Another CompilerTo use gcc for compiling and linking C++ programs use To use gcc for compiling and linking Fortran77 programs use$ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90 Cross-compilation IssuesCompiler and Linker Variables $ mpif90 -f90=ifort $ mpif95 -f95=ifortRunning MPI Programs Mpihosts File Environment for Node Programs Console I/O in MPI Programs$ mpirun -np n -m mpihosts -rcfile mpirunrc program Environment for Multiple Versions of InfiniPath or MPI$ mpirun -np n -m mpihosts -ppn p program-name Multiprocessor NodesNumber of processes to spawn Create up to specified number of processes per nodeRun each process in an xterm window. Default Off Print MPI version. Default OffPrint mpirun help message. Default Off Startup script for setting environment on nodes DefaultSets the working directory for the node program Disable-mpi-progress-checkUsing Other MPI Implementations MPI Over uDAPL# modprobe rdmacm # modprobe rdmaucm File I/O in MPI Using MPD$ mpdboot -f hostsfile MPD DescriptionInfiniPath MPI and Hybrid MPI/OpenMP Applications MPI-IO with RomioMPI Errors Using DebuggersDebugging MPI Programs InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark 1 Measuring MPI Latency Between Two Nodes Benchmark Programs$ mpirun -np 2 -ppn 1 -m mpihosts osulatency Benchmark 2 Measuring MPI Bandwidth Between Two NodesBenchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw Benchmark 4 Measuring MPI Latency in Host Rings Might produce output like this$ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Page Allocating Resources Batch Queuing Script$ batchmpirun -np n mympiprogram Generating the mpihosts File# lsof /dev/ipath Clean Termination of MPI ProcessesSimple Process Management # /sbin/fuser -v /dev/ipathLock Enough Memory on Nodes When Using Slurm # /sbin/fuser -k /dev/ipath# ulimit -l Mechanical and Electrical Considerations Troubleshooting InfiniPath Adapter InstallationSome HTX Motherboards May Need 2 or More CPUs in Use Table C-1. LED Link and Data IndicatorsMtrr Mapping and Write Combining Incorrect Mtrr Mapping$ ipathpkttest -B To check your bandwidth try Change Setting for Mapping MemoryIssue with SuperMicro H8DCE-HTe and QHT7040 Incorrect Mtrr Mapping Causes Unexpected Low BandwidthOpenFabrics Dependencies Software Installation IssuesInstall Warning with RHEL4U2 Mpirun Installation Requires 32-bit Support$ rpm -e infinipath-kernel --nodeps Installing Newer Drivers from Other DistributionsRemove the InfiniPath kernel components with the command Check your distribution for the exact RPM nameThese are used for SLES, SUSE, and Fedora Installing for Your DistributionKernel and Initialization Issues Reload all modules by using this command as rootPcimsiquirk Kernel Needs CONFIGPCIMSI=y$ grep ibipath /proc/interrupts Driver Load Fails Due to Unsupported KernelInfiniPath Interrupts Not Working Normal output will like similar to this$ grep -i acpi /proc/cmdline OpenFabrics Load Errors If ibipath Driver Load FailsInfiniPath ibipath Initialization Failure MPI Job Failures Due to Initialization ProblemsStop OpenSM Before Stopping/Restarting InfiniPath OpenFabrics IssuesLoad and Configure IPoIB Before Loading SDP System Administration TroubleshootingBroken Intermediate Link Performance IssuesMvapich Performance Issues InfiniPath MPI Troubleshooting$ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatency $ mpirun$ mpicc myprogram.c Compiler/Linker MismatchCompiler Can’t Find Include, Module or Library Files $ export MPICHCC=gcc $ mpicc mpiworld.c$ mpif90 myprogramf90.f90 -I/path/to/devel/include Compiling on Development NodesSpecifying the Run-time Library Path $ mpicc myprogram.c -I/path/to/devel/include# /etc/ldconfig Run Time Errors With Different MPI ImplementationsTroubleshooting InfiniPath MPI Troubleshooting Using MPI.mod Files When compiling, use descriptive names for the object filesHP-MPI mpirun and executable used together Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting Error Messages Generated by mpirun Messages from the InfiniPath Library$ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Troubleshooting InfiniPath MPI Troubleshooting MPI Messages Following indicates an unknown host There is no route to a valid host$ mpirun -np 2 -m ~/tmp/q mpilatency 100 There is no route to any host One program on one node died$ mpirun -np 2 -m ~/tmp/q mpilatency 100000 $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000 Driver and Link Error Messages Reported by MPI ProgramsMPI Stats Restarting InfiniPath These two commands perform the same functionsUseful Programs and Files for Debugging Check Cluster Homogeneity with ipathcheckoutShell script that gathers status Summary of Useful Programs and FilesModprobe Example contents areIbstatus Ipathcheckout Ipathcontrol Ipathbug-helper $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1 To check the contents of an RPM, use these commandsOption-qwill query and --qawill query all # lsmod egrep ’ipathibrdmafindex’Table C-3. statusstr File Example contents for QLogic-built drivers Command strings can also be used. Its format is as followsInfiniBand OpenFabrics References for MPIBooks for Learning MPI Programming Reference and Source for SlurmRocks ClustersGlossary GID LID Mtrr SDP Glossary IB6054601-00 D Index OpenFabrics Configuration Configuration of on Suse and Sles