Manuals / Q-Logic / Computer Equipment / Switch

Q-Logic IB6054601-00 D manual Installing for Your Distribution, Kernel and Initialization Issues

Models: IB6054601-00 D

1 122

Download 122 pages 48.66 Kb

Page 81

Q

C – Troubleshooting Kernel and Initialization Issues

8. Reload all modules by using this command (as root):

#/etc/init.d/infinipath start

An alternate mechanism can be used, if provided as part of your alternate installation.

9.Run an OpenFabrics test program, such as ibstatus, to verify that your InfiniPath card(s) work correctly.

C.3.5

Installing for Your Distribution

You may be using a kernel which is compatible with one of the supported distributions, but which may not be picked up during infinipath-kernelinstallation. It may also happen when using make-install.shto manually recompile the drivers.

In this case, you can set your distribution with the $IPATH_DISTRO override. Run this command before installation, or before running make-install.sh. We use the RHEL4 Update 4 distribution as an example in this command for bash or sh users:

$ export IPATH_DISTRO=rhel4_U4

The distribution arguments that are currently understood are listed below. They are found in the file build-guards.sh.

These are used for RHEL, CentOS(Rocks), and Scientific Linux.

rhel4_U2 rhel4_U3 rhel4_U4

These are used for SLES, SUSE, and Fedora:

sles9

sles10

suse9.3

fc3

fc4

make-install.shand build-guards.share both found in this directory:

/usr/src/infinipath/drivers

C.4

Kernel and Initialization Issues

Issues that may prevent the system from coming up properly are described.

IB6054601-00 D

C-7

Image 81

Q-Logic IB6054601-00 D Installing for Your Distribution, Kernel and Initialization Issues, $ export IPATHDISTRO=rhel4U4

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version InfiniPath Software Structure Appendix D Recommended ReadingInfiniPath User Guide Version IB6054601-00 D How this Guide is Organized Who Should Read this GuideOverview SwitchesInteroperability Appendix E Glossary of technical termsPathScale-QLogic Adapter Model Numbers What’s New in this ReleaseInfiniPath/OpenFabrics Supported Distributions and Kernels Supported Distributions and KernelsSoftware Components Documentation and Technical Support Conventions Used in this DocumentThis Guide uses these typographical conventions InfiniPath product documentation includesIntroduction Documentation and Technical Support Page Introduction Installed LayoutMemory Footprint MPI include files areDocumentation can be found Adapter Required Component Optional Memory Footprint Comment Memory Footprint, 331 MB per Node Configuration and Startup Bios SettingsInfiniPath Driver Software Configuration InfiniPath Driver StartupInfiniPath Driver Filesystem Layered Ethernet Driver Subnet Management AgentIpathether Configuration on Fedora and RHEL4 $ ifconfig -a $ ls /sys/class/netCreate or edit the following file as root ONBOOT=YES Ipathether Configuration on Suse 9.3, Sles 9, and SlesGuid can also be returned by running Determine the MAC address that will be usedIf this file does not exist, skip to Step # lsmod grep -q ipathether modprobe ipathetherAdd the following lines to the file Configuring the IPoIB Network Interface OpenFabrics Configuration and Startup# ifup eth2 # ifconfig eth2 # ifconfig ib0 10.1.17.3 netmask 0xffffff00OpenSM To enable the driver, use the command as root Further Information on Configuring and Loading DriversStarting and Stopping the InfiniPath Software You can stop it again like this# /sbin/chkconfig --list opensmd grep -w on # chkconfig infinipath off# /etc/init.d/infinipath start stop restart $ /sbin/lsmod grep ipathetherSoftware Status Configuring ssh and sshd Using shosts.equivChange the file to mode 600 when finished editing Process Limitation with ssh Remove Unneeded ServicesPerformance and Management Tips # killall -HUP sshdDisable Powersaving Features For Suse 9.3 and 10.0 run this command as root Balanced Processor PowerSDP Module Parameters for Best Performance CPU AffinityHomogeneous Nodes Hyper-ThreadingRun the shell script ipathcontrol as follows IpathbughelperOption -qwill query and -qawill query all Non-QLogic builtFor non-QLogic RPMs, it will look like Command strings can also be used. Here is a sample Customer Acceptance UtilityWill produce output like this Displays help messages giving defined usage Page InfiniPath MPI Other MPI ImplementationsGetting Started with MPI There is more information on the mpihosts file in section An Example C Program$ cp /usr/share/mpich/examples/basic $ mpicc -o cpi cpi.cRun it with Examples Using Other Languages$ mpif77 -o fpi3 fpi3.f $ mpirun -np 2 -m mpihosts ./fpi3Run it Configuring MPI Programs for InfiniPath MPIInfiniPath MPI Details Configuring for ssh Using ssh-agentThis tells ssh that your key pair should let you Edit ~/.ssh/config so that it reads like thisShows how to compile a program Gets verbose output of all the commands in the scriptCompiling and Linking Provides helpTo use gcc for compiling and linking C++ programs use To Use Another CompilerTo use gcc for compiling and linking Fortran77 programs use To use PGI for Fortran90/Fortran95 programs, useCompiler and Linker Variables Cross-compilation Issues$ mpif90 -f90=ifort $ mpif95 -f95=ifort $ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90Running MPI Programs Mpihosts File Environment for Node Programs Console I/O in MPI Programs$ mpirun -np n -m mpihosts -rcfile mpirunrc program Environment for Multiple Versions of InfiniPath or MPINumber of processes to spawn Multiprocessor NodesCreate up to specified number of processes per node $ mpirun -np n -m mpihosts -ppn p program-namePrint mpirun help message. Default Off Print MPI version. Default OffStartup script for setting environment on nodes Default Run each process in an xterm window. Default OffSets the working directory for the node program Disable-mpi-progress-checkUsing Other MPI Implementations MPI Over uDAPL# modprobe rdmacm # modprobe rdmaucm $ mpdboot -f hostsfile Using MPDMPD Description File I/O in MPIInfiniPath MPI and Hybrid MPI/OpenMP Applications MPI-IO with RomioMPI Errors Using DebuggersDebugging MPI Programs InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark 1 Measuring MPI Latency Between Two Nodes Benchmark Programs$ mpirun -np 2 -ppn 1 -m mpihosts osulatency Benchmark 2 Measuring MPI Bandwidth Between Two NodesBenchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw Benchmark 4 Measuring MPI Latency in Host Rings Might produce output like this$ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Page Allocating Resources Batch Queuing Script$ batchmpirun -np n mympiprogram Generating the mpihosts FileSimple Process Management Clean Termination of MPI Processes# /sbin/fuser -v /dev/ipath # lsof /dev/ipathLock Enough Memory on Nodes When Using Slurm # /sbin/fuser -k /dev/ipath# ulimit -l Mechanical and Electrical Considerations Troubleshooting InfiniPath Adapter InstallationSome HTX Motherboards May Need 2 or More CPUs in Use Table C-1. LED Link and Data IndicatorsMtrr Mapping and Write Combining Incorrect Mtrr Mapping$ ipathpkttest -B Issue with SuperMicro H8DCE-HTe and QHT7040 Change Setting for Mapping MemoryIncorrect Mtrr Mapping Causes Unexpected Low Bandwidth To check your bandwidth tryInstall Warning with RHEL4U2 Software Installation IssuesMpirun Installation Requires 32-bit Support OpenFabrics DependenciesRemove the InfiniPath kernel components with the command Installing Newer Drivers from Other DistributionsCheck your distribution for the exact RPM name $ rpm -e infinipath-kernel --nodepsKernel and Initialization Issues Installing for Your DistributionReload all modules by using this command as root These are used for SLES, SUSE, and FedoraPcimsiquirk Kernel Needs CONFIGPCIMSI=yInfiniPath Interrupts Not Working Driver Load Fails Due to Unsupported KernelNormal output will like similar to this $ grep ibipath /proc/interrupts$ grep -i acpi /proc/cmdline OpenFabrics Load Errors If ibipath Driver Load FailsInfiniPath ibipath Initialization Failure MPI Job Failures Due to Initialization ProblemsLoad and Configure IPoIB Before Loading SDP OpenFabrics IssuesSystem Administration Troubleshooting Stop OpenSM Before Stopping/Restarting InfiniPathMvapich Performance Issues Performance IssuesInfiniPath MPI Troubleshooting Broken Intermediate Link$ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatency $ mpirunCompiler Can’t Find Include, Module or Library Files Compiler/Linker Mismatch$ export MPICHCC=gcc $ mpicc mpiworld.c $ mpicc myprogram.cSpecifying the Run-time Library Path Compiling on Development Nodes$ mpicc myprogram.c -I/path/to/devel/include $ mpif90 myprogramf90.f90 -I/path/to/devel/include# /etc/ldconfig Run Time Errors With Different MPI ImplementationsTroubleshooting InfiniPath MPI Troubleshooting Using MPI.mod Files When compiling, use descriptive names for the object filesHP-MPI mpirun and executable used together Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting Error Messages Generated by mpirun Messages from the InfiniPath Library$ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Troubleshooting InfiniPath MPI Troubleshooting MPI Messages Following indicates an unknown host There is no route to a valid host$ mpirun -np 2 -m ~/tmp/q mpilatency 100 There is no route to any host One program on one node died$ mpirun -np 2 -m ~/tmp/q mpilatency 100000 $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000 Driver and Link Error Messages Reported by MPI ProgramsMPI Stats Useful Programs and Files for Debugging These two commands perform the same functionsCheck Cluster Homogeneity with ipathcheckout Restarting InfiniPathShell script that gathers status Summary of Useful Programs and FilesModprobe Example contents areIbstatus Ipathcheckout Ipathcontrol Ipathbug-helper Option-qwill query and --qawill query all To check the contents of an RPM, use these commands# lsmod egrep ’ipathibrdmafindex’ $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1Table C-3. statusstr File Example contents for QLogic-built drivers Command strings can also be used. Its format is as followsBooks for Learning MPI Programming References for MPIReference and Source for Slurm InfiniBand OpenFabricsRocks ClustersGlossary GID LID Mtrr SDP Glossary IB6054601-00 D Index OpenFabrics Configuration Configuration of on Suse and Sles