Manuals / Q-Logic / Computer Equipment / Switch

Q-Logic IB6054601-00 D manual InfiniPath User Guide Version

Models: IB6054601-00 D

1 122

Download 122 pages 48.66 Kb

Page 8

InfiniPath User Guide Version 2.0

Q

C.4.5	OpenFabrics Load Errors If ib_ipath Driver Load Fails	C-10
C.4.6	InfiniPath ib_ipath Initialization Failure	C-11
C.4.7	MPI Job Failures Due to Initialization Problems	C-11
C.5	OpenFabrics Issues	C-12
C.5.1	Stop OpenSM Before Stopping/Restarting InfiniPath	C-12
C.5.2	Load and Configure IPoIB Before Loading SDP	C-12
C.5.3	Set $IBPATH for OpenFabrics Scripts	C-12
C.6	System Administration Troubleshooting	C-12
C.6.1	Broken Intermediate Link	C-13
C.7	Performance Issues	C-13
C.7.1	MVAPICH Performance Issues	C-13
C.8	InfiniPath MPI Troubleshooting	C-13
C.8.1	Mixed Releases of MPI RPMs	C-13
C.8.2	Cross-compilation Issues	C-14
C.8.3	Compiler/Linker Mismatch	C-15
C.8.4	Compiler Can’t Find Include, Module or Library Files	C-15
C.8.5	Compiling on Development Nodes	C-16
C.8.6	Specifying the Run-time Library Path	C-16
C.8.7	Run Time Errors With Different MPI Implementations	C-17
C.8.8	Process Limitation with ssh	C-19
C.8.9	Using MPI.mod Files	C-19
C.8.10	Extending MPI Modules	C-20
C.8.11	Lock Enough Memory on Nodes When Using a Batch Queuing
	System	C-21
C.8.12	Error Messages Generated by mpirun	C-22
C.8.12.1	Messages from the InfiniPath Library	C-22
C.8.12.2	MPI Messages	C-24
C.8.12.3	Driver and Link Error Messages Reported by MPI Programs	C-27
C.8.13	MPI Stats	C-28
C.9	Useful Programs and Files for Debugging	C-29
C.9.1	Check Cluster Homogeneity with ipath_checkout	C-29
C.9.2	Restarting InfiniPath	C-29
C.9.3	Summary of Useful Programs and Files	C-30
C.9.4	boardversion	C-31
C.9.5	ibstatus	C-32
C.9.6	ibv_devinfo	C-32
C.9.7	ident	C-32
C.9.8	ipath_checkout	C-33
C.9.9	ipath_control	C-34
C.9.10	ipathbug-helper	C-35

Page viii

IB6054601-00 D

Image 8

Q-Logic IB6054601-00 D manual InfiniPath User Guide Version

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version Appendix D Recommended Reading InfiniPath Software StructureInfiniPath User Guide Version IB6054601-00 D Who Should Read this Guide How this Guide is OrganizedSwitches OverviewInteroperability Appendix E Glossary of technical termsWhat’s New in this Release PathScale-QLogic Adapter Model NumbersSupported Distributions and Kernels InfiniPath/OpenFabrics Supported Distributions and KernelsSoftware Components Conventions Used in this Document Documentation and Technical SupportThis Guide uses these typographical conventions InfiniPath product documentation includesIntroduction Documentation and Technical Support Page Installed Layout IntroductionDocumentation can be found Memory FootprintMPI include files are Adapter Required Component Optional Memory Footprint Comment Configuration and Startup Bios Settings Memory Footprint, 331 MB per NodeInfiniPath Driver Filesystem InfiniPath Driver Software ConfigurationInfiniPath Driver Startup Subnet Management Agent Layered Ethernet DriverCreate or edit the following file as root Ipathether Configuration on Fedora and RHEL4$ ifconfig -a $ ls /sys/class/net Ipathether Configuration on Suse 9.3, Sles 9, and Sles ONBOOT=YESDetermine the MAC address that will be used Guid can also be returned by runningIf this file does not exist, skip to Step # lsmod grep -q ipathether modprobe ipathetherAdd the following lines to the file OpenFabrics Configuration and Startup Configuring the IPoIB Network Interface# ifup eth2 # ifconfig eth2 # ifconfig ib0 10.1.17.3 netmask 0xffffff00OpenSM Further Information on Configuring and Loading Drivers To enable the driver, use the command as rootStarting and Stopping the InfiniPath Software You can stop it again like this# chkconfig infinipath off # /sbin/chkconfig --list opensmd grep -w on# /etc/init.d/infinipath start stop restart $ /sbin/lsmod grep ipathetherConfiguring ssh and sshd Using shosts.equiv Software StatusChange the file to mode 600 when finished editing Remove Unneeded Services Process Limitation with sshPerformance and Management Tips # killall -HUP sshdDisable Powersaving Features Balanced Processor Power For Suse 9.3 and 10.0 run this command as rootSDP Module Parameters for Best Performance CPU AffinityHyper-Threading Homogeneous NodesRun the shell script ipathcontrol as follows IpathbughelperFor non-QLogic RPMs, it will look like Option -qwill query and -qawill query allNon-QLogic built Will produce output like this Command strings can also be used. Here is a sampleCustomer Acceptance Utility Displays help messages giving defined usage Page Getting Started with MPI InfiniPath MPIOther MPI Implementations An Example C Program There is more information on the mpihosts file in section$ cp /usr/share/mpich/examples/basic $ mpicc -o cpi cpi.cExamples Using Other Languages Run it with$ mpif77 -o fpi3 fpi3.f $ mpirun -np 2 -m mpihosts ./fpi3Configuring MPI Programs for InfiniPath MPI Run itConfiguring for ssh Using ssh-agent InfiniPath MPI DetailsEdit ~/.ssh/config so that it reads like this This tells ssh that your key pair should let youGets verbose output of all the commands in the script Shows how to compile a programCompiling and Linking Provides helpTo Use Another Compiler To use gcc for compiling and linking C++ programs useTo use gcc for compiling and linking Fortran77 programs use To use PGI for Fortran90/Fortran95 programs, useCross-compilation Issues Compiler and Linker Variables$ mpif90 -f90=ifort $ mpif95 -f95=ifort $ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90Running MPI Programs Mpihosts File Console I/O in MPI Programs Environment for Node ProgramsEnvironment for Multiple Versions of InfiniPath or MPI $ mpirun -np n -m mpihosts -rcfile mpirunrc programMultiprocessor Nodes Number of processes to spawnCreate up to specified number of processes per node $ mpirun -np n -m mpihosts -ppn p program-namePrint MPI version. Default Off Print mpirun help message. Default OffStartup script for setting environment on nodes Default Run each process in an xterm window. Default OffDisable-mpi-progress-check Sets the working directory for the node program# modprobe rdmacm # modprobe rdmaucm Using Other MPI ImplementationsMPI Over uDAPL Using MPD $ mpdboot -f hostsfileMPD Description File I/O in MPIMPI-IO with Romio InfiniPath MPI and Hybrid MPI/OpenMP ApplicationsDebugging MPI Programs MPI ErrorsUsing Debuggers InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark Programs Benchmark 1 Measuring MPI Latency Between Two NodesBenchmark 2 Measuring MPI Bandwidth Between Two Nodes $ mpirun -np 2 -ppn 1 -m mpihosts osulatencyBenchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw $ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Benchmark 4 Measuring MPI Latency in Host RingsMight produce output like this Page Batch Queuing Script Allocating ResourcesGenerating the mpihosts File $ batchmpirun -np n mympiprogramClean Termination of MPI Processes Simple Process Management# /sbin/fuser -v /dev/ipath # lsof /dev/ipath# ulimit -l Lock Enough Memory on Nodes When Using Slurm# /sbin/fuser -k /dev/ipath Troubleshooting InfiniPath Adapter Installation Mechanical and Electrical ConsiderationsTable C-1. LED Link and Data Indicators Some HTX Motherboards May Need 2 or More CPUs in Use$ ipathpkttest -B Mtrr Mapping and Write CombiningIncorrect Mtrr Mapping Change Setting for Mapping Memory Issue with SuperMicro H8DCE-HTe and QHT7040Incorrect Mtrr Mapping Causes Unexpected Low Bandwidth To check your bandwidth trySoftware Installation Issues Install Warning with RHEL4U2Mpirun Installation Requires 32-bit Support OpenFabrics DependenciesInstalling Newer Drivers from Other Distributions Remove the InfiniPath kernel components with the commandCheck your distribution for the exact RPM name $ rpm -e infinipath-kernel --nodepsInstalling for Your Distribution Kernel and Initialization IssuesReload all modules by using this command as root These are used for SLES, SUSE, and FedoraKernel Needs CONFIGPCIMSI=y PcimsiquirkDriver Load Fails Due to Unsupported Kernel InfiniPath Interrupts Not WorkingNormal output will like similar to this $ grep ibipath /proc/interruptsOpenFabrics Load Errors If ibipath Driver Load Fails $ grep -i acpi /proc/cmdlineMPI Job Failures Due to Initialization Problems InfiniPath ibipath Initialization FailureOpenFabrics Issues Load and Configure IPoIB Before Loading SDPSystem Administration Troubleshooting Stop OpenSM Before Stopping/Restarting InfiniPathPerformance Issues Mvapich Performance IssuesInfiniPath MPI Troubleshooting Broken Intermediate Link$ mpirun $ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatencyCompiler/Linker Mismatch Compiler Can’t Find Include, Module or Library Files$ export MPICHCC=gcc $ mpicc mpiworld.c $ mpicc myprogram.cCompiling on Development Nodes Specifying the Run-time Library Path$ mpicc myprogram.c -I/path/to/devel/include $ mpif90 myprogramf90.f90 -I/path/to/devel/includeRun Time Errors With Different MPI Implementations # /etc/ldconfigTroubleshooting InfiniPath MPI Troubleshooting HP-MPI mpirun and executable used together Using MPI.mod FilesWhen compiling, use descriptive names for the object files Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting $ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Error Messages Generated by mpirunMessages from the InfiniPath Library Troubleshooting InfiniPath MPI Troubleshooting MPI Messages $ mpirun -np 2 -m ~/tmp/q mpilatency 100 Following indicates an unknown hostThere is no route to a valid host $ mpirun -np 2 -m ~/tmp/q mpilatency 100000 There is no route to any hostOne program on one node died Driver and Link Error Messages Reported by MPI Programs $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000MPI Stats These two commands perform the same functions Useful Programs and Files for DebuggingCheck Cluster Homogeneity with ipathcheckout Restarting InfiniPathSummary of Useful Programs and Files Shell script that gathers statusExample contents are ModprobeIbstatus Ipathcheckout Ipathcontrol Ipathbug-helper To check the contents of an RPM, use these commands Option-qwill query and --qawill query all# lsmod egrep ’ipathibrdmafindex’ $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1Table C-3. statusstr File Command strings can also be used. Its format is as follows Example contents for QLogic-built driversReferences for MPI Books for Learning MPI ProgrammingReference and Source for Slurm InfiniBand OpenFabricsClusters RocksGlossary GID LID Mtrr SDP Glossary IB6054601-00 D Index Configuration of on Suse and Sles OpenFabrics Configuration