Manuals / Q-Logic / Computer Equipment / Switch

Q-Logic IB6054601-00 D manual Overview, Switches, Interoperability

Models: IB6054601-00 D

1 122

Download 122 pages 48.66 Kb

Page 12

1 – Introduction Interoperability

Q

■Appendix E Glossary of technical terms

■Index

In addition, the InfiniPath Install Guide contains information on InfiniPath hardware and software installation.

1.3

Overview

The material in this documentation pertains to an InfiniPath cluster. This is defined as a collection of nodes, each attached to an InfiniBand™-based fabric through the InfiniPath Interconnect. The nodes are Linux-based computers, each having up to eight processors.

The InfiniPath interconnect is InfiniBand 4X, with a raw data rate of 10 Gb/s (data rate of 8Gb/s).

InfiniPath utilizes standard, off-the-shelf InfiniBand 4X switches and cabling. InfiniPath OpenFabrics software is interoperable with other vendors’ InfiniBand HCAs running compatible OpenFabrics releases. There are two options for Subnet Management in your cluster:

■Use the Subnet Manager on one or more managed switches supplied with your Infiniband switches.

■Use the OpenSM component of OpenFabrics.

1.4

Switches

The InfiniPath interconnect is designed to work with all InfiniBand-compliant switches. Use of OpenSM as a subnet manager is now supported. OpenSM is part of the OpenFabrics component of this release.

1.5

Interoperability

InfiniPath participates in the standard InfiniBand Subnet Management protocols for configuration and monitoring. InfiniPath OpenFabrics (including IPoIB) is interoperable with other vendors’ InfiniBand HCAs running compatible OpenFabrics releases. The InfiniPath MPI and Ethernet emulation stacks (ipath_ether) are not interoperable with other InfiniBand Host Channel Adapters (HCA) and Target Channel Adapters (TCA). Instead, InfiniPath uses an InfiniBand-compliant vendor-specific protocol that is highly optimized for MPI and TCP between InfiniPath-equipped hosts.

1-2	IB6054601-00 D

Image 12

Q-Logic IB6054601-00 D manual Overview, Switches, Interoperability, Appendix E Glossary of technical terms

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version Appendix D Recommended Reading InfiniPath Software StructureInfiniPath User Guide Version IB6054601-00 D Who Should Read this Guide How this Guide is OrganizedSwitches OverviewInteroperability Appendix E Glossary of technical termsWhat’s New in this Release PathScale-QLogic Adapter Model NumbersSupported Distributions and Kernels InfiniPath/OpenFabrics Supported Distributions and KernelsSoftware Components Conventions Used in this Document Documentation and Technical SupportThis Guide uses these typographical conventions InfiniPath product documentation includesIntroduction Documentation and Technical Support Page Installed Layout IntroductionMemory Footprint MPI include files areDocumentation can be found Adapter Required Component Optional Memory Footprint Comment Configuration and Startup Bios Settings Memory Footprint, 331 MB per NodeInfiniPath Driver Software Configuration InfiniPath Driver StartupInfiniPath Driver Filesystem Subnet Management Agent Layered Ethernet DriverIpathether Configuration on Fedora and RHEL4 $ ifconfig -a $ ls /sys/class/netCreate or edit the following file as root Ipathether Configuration on Suse 9.3, Sles 9, and Sles ONBOOT=YESDetermine the MAC address that will be used Guid can also be returned by runningIf this file does not exist, skip to Step # lsmod grep -q ipathether modprobe ipathetherAdd the following lines to the file OpenFabrics Configuration and Startup Configuring the IPoIB Network Interface# ifup eth2 # ifconfig eth2 # ifconfig ib0 10.1.17.3 netmask 0xffffff00OpenSM Further Information on Configuring and Loading Drivers To enable the driver, use the command as rootStarting and Stopping the InfiniPath Software You can stop it again like this# chkconfig infinipath off # /sbin/chkconfig --list opensmd grep -w on# /etc/init.d/infinipath start stop restart $ /sbin/lsmod grep ipathetherConfiguring ssh and sshd Using shosts.equiv Software StatusChange the file to mode 600 when finished editing Remove Unneeded Services Process Limitation with sshPerformance and Management Tips # killall -HUP sshdDisable Powersaving Features Balanced Processor Power For Suse 9.3 and 10.0 run this command as rootSDP Module Parameters for Best Performance CPU AffinityHyper-Threading Homogeneous NodesRun the shell script ipathcontrol as follows IpathbughelperOption -qwill query and -qawill query all Non-QLogic builtFor non-QLogic RPMs, it will look like Command strings can also be used. Here is a sample Customer Acceptance UtilityWill produce output like this Displays help messages giving defined usage Page InfiniPath MPI Other MPI ImplementationsGetting Started with MPI An Example C Program There is more information on the mpihosts file in section$ cp /usr/share/mpich/examples/basic $ mpicc -o cpi cpi.cExamples Using Other Languages Run it with$ mpif77 -o fpi3 fpi3.f $ mpirun -np 2 -m mpihosts ./fpi3Configuring MPI Programs for InfiniPath MPI Run itConfiguring for ssh Using ssh-agent InfiniPath MPI DetailsEdit ~/.ssh/config so that it reads like this This tells ssh that your key pair should let youGets verbose output of all the commands in the script Shows how to compile a programCompiling and Linking Provides helpTo Use Another Compiler To use gcc for compiling and linking C++ programs useTo use gcc for compiling and linking Fortran77 programs use To use PGI for Fortran90/Fortran95 programs, useCross-compilation Issues Compiler and Linker Variables$ mpif90 -f90=ifort $ mpif95 -f95=ifort $ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90Running MPI Programs Mpihosts File Console I/O in MPI Programs Environment for Node ProgramsEnvironment for Multiple Versions of InfiniPath or MPI $ mpirun -np n -m mpihosts -rcfile mpirunrc programMultiprocessor Nodes Number of processes to spawnCreate up to specified number of processes per node $ mpirun -np n -m mpihosts -ppn p program-namePrint MPI version. Default Off Print mpirun help message. Default OffStartup script for setting environment on nodes Default Run each process in an xterm window. Default OffDisable-mpi-progress-check Sets the working directory for the node programUsing Other MPI Implementations MPI Over uDAPL# modprobe rdmacm # modprobe rdmaucm Using MPD $ mpdboot -f hostsfileMPD Description File I/O in MPIMPI-IO with Romio InfiniPath MPI and Hybrid MPI/OpenMP ApplicationsMPI Errors Using DebuggersDebugging MPI Programs InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark Programs Benchmark 1 Measuring MPI Latency Between Two NodesBenchmark 2 Measuring MPI Bandwidth Between Two Nodes $ mpirun -np 2 -ppn 1 -m mpihosts osulatencyBenchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw Benchmark 4 Measuring MPI Latency in Host Rings Might produce output like this$ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Page Batch Queuing Script Allocating ResourcesGenerating the mpihosts File $ batchmpirun -np n mympiprogramClean Termination of MPI Processes Simple Process Management# /sbin/fuser -v /dev/ipath # lsof /dev/ipathLock Enough Memory on Nodes When Using Slurm # /sbin/fuser -k /dev/ipath# ulimit -l Troubleshooting InfiniPath Adapter Installation Mechanical and Electrical ConsiderationsTable C-1. LED Link and Data Indicators Some HTX Motherboards May Need 2 or More CPUs in UseMtrr Mapping and Write Combining Incorrect Mtrr Mapping$ ipathpkttest -B Change Setting for Mapping Memory Issue with SuperMicro H8DCE-HTe and QHT7040Incorrect Mtrr Mapping Causes Unexpected Low Bandwidth To check your bandwidth trySoftware Installation Issues Install Warning with RHEL4U2Mpirun Installation Requires 32-bit Support OpenFabrics DependenciesInstalling Newer Drivers from Other Distributions Remove the InfiniPath kernel components with the commandCheck your distribution for the exact RPM name $ rpm -e infinipath-kernel --nodepsInstalling for Your Distribution Kernel and Initialization IssuesReload all modules by using this command as root These are used for SLES, SUSE, and FedoraKernel Needs CONFIGPCIMSI=y PcimsiquirkDriver Load Fails Due to Unsupported Kernel InfiniPath Interrupts Not WorkingNormal output will like similar to this $ grep ibipath /proc/interruptsOpenFabrics Load Errors If ibipath Driver Load Fails $ grep -i acpi /proc/cmdlineMPI Job Failures Due to Initialization Problems InfiniPath ibipath Initialization FailureOpenFabrics Issues Load and Configure IPoIB Before Loading SDPSystem Administration Troubleshooting Stop OpenSM Before Stopping/Restarting InfiniPathPerformance Issues Mvapich Performance IssuesInfiniPath MPI Troubleshooting Broken Intermediate Link$ mpirun $ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatencyCompiler/Linker Mismatch Compiler Can’t Find Include, Module or Library Files$ export MPICHCC=gcc $ mpicc mpiworld.c $ mpicc myprogram.cCompiling on Development Nodes Specifying the Run-time Library Path$ mpicc myprogram.c -I/path/to/devel/include $ mpif90 myprogramf90.f90 -I/path/to/devel/includeRun Time Errors With Different MPI Implementations # /etc/ldconfigTroubleshooting InfiniPath MPI Troubleshooting Using MPI.mod Files When compiling, use descriptive names for the object filesHP-MPI mpirun and executable used together Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting Error Messages Generated by mpirun Messages from the InfiniPath Library$ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Troubleshooting InfiniPath MPI Troubleshooting MPI Messages Following indicates an unknown host There is no route to a valid host$ mpirun -np 2 -m ~/tmp/q mpilatency 100 There is no route to any host One program on one node died$ mpirun -np 2 -m ~/tmp/q mpilatency 100000 Driver and Link Error Messages Reported by MPI Programs $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000MPI Stats These two commands perform the same functions Useful Programs and Files for DebuggingCheck Cluster Homogeneity with ipathcheckout Restarting InfiniPathSummary of Useful Programs and Files Shell script that gathers statusExample contents are ModprobeIbstatus Ipathcheckout Ipathcontrol Ipathbug-helper To check the contents of an RPM, use these commands Option-qwill query and --qawill query all# lsmod egrep ’ipathibrdmafindex’ $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1Table C-3. statusstr File Command strings can also be used. Its format is as follows Example contents for QLogic-built driversReferences for MPI Books for Learning MPI ProgrammingReference and Source for Slurm InfiniBand OpenFabricsClusters RocksGlossary GID LID Mtrr SDP Glossary IB6054601-00 D Index Configuration of on Suse and Sles OpenFabrics Configuration