Manuals / Q-Logic / Computer Equipment / Switch

Q-Logic IB6054601-00 D manual InfiniPath User Guide

Models: IB6054601-00 D

1 122

Download 122 pages 48.66 Kb

Page 1

Q

S i m p l i f y

InfiniPath User Guide

Version 2.0

IB6054601-00 D

Page i

Image 1

Q-Logic IB6054601-00 D manual InfiniPath User Guide

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version InfiniPath Software Structure Appendix D Recommended ReadingInfiniPath User Guide Version IB6054601-00 D How this Guide is Organized Who Should Read this GuideOverview SwitchesInteroperability Appendix E Glossary of technical termsPathScale-QLogic Adapter Model Numbers What’s New in this ReleaseInfiniPath/OpenFabrics Supported Distributions and Kernels Supported Distributions and KernelsSoftware Components Documentation and Technical Support Conventions Used in this DocumentThis Guide uses these typographical conventions InfiniPath product documentation includesIntroduction Documentation and Technical Support Page Introduction Installed LayoutMPI include files are Memory FootprintDocumentation can be found Adapter Required Component Optional Memory Footprint Comment Memory Footprint, 331 MB per Node Configuration and Startup Bios SettingsInfiniPath Driver Startup InfiniPath Driver Software ConfigurationInfiniPath Driver Filesystem Layered Ethernet Driver Subnet Management Agent$ ifconfig -a $ ls /sys/class/net Ipathether Configuration on Fedora and RHEL4Create or edit the following file as root ONBOOT=YES Ipathether Configuration on Suse 9.3, Sles 9, and SlesGuid can also be returned by running Determine the MAC address that will be usedIf this file does not exist, skip to Step # lsmod grep -q ipathether modprobe ipathetherAdd the following lines to the file Configuring the IPoIB Network Interface OpenFabrics Configuration and Startup# ifup eth2 # ifconfig eth2 # ifconfig ib0 10.1.17.3 netmask 0xffffff00OpenSM To enable the driver, use the command as root Further Information on Configuring and Loading DriversStarting and Stopping the InfiniPath Software You can stop it again like this# /sbin/chkconfig --list opensmd grep -w on # chkconfig infinipath off# /etc/init.d/infinipath start stop restart $ /sbin/lsmod grep ipathetherSoftware Status Configuring ssh and sshd Using shosts.equivChange the file to mode 600 when finished editing Process Limitation with ssh Remove Unneeded ServicesPerformance and Management Tips # killall -HUP sshdDisable Powersaving Features For Suse 9.3 and 10.0 run this command as root Balanced Processor PowerSDP Module Parameters for Best Performance CPU AffinityHomogeneous Nodes Hyper-ThreadingRun the shell script ipathcontrol as follows IpathbughelperNon-QLogic built Option -qwill query and -qawill query allFor non-QLogic RPMs, it will look like Customer Acceptance Utility Command strings can also be used. Here is a sampleWill produce output like this Displays help messages giving defined usage Page Other MPI Implementations InfiniPath MPIGetting Started with MPI There is more information on the mpihosts file in section An Example C Program$ cp /usr/share/mpich/examples/basic $ mpicc -o cpi cpi.cRun it with Examples Using Other Languages$ mpif77 -o fpi3 fpi3.f $ mpirun -np 2 -m mpihosts ./fpi3Run it Configuring MPI Programs for InfiniPath MPIInfiniPath MPI Details Configuring for ssh Using ssh-agentThis tells ssh that your key pair should let you Edit ~/.ssh/config so that it reads like thisShows how to compile a program Gets verbose output of all the commands in the scriptCompiling and Linking Provides helpTo use gcc for compiling and linking C++ programs use To Use Another CompilerTo use gcc for compiling and linking Fortran77 programs use To use PGI for Fortran90/Fortran95 programs, useCompiler and Linker Variables Cross-compilation Issues$ mpif90 -f90=ifort $ mpif95 -f95=ifort $ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90Running MPI Programs Mpihosts File Environment for Node Programs Console I/O in MPI Programs$ mpirun -np n -m mpihosts -rcfile mpirunrc program Environment for Multiple Versions of InfiniPath or MPINumber of processes to spawn Multiprocessor NodesCreate up to specified number of processes per node $ mpirun -np n -m mpihosts -ppn p program-namePrint mpirun help message. Default Off Print MPI version. Default OffStartup script for setting environment on nodes Default Run each process in an xterm window. Default OffSets the working directory for the node program Disable-mpi-progress-checkMPI Over uDAPL Using Other MPI Implementations# modprobe rdmacm # modprobe rdmaucm $ mpdboot -f hostsfile Using MPDMPD Description File I/O in MPIInfiniPath MPI and Hybrid MPI/OpenMP Applications MPI-IO with RomioUsing Debuggers MPI ErrorsDebugging MPI Programs InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark 1 Measuring MPI Latency Between Two Nodes Benchmark Programs$ mpirun -np 2 -ppn 1 -m mpihosts osulatency Benchmark 2 Measuring MPI Bandwidth Between Two NodesBenchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw Might produce output like this Benchmark 4 Measuring MPI Latency in Host Rings$ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Page Allocating Resources Batch Queuing Script$ batchmpirun -np n mympiprogram Generating the mpihosts FileSimple Process Management Clean Termination of MPI Processes# /sbin/fuser -v /dev/ipath # lsof /dev/ipath# /sbin/fuser -k /dev/ipath Lock Enough Memory on Nodes When Using Slurm# ulimit -l Mechanical and Electrical Considerations Troubleshooting InfiniPath Adapter InstallationSome HTX Motherboards May Need 2 or More CPUs in Use Table C-1. LED Link and Data IndicatorsIncorrect Mtrr Mapping Mtrr Mapping and Write Combining$ ipathpkttest -B Issue with SuperMicro H8DCE-HTe and QHT7040 Change Setting for Mapping MemoryIncorrect Mtrr Mapping Causes Unexpected Low Bandwidth To check your bandwidth tryInstall Warning with RHEL4U2 Software Installation IssuesMpirun Installation Requires 32-bit Support OpenFabrics DependenciesRemove the InfiniPath kernel components with the command Installing Newer Drivers from Other DistributionsCheck your distribution for the exact RPM name $ rpm -e infinipath-kernel --nodepsKernel and Initialization Issues Installing for Your DistributionReload all modules by using this command as root These are used for SLES, SUSE, and FedoraPcimsiquirk Kernel Needs CONFIGPCIMSI=yInfiniPath Interrupts Not Working Driver Load Fails Due to Unsupported KernelNormal output will like similar to this $ grep ibipath /proc/interrupts$ grep -i acpi /proc/cmdline OpenFabrics Load Errors If ibipath Driver Load FailsInfiniPath ibipath Initialization Failure MPI Job Failures Due to Initialization ProblemsLoad and Configure IPoIB Before Loading SDP OpenFabrics IssuesSystem Administration Troubleshooting Stop OpenSM Before Stopping/Restarting InfiniPathMvapich Performance Issues Performance IssuesInfiniPath MPI Troubleshooting Broken Intermediate Link$ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatency $ mpirunCompiler Can’t Find Include, Module or Library Files Compiler/Linker Mismatch$ export MPICHCC=gcc $ mpicc mpiworld.c $ mpicc myprogram.cSpecifying the Run-time Library Path Compiling on Development Nodes$ mpicc myprogram.c -I/path/to/devel/include $ mpif90 myprogramf90.f90 -I/path/to/devel/include# /etc/ldconfig Run Time Errors With Different MPI ImplementationsTroubleshooting InfiniPath MPI Troubleshooting When compiling, use descriptive names for the object files Using MPI.mod FilesHP-MPI mpirun and executable used together Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting Messages from the InfiniPath Library Error Messages Generated by mpirun$ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Troubleshooting InfiniPath MPI Troubleshooting MPI Messages There is no route to a valid host Following indicates an unknown host$ mpirun -np 2 -m ~/tmp/q mpilatency 100 One program on one node died There is no route to any host$ mpirun -np 2 -m ~/tmp/q mpilatency 100000 $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000 Driver and Link Error Messages Reported by MPI ProgramsMPI Stats Useful Programs and Files for Debugging These two commands perform the same functionsCheck Cluster Homogeneity with ipathcheckout Restarting InfiniPathShell script that gathers status Summary of Useful Programs and FilesModprobe Example contents areIbstatus Ipathcheckout Ipathcontrol Ipathbug-helper Option-qwill query and --qawill query all To check the contents of an RPM, use these commands# lsmod egrep ’ipathibrdmafindex’ $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1Table C-3. statusstr File Example contents for QLogic-built drivers Command strings can also be used. Its format is as followsBooks for Learning MPI Programming References for MPIReference and Source for Slurm InfiniBand OpenFabricsRocks ClustersGlossary GID LID Mtrr SDP Glossary IB6054601-00 D Index OpenFabrics Configuration Configuration of on Suse and Sles