Manuals / Q-Logic / Computer Equipment / Switch

Q-Logic IB6054601-00 D manual

Models: IB6054601-00 D

1 122

Download 122 pages 48.66 Kb

Page 18

1 – Introduction

Documentation and Technical Support

Notes

Q

1-8	IB6054601-00 D

Image 18

Q-Logic IB6054601-00 D manual

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version Appendix D Recommended Reading InfiniPath Software StructureInfiniPath User Guide Version IB6054601-00 D Who Should Read this Guide How this Guide is OrganizedInteroperability SwitchesOverview Appendix E Glossary of technical termsWhat’s New in this Release PathScale-QLogic Adapter Model NumbersSupported Distributions and Kernels InfiniPath/OpenFabrics Supported Distributions and KernelsSoftware Components This Guide uses these typographical conventions Conventions Used in this DocumentDocumentation and Technical Support InfiniPath product documentation includesIntroduction Documentation and Technical Support Page Installed Layout IntroductionMemory Footprint MPI include files areDocumentation can be found Adapter Required Component Optional Memory Footprint Comment Configuration and Startup Bios Settings Memory Footprint, 331 MB per NodeInfiniPath Driver Software Configuration InfiniPath Driver StartupInfiniPath Driver Filesystem Subnet Management Agent Layered Ethernet DriverIpathether Configuration on Fedora and RHEL4 $ ifconfig -a $ ls /sys/class/netCreate or edit the following file as root Ipathether Configuration on Suse 9.3, Sles 9, and Sles ONBOOT=YESIf this file does not exist, skip to Step Determine the MAC address that will be usedGuid can also be returned by running # lsmod grep -q ipathether modprobe ipathetherAdd the following lines to the file # ifup eth2 # ifconfig eth2 OpenFabrics Configuration and StartupConfiguring the IPoIB Network Interface # ifconfig ib0 10.1.17.3 netmask 0xffffff00OpenSM Starting and Stopping the InfiniPath Software Further Information on Configuring and Loading DriversTo enable the driver, use the command as root You can stop it again like this# /etc/init.d/infinipath start stop restart # chkconfig infinipath off# /sbin/chkconfig --list opensmd grep -w on $ /sbin/lsmod grep ipathetherConfiguring ssh and sshd Using shosts.equiv Software StatusChange the file to mode 600 when finished editing Performance and Management Tips Remove Unneeded ServicesProcess Limitation with ssh # killall -HUP sshdDisable Powersaving Features SDP Module Parameters for Best Performance Balanced Processor PowerFor Suse 9.3 and 10.0 run this command as root CPU AffinityRun the shell script ipathcontrol as follows Hyper-ThreadingHomogeneous Nodes IpathbughelperOption -qwill query and -qawill query all Non-QLogic builtFor non-QLogic RPMs, it will look like Command strings can also be used. Here is a sample Customer Acceptance UtilityWill produce output like this Displays help messages giving defined usage Page InfiniPath MPI Other MPI ImplementationsGetting Started with MPI $ cp /usr/share/mpich/examples/basic An Example C ProgramThere is more information on the mpihosts file in section $ mpicc -o cpi cpi.c$ mpif77 -o fpi3 fpi3.f Examples Using Other LanguagesRun it with $ mpirun -np 2 -m mpihosts ./fpi3Configuring MPI Programs for InfiniPath MPI Run itConfiguring for ssh Using ssh-agent InfiniPath MPI DetailsEdit ~/.ssh/config so that it reads like this This tells ssh that your key pair should let youCompiling and Linking Gets verbose output of all the commands in the scriptShows how to compile a program Provides helpTo use gcc for compiling and linking Fortran77 programs use To Use Another CompilerTo use gcc for compiling and linking C++ programs use To use PGI for Fortran90/Fortran95 programs, use$ mpif90 -f90=ifort $ mpif95 -f95=ifort Cross-compilation IssuesCompiler and Linker Variables $ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90Running MPI Programs Mpihosts File Console I/O in MPI Programs Environment for Node ProgramsEnvironment for Multiple Versions of InfiniPath or MPI $ mpirun -np n -m mpihosts -rcfile mpirunrc programCreate up to specified number of processes per node Multiprocessor NodesNumber of processes to spawn $ mpirun -np n -m mpihosts -ppn p program-nameStartup script for setting environment on nodes Default Print MPI version. Default OffPrint mpirun help message. Default Off Run each process in an xterm window. Default OffDisable-mpi-progress-check Sets the working directory for the node programUsing Other MPI Implementations MPI Over uDAPL# modprobe rdmacm # modprobe rdmaucm MPD Description Using MPD$ mpdboot -f hostsfile File I/O in MPIMPI-IO with Romio InfiniPath MPI and Hybrid MPI/OpenMP ApplicationsMPI Errors Using DebuggersDebugging MPI Programs InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark Programs Benchmark 1 Measuring MPI Latency Between Two NodesBenchmark 2 Measuring MPI Bandwidth Between Two Nodes $ mpirun -np 2 -ppn 1 -m mpihosts osulatencyBenchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw Benchmark 4 Measuring MPI Latency in Host Rings Might produce output like this$ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Page Batch Queuing Script Allocating ResourcesGenerating the mpihosts File $ batchmpirun -np n mympiprogram# /sbin/fuser -v /dev/ipath Clean Termination of MPI ProcessesSimple Process Management # lsof /dev/ipathLock Enough Memory on Nodes When Using Slurm # /sbin/fuser -k /dev/ipath# ulimit -l Troubleshooting InfiniPath Adapter Installation Mechanical and Electrical ConsiderationsTable C-1. LED Link and Data Indicators Some HTX Motherboards May Need 2 or More CPUs in UseMtrr Mapping and Write Combining Incorrect Mtrr Mapping$ ipathpkttest -B Incorrect Mtrr Mapping Causes Unexpected Low Bandwidth Change Setting for Mapping MemoryIssue with SuperMicro H8DCE-HTe and QHT7040 To check your bandwidth tryMpirun Installation Requires 32-bit Support Software Installation IssuesInstall Warning with RHEL4U2 OpenFabrics DependenciesCheck your distribution for the exact RPM name Installing Newer Drivers from Other DistributionsRemove the InfiniPath kernel components with the command $ rpm -e infinipath-kernel --nodepsReload all modules by using this command as root Installing for Your DistributionKernel and Initialization Issues These are used for SLES, SUSE, and FedoraKernel Needs CONFIGPCIMSI=y PcimsiquirkNormal output will like similar to this Driver Load Fails Due to Unsupported KernelInfiniPath Interrupts Not Working $ grep ibipath /proc/interruptsOpenFabrics Load Errors If ibipath Driver Load Fails $ grep -i acpi /proc/cmdlineMPI Job Failures Due to Initialization Problems InfiniPath ibipath Initialization FailureSystem Administration Troubleshooting OpenFabrics IssuesLoad and Configure IPoIB Before Loading SDP Stop OpenSM Before Stopping/Restarting InfiniPathInfiniPath MPI Troubleshooting Performance IssuesMvapich Performance Issues Broken Intermediate Link$ mpirun $ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatency$ export MPICHCC=gcc $ mpicc mpiworld.c Compiler/Linker MismatchCompiler Can’t Find Include, Module or Library Files $ mpicc myprogram.c$ mpicc myprogram.c -I/path/to/devel/include Compiling on Development NodesSpecifying the Run-time Library Path $ mpif90 myprogramf90.f90 -I/path/to/devel/includeRun Time Errors With Different MPI Implementations # /etc/ldconfigTroubleshooting InfiniPath MPI Troubleshooting Using MPI.mod Files When compiling, use descriptive names for the object filesHP-MPI mpirun and executable used together Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting Error Messages Generated by mpirun Messages from the InfiniPath Library$ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Troubleshooting InfiniPath MPI Troubleshooting MPI Messages Following indicates an unknown host There is no route to a valid host$ mpirun -np 2 -m ~/tmp/q mpilatency 100 There is no route to any host One program on one node died$ mpirun -np 2 -m ~/tmp/q mpilatency 100000 Driver and Link Error Messages Reported by MPI Programs $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000MPI Stats Check Cluster Homogeneity with ipathcheckout These two commands perform the same functionsUseful Programs and Files for Debugging Restarting InfiniPathSummary of Useful Programs and Files Shell script that gathers statusExample contents are ModprobeIbstatus Ipathcheckout Ipathcontrol Ipathbug-helper # lsmod egrep ’ipathibrdmafindex’ To check the contents of an RPM, use these commandsOption-qwill query and --qawill query all $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1Table C-3. statusstr File Command strings can also be used. Its format is as follows Example contents for QLogic-built driversReference and Source for Slurm References for MPIBooks for Learning MPI Programming InfiniBand OpenFabricsClusters RocksGlossary GID LID Mtrr SDP Glossary IB6054601-00 D Index Configuration of on Suse and Sles OpenFabrics Configuration