Manuals / Q-Logic / Computer Equipment / Switch

Q-Logic IB6054601-00 D manual

Models: IB6054601-00 D

1 122

Download 122 pages 48.66 Kb

Page 70

A – Benchmark Programs

Benchmark 4: Measuring MPI Latency in Host Rings

Notes

Q

A-6	IB6054601-00 D

Image 70

Q-Logic IB6054601-00 D manual

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version Appendix D Recommended Reading InfiniPath Software StructureInfiniPath User Guide Version IB6054601-00 D Who Should Read this Guide How this Guide is OrganizedInteroperability SwitchesOverview Appendix E Glossary of technical termsWhat’s New in this Release PathScale-QLogic Adapter Model NumbersSupported Distributions and Kernels InfiniPath/OpenFabrics Supported Distributions and KernelsSoftware Components This Guide uses these typographical conventions Conventions Used in this DocumentDocumentation and Technical Support InfiniPath product documentation includesIntroduction Documentation and Technical Support Page Installed Layout IntroductionMPI include files are Memory FootprintDocumentation can be found Adapter Required Component Optional Memory Footprint Comment Configuration and Startup Bios Settings Memory Footprint, 331 MB per NodeInfiniPath Driver Startup InfiniPath Driver Software ConfigurationInfiniPath Driver Filesystem Subnet Management Agent Layered Ethernet Driver$ ifconfig -a $ ls /sys/class/net Ipathether Configuration on Fedora and RHEL4Create or edit the following file as root Ipathether Configuration on Suse 9.3, Sles 9, and Sles ONBOOT=YESIf this file does not exist, skip to Step Determine the MAC address that will be usedGuid can also be returned by running # lsmod grep -q ipathether modprobe ipathetherAdd the following lines to the file # ifup eth2 # ifconfig eth2 OpenFabrics Configuration and StartupConfiguring the IPoIB Network Interface # ifconfig ib0 10.1.17.3 netmask 0xffffff00OpenSM Starting and Stopping the InfiniPath Software Further Information on Configuring and Loading DriversTo enable the driver, use the command as root You can stop it again like this# /etc/init.d/infinipath start stop restart # chkconfig infinipath off# /sbin/chkconfig --list opensmd grep -w on $ /sbin/lsmod grep ipathetherConfiguring ssh and sshd Using shosts.equiv Software StatusChange the file to mode 600 when finished editing Performance and Management Tips Remove Unneeded ServicesProcess Limitation with ssh # killall -HUP sshdDisable Powersaving Features SDP Module Parameters for Best Performance Balanced Processor PowerFor Suse 9.3 and 10.0 run this command as root CPU AffinityRun the shell script ipathcontrol as follows Hyper-ThreadingHomogeneous Nodes IpathbughelperNon-QLogic built Option -qwill query and -qawill query allFor non-QLogic RPMs, it will look like Customer Acceptance Utility Command strings can also be used. Here is a sampleWill produce output like this Displays help messages giving defined usage Page Other MPI Implementations InfiniPath MPIGetting Started with MPI $ cp /usr/share/mpich/examples/basic An Example C ProgramThere is more information on the mpihosts file in section $ mpicc -o cpi cpi.c$ mpif77 -o fpi3 fpi3.f Examples Using Other LanguagesRun it with $ mpirun -np 2 -m mpihosts ./fpi3Configuring MPI Programs for InfiniPath MPI Run itConfiguring for ssh Using ssh-agent InfiniPath MPI DetailsEdit ~/.ssh/config so that it reads like this This tells ssh that your key pair should let youCompiling and Linking Gets verbose output of all the commands in the scriptShows how to compile a program Provides helpTo use gcc for compiling and linking Fortran77 programs use To Use Another CompilerTo use gcc for compiling and linking C++ programs use To use PGI for Fortran90/Fortran95 programs, use$ mpif90 -f90=ifort $ mpif95 -f95=ifort Cross-compilation IssuesCompiler and Linker Variables $ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90Running MPI Programs Mpihosts File Console I/O in MPI Programs Environment for Node ProgramsEnvironment for Multiple Versions of InfiniPath or MPI $ mpirun -np n -m mpihosts -rcfile mpirunrc programCreate up to specified number of processes per node Multiprocessor NodesNumber of processes to spawn $ mpirun -np n -m mpihosts -ppn p program-nameStartup script for setting environment on nodes Default Print MPI version. Default OffPrint mpirun help message. Default Off Run each process in an xterm window. Default OffDisable-mpi-progress-check Sets the working directory for the node programMPI Over uDAPL Using Other MPI Implementations# modprobe rdmacm # modprobe rdmaucm MPD Description Using MPD$ mpdboot -f hostsfile File I/O in MPIMPI-IO with Romio InfiniPath MPI and Hybrid MPI/OpenMP ApplicationsUsing Debuggers MPI ErrorsDebugging MPI Programs InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark Programs Benchmark 1 Measuring MPI Latency Between Two NodesBenchmark 2 Measuring MPI Bandwidth Between Two Nodes $ mpirun -np 2 -ppn 1 -m mpihosts osulatencyBenchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw Might produce output like this Benchmark 4 Measuring MPI Latency in Host Rings$ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Page Batch Queuing Script Allocating ResourcesGenerating the mpihosts File $ batchmpirun -np n mympiprogram# /sbin/fuser -v /dev/ipath Clean Termination of MPI ProcessesSimple Process Management # lsof /dev/ipath# /sbin/fuser -k /dev/ipath Lock Enough Memory on Nodes When Using Slurm# ulimit -l Troubleshooting InfiniPath Adapter Installation Mechanical and Electrical ConsiderationsTable C-1. LED Link and Data Indicators Some HTX Motherboards May Need 2 or More CPUs in UseIncorrect Mtrr Mapping Mtrr Mapping and Write Combining$ ipathpkttest -B Incorrect Mtrr Mapping Causes Unexpected Low Bandwidth Change Setting for Mapping MemoryIssue with SuperMicro H8DCE-HTe and QHT7040 To check your bandwidth tryMpirun Installation Requires 32-bit Support Software Installation IssuesInstall Warning with RHEL4U2 OpenFabrics DependenciesCheck your distribution for the exact RPM name Installing Newer Drivers from Other DistributionsRemove the InfiniPath kernel components with the command $ rpm -e infinipath-kernel --nodepsReload all modules by using this command as root Installing for Your DistributionKernel and Initialization Issues These are used for SLES, SUSE, and FedoraKernel Needs CONFIGPCIMSI=y PcimsiquirkNormal output will like similar to this Driver Load Fails Due to Unsupported KernelInfiniPath Interrupts Not Working $ grep ibipath /proc/interruptsOpenFabrics Load Errors If ibipath Driver Load Fails $ grep -i acpi /proc/cmdlineMPI Job Failures Due to Initialization Problems InfiniPath ibipath Initialization FailureSystem Administration Troubleshooting OpenFabrics IssuesLoad and Configure IPoIB Before Loading SDP Stop OpenSM Before Stopping/Restarting InfiniPathInfiniPath MPI Troubleshooting Performance IssuesMvapich Performance Issues Broken Intermediate Link$ mpirun $ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatency$ export MPICHCC=gcc $ mpicc mpiworld.c Compiler/Linker MismatchCompiler Can’t Find Include, Module or Library Files $ mpicc myprogram.c$ mpicc myprogram.c -I/path/to/devel/include Compiling on Development NodesSpecifying the Run-time Library Path $ mpif90 myprogramf90.f90 -I/path/to/devel/includeRun Time Errors With Different MPI Implementations # /etc/ldconfigTroubleshooting InfiniPath MPI Troubleshooting When compiling, use descriptive names for the object files Using MPI.mod FilesHP-MPI mpirun and executable used together Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting Messages from the InfiniPath Library Error Messages Generated by mpirun$ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Troubleshooting InfiniPath MPI Troubleshooting MPI Messages There is no route to a valid host Following indicates an unknown host$ mpirun -np 2 -m ~/tmp/q mpilatency 100 One program on one node died There is no route to any host$ mpirun -np 2 -m ~/tmp/q mpilatency 100000 Driver and Link Error Messages Reported by MPI Programs $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000MPI Stats Check Cluster Homogeneity with ipathcheckout These two commands perform the same functionsUseful Programs and Files for Debugging Restarting InfiniPathSummary of Useful Programs and Files Shell script that gathers statusExample contents are ModprobeIbstatus Ipathcheckout Ipathcontrol Ipathbug-helper # lsmod egrep ’ipathibrdmafindex’ To check the contents of an RPM, use these commandsOption-qwill query and --qawill query all $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1Table C-3. statusstr File Command strings can also be used. Its format is as follows Example contents for QLogic-built driversReference and Source for Slurm References for MPIBooks for Learning MPI Programming InfiniBand OpenFabricsClusters RocksGlossary GID LID Mtrr SDP Glossary IB6054601-00 D Index Configuration of on Suse and Sles OpenFabrics Configuration