Manuals / Q-Logic / Computer Equipment / Switch

Q-Logic IB6054601-00 D manual Using Other MPI Implementations, MPI Over uDAPL

Models: IB6054601-00 D

1 122

Download 122 pages 48.66 Kb

Page 59

Q

3 – Using InfiniPath MPI MPD

-statsfile file-prefix

Specifies alternate file to receive the output from the -print-statsoption. Default: stderr

3.6

Using Other MPI Implementations

Support for multiple MPI implementations has been added. You can use a different version of MPI and achieve the high-bandwidth and low-latency performance that it is standard with InfiniPath MPI.

The currently supported implementations are HP-MPI, OpenMPI and Scali.

These MPI implementations will run on multiple interconnects, and have their own mechanisms for selecting which one you will run on. Please see the documentation provided with the version of MPI that you wish to use.

If you have downloaded and installed another MPI implementation, you will need to set your PATH up to pick up the version of MPI you wish to use.

You will also need to set LD_LIBRARY_PATH, both in your local environment and in an rcfile (such as .mpirunrc, .bashrc or .cshrc) that will be invoked on remote nodes. See section 3.5.8 and section 3.5.3.1 for information on setting up your environment and section C.8.6 for information on setting your run-time library path. See also section C.8.7 for information on run time errors that may occur if there are MPI version mismatches.

3.7

MPI Over uDAPL

Some MPI implementations can be run over uDAPL. uDAPL is the user mode version of the Direct Access Provider Library (DAPL). Examples of such MPI implementations are Intel MPI and one option on OpenMPI.

If you are running such an MPI implementation, the rdma_cm and rdma_ucm modules will need to be loaded. To test these modules, use these commands (as root):

#modprobe rdma_cm

#modprobe rdma_ucm

To ensure that the modules are loaded whenever the driver is loaded, add rdma_cm and rdma_ucm to the OPENFABRICS_MODULES assignment in

/etc/sysconfig/infinipath.

3.8

MPD

MPD is an alternative to mpirun for launching MPI jobs. It is described briefly in the following sections.

IB6054601-00 D

3-17

Image 59

Q-Logic IB6054601-00 D manual Using Other MPI Implementations, MPI Over uDAPL, # modprobe rdmacm # modprobe rdmaucm

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version InfiniPath Software Structure Appendix D Recommended ReadingInfiniPath User Guide Version IB6054601-00 D How this Guide is Organized Who Should Read this GuideAppendix E Glossary of technical terms SwitchesOverview InteroperabilityPathScale-QLogic Adapter Model Numbers What’s New in this ReleaseInfiniPath/OpenFabrics Supported Distributions and Kernels Supported Distributions and KernelsSoftware Components InfiniPath product documentation includes Conventions Used in this DocumentDocumentation and Technical Support This Guide uses these typographical conventionsIntroduction Documentation and Technical Support Page Introduction Installed LayoutDocumentation can be found Memory FootprintMPI include files are Adapter Required Component Optional Memory Footprint Comment Memory Footprint, 331 MB per Node Configuration and Startup Bios SettingsInfiniPath Driver Filesystem InfiniPath Driver Software ConfigurationInfiniPath Driver Startup Layered Ethernet Driver Subnet Management AgentCreate or edit the following file as root Ipathether Configuration on Fedora and RHEL4$ ifconfig -a $ ls /sys/class/net ONBOOT=YES Ipathether Configuration on Suse 9.3, Sles 9, and Sles# lsmod grep -q ipathether modprobe ipathether Determine the MAC address that will be usedGuid can also be returned by running If this file does not exist, skip to StepAdd the following lines to the file # ifconfig ib0 10.1.17.3 netmask 0xffffff00 OpenFabrics Configuration and StartupConfiguring the IPoIB Network Interface # ifup eth2 # ifconfig eth2OpenSM You can stop it again like this Further Information on Configuring and Loading DriversTo enable the driver, use the command as root Starting and Stopping the InfiniPath Software$ /sbin/lsmod grep ipathether # chkconfig infinipath off# /sbin/chkconfig --list opensmd grep -w on # /etc/init.d/infinipath start stop restartSoftware Status Configuring ssh and sshd Using shosts.equivChange the file to mode 600 when finished editing # killall -HUP sshd Remove Unneeded ServicesProcess Limitation with ssh Performance and Management TipsDisable Powersaving Features CPU Affinity Balanced Processor PowerFor Suse 9.3 and 10.0 run this command as root SDP Module Parameters for Best PerformanceIpathbughelper Hyper-ThreadingHomogeneous Nodes Run the shell script ipathcontrol as followsFor non-QLogic RPMs, it will look like Option -qwill query and -qawill query allNon-QLogic built Will produce output like this Command strings can also be used. Here is a sampleCustomer Acceptance Utility Displays help messages giving defined usage Page Getting Started with MPI InfiniPath MPIOther MPI Implementations $ mpicc -o cpi cpi.c An Example C ProgramThere is more information on the mpihosts file in section $ cp /usr/share/mpich/examples/basic$ mpirun -np 2 -m mpihosts ./fpi3 Examples Using Other LanguagesRun it with $ mpif77 -o fpi3 fpi3.fRun it Configuring MPI Programs for InfiniPath MPIInfiniPath MPI Details Configuring for ssh Using ssh-agentThis tells ssh that your key pair should let you Edit ~/.ssh/config so that it reads like thisProvides help Gets verbose output of all the commands in the scriptShows how to compile a program Compiling and LinkingTo use PGI for Fortran90/Fortran95 programs, use To Use Another CompilerTo use gcc for compiling and linking C++ programs use To use gcc for compiling and linking Fortran77 programs use$ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90 Cross-compilation IssuesCompiler and Linker Variables $ mpif90 -f90=ifort $ mpif95 -f95=ifortRunning MPI Programs Mpihosts File Environment for Node Programs Console I/O in MPI Programs$ mpirun -np n -m mpihosts -rcfile mpirunrc program Environment for Multiple Versions of InfiniPath or MPI$ mpirun -np n -m mpihosts -ppn p program-name Multiprocessor NodesNumber of processes to spawn Create up to specified number of processes per nodeRun each process in an xterm window. Default Off Print MPI version. Default OffPrint mpirun help message. Default Off Startup script for setting environment on nodes DefaultSets the working directory for the node program Disable-mpi-progress-check# modprobe rdmacm # modprobe rdmaucm Using Other MPI ImplementationsMPI Over uDAPL File I/O in MPI Using MPD$ mpdboot -f hostsfile MPD DescriptionInfiniPath MPI and Hybrid MPI/OpenMP Applications MPI-IO with RomioDebugging MPI Programs MPI ErrorsUsing Debuggers InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark 1 Measuring MPI Latency Between Two Nodes Benchmark Programs$ mpirun -np 2 -ppn 1 -m mpihosts osulatency Benchmark 2 Measuring MPI Bandwidth Between Two NodesBenchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw $ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Benchmark 4 Measuring MPI Latency in Host RingsMight produce output like this Page Allocating Resources Batch Queuing Script$ batchmpirun -np n mympiprogram Generating the mpihosts File# lsof /dev/ipath Clean Termination of MPI ProcessesSimple Process Management # /sbin/fuser -v /dev/ipath# ulimit -l Lock Enough Memory on Nodes When Using Slurm# /sbin/fuser -k /dev/ipath Mechanical and Electrical Considerations Troubleshooting InfiniPath Adapter InstallationSome HTX Motherboards May Need 2 or More CPUs in Use Table C-1. LED Link and Data Indicators$ ipathpkttest -B Mtrr Mapping and Write CombiningIncorrect Mtrr Mapping To check your bandwidth try Change Setting for Mapping MemoryIssue with SuperMicro H8DCE-HTe and QHT7040 Incorrect Mtrr Mapping Causes Unexpected Low BandwidthOpenFabrics Dependencies Software Installation IssuesInstall Warning with RHEL4U2 Mpirun Installation Requires 32-bit Support$ rpm -e infinipath-kernel --nodeps Installing Newer Drivers from Other DistributionsRemove the InfiniPath kernel components with the command Check your distribution for the exact RPM nameThese are used for SLES, SUSE, and Fedora Installing for Your DistributionKernel and Initialization Issues Reload all modules by using this command as rootPcimsiquirk Kernel Needs CONFIGPCIMSI=y$ grep ibipath /proc/interrupts Driver Load Fails Due to Unsupported KernelInfiniPath Interrupts Not Working Normal output will like similar to this$ grep -i acpi /proc/cmdline OpenFabrics Load Errors If ibipath Driver Load FailsInfiniPath ibipath Initialization Failure MPI Job Failures Due to Initialization ProblemsStop OpenSM Before Stopping/Restarting InfiniPath OpenFabrics IssuesLoad and Configure IPoIB Before Loading SDP System Administration TroubleshootingBroken Intermediate Link Performance IssuesMvapich Performance Issues InfiniPath MPI Troubleshooting$ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatency $ mpirun$ mpicc myprogram.c Compiler/Linker MismatchCompiler Can’t Find Include, Module or Library Files $ export MPICHCC=gcc $ mpicc mpiworld.c$ mpif90 myprogramf90.f90 -I/path/to/devel/include Compiling on Development NodesSpecifying the Run-time Library Path $ mpicc myprogram.c -I/path/to/devel/include# /etc/ldconfig Run Time Errors With Different MPI ImplementationsTroubleshooting InfiniPath MPI Troubleshooting HP-MPI mpirun and executable used together Using MPI.mod FilesWhen compiling, use descriptive names for the object files Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting $ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Error Messages Generated by mpirunMessages from the InfiniPath Library Troubleshooting InfiniPath MPI Troubleshooting MPI Messages $ mpirun -np 2 -m ~/tmp/q mpilatency 100 Following indicates an unknown hostThere is no route to a valid host $ mpirun -np 2 -m ~/tmp/q mpilatency 100000 There is no route to any hostOne program on one node died $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000 Driver and Link Error Messages Reported by MPI ProgramsMPI Stats Restarting InfiniPath These two commands perform the same functionsUseful Programs and Files for Debugging Check Cluster Homogeneity with ipathcheckoutShell script that gathers status Summary of Useful Programs and FilesModprobe Example contents areIbstatus Ipathcheckout Ipathcontrol Ipathbug-helper $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1 To check the contents of an RPM, use these commandsOption-qwill query and --qawill query all # lsmod egrep ’ipathibrdmafindex’Table C-3. statusstr File Example contents for QLogic-built drivers Command strings can also be used. Its format is as followsInfiniBand OpenFabrics References for MPIBooks for Learning MPI Programming Reference and Source for SlurmRocks ClustersGlossary GID LID Mtrr SDP Glossary IB6054601-00 D Index OpenFabrics Configuration Configuration of on Suse and Sles