Manuals / Q-Logic / Computer Equipment / Switch

Q-Logic IB6054601-00 D manual $ mpirun -np 8 ./mpimultibw

Models: IB6054601-00 D

1 122

Download 122 pages 48.66 Kb

Page 68

A – Benchmark Programs

Benchmark 3: Messaging Rate Microbenchmarks

Q

benchmark (as shown in the example above). It has been enhanced with the following additional functionality:

■Messaging rate reported as well as bandwidth

■N/2 dynamically calculated at end of run

■Allows user to run multiple processes per node and see aggregate bandwidth and messaging rates

The benchmark has been updated with code to dynamically determine which processes are on which host. This is an example showing the type of output you will see when you run mpi_multibw:

$ mpirun -np 8 ./mpi_multibw

This will run on four processes per node. Typical output might look like:

#PathScale Modified OSU MPI Bandwidth Test

(OSU Version 2.2, PathScale $Revision: 1.1 $)

#Running on 4 procs per node

# Size	Aggregate Bandwidth	(MB/s) Messages/s
1	8.150462	8150461.697283
2	16.693747	8346873.631841
4	33.086567	8271641.814960
8	66.733488	8341686.016159
16	108.097082	6756067.602089
32	213.733132	6679160.388156
64	389.544112	6086626.744516
128	569.671531	4450558.832794
256	725.826904	2835261.345093
512	839.450014	1639550.807757
1024	913.428063	892019.592596
2048	954.482747	466056.028717
4096	954.474461	233025.991467
8192	954.452712	116510.340772
16384	954.496729	58257.857017
32768	954.547225	29130.469523
65536	949.074433	14481.726572
131072	951.786548	7261.555084
262144	952.193849	3632.331272
524288	952.391830	1816.543255
1048576	952.490368	908.365600
2097152	952.539382	454.206172
4194304	952.566591	227.109573

Searching for N/2 bandwidth. Maximum Bandwidth of 954.547225 MB/s...

Found N/2 bandwidth of 476.993060 MB/s at size 94 bytes

This microbenchmark is available and can be downloaded from the QLogic website:

http://www.qlogic.com

A-4	IB6054601-00 D

Image 68

Q-Logic IB6054601-00 D manual $ mpirun -np 8 ./mpimultibw

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version Appendix D Recommended Reading InfiniPath Software StructureInfiniPath User Guide Version IB6054601-00 D Who Should Read this Guide How this Guide is OrganizedSwitches OverviewInteroperability Appendix E Glossary of technical termsWhat’s New in this Release PathScale-QLogic Adapter Model NumbersSupported Distributions and Kernels InfiniPath/OpenFabrics Supported Distributions and KernelsSoftware Components Conventions Used in this Document Documentation and Technical SupportThis Guide uses these typographical conventions InfiniPath product documentation includesIntroduction Documentation and Technical Support Page Installed Layout IntroductionDocumentation can be found Memory FootprintMPI include files are Adapter Required Component Optional Memory Footprint Comment Configuration and Startup Bios Settings Memory Footprint, 331 MB per NodeInfiniPath Driver Filesystem InfiniPath Driver Software ConfigurationInfiniPath Driver Startup Subnet Management Agent Layered Ethernet DriverCreate or edit the following file as root Ipathether Configuration on Fedora and RHEL4$ ifconfig -a $ ls /sys/class/net Ipathether Configuration on Suse 9.3, Sles 9, and Sles ONBOOT=YESDetermine the MAC address that will be used Guid can also be returned by runningIf this file does not exist, skip to Step # lsmod grep -q ipathether modprobe ipathetherAdd the following lines to the file OpenFabrics Configuration and Startup Configuring the IPoIB Network Interface# ifup eth2 # ifconfig eth2 # ifconfig ib0 10.1.17.3 netmask 0xffffff00OpenSM Further Information on Configuring and Loading Drivers To enable the driver, use the command as rootStarting and Stopping the InfiniPath Software You can stop it again like this# chkconfig infinipath off # /sbin/chkconfig --list opensmd grep -w on# /etc/init.d/infinipath start stop restart $ /sbin/lsmod grep ipathetherConfiguring ssh and sshd Using shosts.equiv Software StatusChange the file to mode 600 when finished editing Remove Unneeded Services Process Limitation with sshPerformance and Management Tips # killall -HUP sshdDisable Powersaving Features Balanced Processor Power For Suse 9.3 and 10.0 run this command as rootSDP Module Parameters for Best Performance CPU AffinityHyper-Threading Homogeneous NodesRun the shell script ipathcontrol as follows IpathbughelperFor non-QLogic RPMs, it will look like Option -qwill query and -qawill query allNon-QLogic built Will produce output like this Command strings can also be used. Here is a sampleCustomer Acceptance Utility Displays help messages giving defined usage Page Getting Started with MPI InfiniPath MPIOther MPI Implementations An Example C Program There is more information on the mpihosts file in section$ cp /usr/share/mpich/examples/basic $ mpicc -o cpi cpi.cExamples Using Other Languages Run it with$ mpif77 -o fpi3 fpi3.f $ mpirun -np 2 -m mpihosts ./fpi3Configuring MPI Programs for InfiniPath MPI Run itConfiguring for ssh Using ssh-agent InfiniPath MPI DetailsEdit ~/.ssh/config so that it reads like this This tells ssh that your key pair should let youGets verbose output of all the commands in the script Shows how to compile a programCompiling and Linking Provides helpTo Use Another Compiler To use gcc for compiling and linking C++ programs useTo use gcc for compiling and linking Fortran77 programs use To use PGI for Fortran90/Fortran95 programs, useCross-compilation Issues Compiler and Linker Variables$ mpif90 -f90=ifort $ mpif95 -f95=ifort $ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90Running MPI Programs Mpihosts File Console I/O in MPI Programs Environment for Node ProgramsEnvironment for Multiple Versions of InfiniPath or MPI $ mpirun -np n -m mpihosts -rcfile mpirunrc programMultiprocessor Nodes Number of processes to spawnCreate up to specified number of processes per node $ mpirun -np n -m mpihosts -ppn p program-namePrint MPI version. Default Off Print mpirun help message. Default OffStartup script for setting environment on nodes Default Run each process in an xterm window. Default OffDisable-mpi-progress-check Sets the working directory for the node program# modprobe rdmacm # modprobe rdmaucm Using Other MPI ImplementationsMPI Over uDAPL Using MPD $ mpdboot -f hostsfileMPD Description File I/O in MPIMPI-IO with Romio InfiniPath MPI and Hybrid MPI/OpenMP ApplicationsDebugging MPI Programs MPI ErrorsUsing Debuggers InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark Programs Benchmark 1 Measuring MPI Latency Between Two NodesBenchmark 2 Measuring MPI Bandwidth Between Two Nodes $ mpirun -np 2 -ppn 1 -m mpihosts osulatencyBenchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw $ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Benchmark 4 Measuring MPI Latency in Host RingsMight produce output like this Page Batch Queuing Script Allocating ResourcesGenerating the mpihosts File $ batchmpirun -np n mympiprogramClean Termination of MPI Processes Simple Process Management# /sbin/fuser -v /dev/ipath # lsof /dev/ipath# ulimit -l Lock Enough Memory on Nodes When Using Slurm# /sbin/fuser -k /dev/ipath Troubleshooting InfiniPath Adapter Installation Mechanical and Electrical ConsiderationsTable C-1. LED Link and Data Indicators Some HTX Motherboards May Need 2 or More CPUs in Use$ ipathpkttest -B Mtrr Mapping and Write CombiningIncorrect Mtrr Mapping Change Setting for Mapping Memory Issue with SuperMicro H8DCE-HTe and QHT7040Incorrect Mtrr Mapping Causes Unexpected Low Bandwidth To check your bandwidth trySoftware Installation Issues Install Warning with RHEL4U2Mpirun Installation Requires 32-bit Support OpenFabrics DependenciesInstalling Newer Drivers from Other Distributions Remove the InfiniPath kernel components with the commandCheck your distribution for the exact RPM name $ rpm -e infinipath-kernel --nodepsInstalling for Your Distribution Kernel and Initialization IssuesReload all modules by using this command as root These are used for SLES, SUSE, and FedoraKernel Needs CONFIGPCIMSI=y PcimsiquirkDriver Load Fails Due to Unsupported Kernel InfiniPath Interrupts Not WorkingNormal output will like similar to this $ grep ibipath /proc/interruptsOpenFabrics Load Errors If ibipath Driver Load Fails $ grep -i acpi /proc/cmdlineMPI Job Failures Due to Initialization Problems InfiniPath ibipath Initialization FailureOpenFabrics Issues Load and Configure IPoIB Before Loading SDPSystem Administration Troubleshooting Stop OpenSM Before Stopping/Restarting InfiniPathPerformance Issues Mvapich Performance IssuesInfiniPath MPI Troubleshooting Broken Intermediate Link$ mpirun $ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatencyCompiler/Linker Mismatch Compiler Can’t Find Include, Module or Library Files$ export MPICHCC=gcc $ mpicc mpiworld.c $ mpicc myprogram.cCompiling on Development Nodes Specifying the Run-time Library Path$ mpicc myprogram.c -I/path/to/devel/include $ mpif90 myprogramf90.f90 -I/path/to/devel/includeRun Time Errors With Different MPI Implementations # /etc/ldconfigTroubleshooting InfiniPath MPI Troubleshooting HP-MPI mpirun and executable used together Using MPI.mod FilesWhen compiling, use descriptive names for the object files Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting $ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Error Messages Generated by mpirunMessages from the InfiniPath Library Troubleshooting InfiniPath MPI Troubleshooting MPI Messages $ mpirun -np 2 -m ~/tmp/q mpilatency 100 Following indicates an unknown hostThere is no route to a valid host $ mpirun -np 2 -m ~/tmp/q mpilatency 100000 There is no route to any hostOne program on one node died Driver and Link Error Messages Reported by MPI Programs $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000MPI Stats These two commands perform the same functions Useful Programs and Files for DebuggingCheck Cluster Homogeneity with ipathcheckout Restarting InfiniPathSummary of Useful Programs and Files Shell script that gathers statusExample contents are ModprobeIbstatus Ipathcheckout Ipathcontrol Ipathbug-helper To check the contents of an RPM, use these commands Option-qwill query and --qawill query all# lsmod egrep ’ipathibrdmafindex’ $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1Table C-3. statusstr File Command strings can also be used. Its format is as follows Example contents for QLogic-built driversReferences for MPI Books for Learning MPI ProgrammingReference and Source for Slurm InfiniBand OpenFabricsClusters RocksGlossary GID LID Mtrr SDP Glossary IB6054601-00 D Index Configuration of on Suse and Sles OpenFabrics Configuration