No ports available on /dev/ipath

3 – Using InfiniPath MPI InfiniPath MPI Limitations

NOTE: If port sharing is enabled, this limit is raised to 16 and 8 respectively. To enable port sharing, set PSM_SHAREDPORTS=1 in your environment

There are no C++ bindings to MPI -- use the extern C MPI function calls.

In MPI-IO file I/O calls in the Fortran binding, offset or displacement arguments are limited to 32 bits. Thus, for example, the second argument of MPI_File_seek must

lie between -231and 231-1, and the argument to MPI_File_read_at must lie between 0 and 232-1.

3-22	IB6054601-00 D

Page 64

Image 64

Q-Logic IB6054601-00 D manual No ports available on /dev/ipath

IB6054601-00 D specifications

The Q-Logic IB6054601-00 D is a high-performance InfiniBand adapter card designed for data centers and enterprise applications requiring robust connectivity and low-latency communication. This adapter is part of QLogic's extensive portfolio of networking solutions, catering to the needs of high-performance computing (HPC), cloud computing, and virtualization environments.

One of the standout features of the IB6054601-00 D is its capability to support data transfer rates of up to 56 Gbps. This makes it ideal for applications demanding large bandwidth and quick data processing. The adapter is optimized for RDMA (Remote Direct Memory Access) technology, which allows data to be transferred directly between the memory of different computers without involving the CPU. This reduces latency and CPU overhead, leading to enhanced overall system performance.

The architecture of the IB6054601-00 D includes support for a dual-port design, which offers increased bandwidth, redundancy, and fault tolerance. This dual-port configuration is especially advantageous for environments that require high availability and reliability, such as financial services and mission-critical applications.

The adapter utilizes advanced error detection and correction mechanisms, ensuring that data integrity is maintained during transmission. With features like adaptive routing and congestion management, the IB6054601-00 D is capable of optimizing the handling of data flows, thereby enhancing performance even under heavy loads.

In terms of compatibility, the Q-Logic IB6054601-00 D supports a wide range of operating systems and virtualization technologies, making it easy to integrate into diverse IT environments. It also includes drivers and software packages that facilitate seamless deployment and management.

In addition to high-speed connectivity, the adapter is designed with power efficiency in mind. It adheres to Energy Star regulations, helping organizations lower their operational costs while minimizing their environmental footprint.

Overall, the Q-Logic IB6054601-00 D stands out for its high throughput, low latency, and reliability. Its combination of advanced features and technologies positions it as an excellent choice for organizations looking to enhance their data center performance and maximize the efficiency of their network infrastructure. With the growing demands for faster and more efficient data transfer, solutions like the IB6054601-00 D are essential in meeting the evolving needs of modern enterprises.

Contents

InfiniPath User Guide InfiniPath User Guide Version Index Page Table of Contents Section Using InfiniPath MPI Appendix a Benchmark Programs InfiniPath User Guide Version Appendix D Recommended Reading InfiniPath Software Structure InfiniPath User Guide Version IB6054601-00 D Who Should Read this Guide How this Guide is Organized Switches OverviewInteroperability Appendix E Glossary of technical terms What’s New in this Release PathScale-QLogic Adapter Model Numbers Supported Distributions and Kernels InfiniPath/OpenFabrics Supported Distributions and Kernels Software Components Conventions Used in this Document Documentation and Technical SupportThis Guide uses these typographical conventions InfiniPath product documentation includes Introduction Documentation and Technical Support Page Installed Layout Introduction MPI include files are Memory FootprintDocumentation can be found Adapter Required Component Optional Memory Footprint Comment Configuration and Startup Bios Settings Memory Footprint, 331 MB per Node InfiniPath Driver Startup InfiniPath Driver Software ConfigurationInfiniPath Driver Filesystem Subnet Management Agent Layered Ethernet Driver $ ifconfig -a $ ls /sys/class/net Ipathether Configuration on Fedora and RHEL4Create or edit the following file as root Ipathether Configuration on Suse 9.3, Sles 9, and Sles ONBOOT=YES Determine the MAC address that will be used Guid can also be returned by runningIf this file does not exist, skip to Step # lsmod grep -q ipathether modprobe ipathether Add the following lines to the file OpenFabrics Configuration and Startup Configuring the IPoIB Network Interface# ifup eth2 # ifconfig eth2 # ifconfig ib0 10.1.17.3 netmask 0xffffff00 OpenSM Further Information on Configuring and Loading Drivers To enable the driver, use the command as rootStarting and Stopping the InfiniPath Software You can stop it again like this # chkconfig infinipath off # /sbin/chkconfig --list opensmd grep -w on# /etc/init.d/infinipath start stop restart $ /sbin/lsmod grep ipathether Configuring ssh and sshd Using shosts.equiv Software Status Change the file to mode 600 when finished editing Remove Unneeded Services Process Limitation with sshPerformance and Management Tips # killall -HUP sshd Disable Powersaving Features Balanced Processor Power For Suse 9.3 and 10.0 run this command as rootSDP Module Parameters for Best Performance CPU Affinity Hyper-Threading Homogeneous NodesRun the shell script ipathcontrol as follows Ipathbughelper Non-QLogic built Option -qwill query and -qawill query allFor non-QLogic RPMs, it will look like Customer Acceptance Utility Command strings can also be used. Here is a sampleWill produce output like this Displays help messages giving defined usage Page Other MPI Implementations InfiniPath MPIGetting Started with MPI An Example C Program There is more information on the mpihosts file in section$ cp /usr/share/mpich/examples/basic $ mpicc -o cpi cpi.c Examples Using Other Languages Run it with$ mpif77 -o fpi3 fpi3.f $ mpirun -np 2 -m mpihosts ./fpi3 Configuring MPI Programs for InfiniPath MPI Run it Configuring for ssh Using ssh-agent InfiniPath MPI Details Edit ~/.ssh/config so that it reads like this This tells ssh that your key pair should let you Gets verbose output of all the commands in the script Shows how to compile a programCompiling and Linking Provides help To Use Another Compiler To use gcc for compiling and linking C++ programs useTo use gcc for compiling and linking Fortran77 programs use To use PGI for Fortran90/Fortran95 programs, use Cross-compilation Issues Compiler and Linker Variables$ mpif90 -f90=ifort $ mpif95 -f95=ifort $ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90 Running MPI Programs Mpihosts File Console I/O in MPI Programs Environment for Node Programs Environment for Multiple Versions of InfiniPath or MPI $ mpirun -np n -m mpihosts -rcfile mpirunrc program Multiprocessor Nodes Number of processes to spawnCreate up to specified number of processes per node $ mpirun -np n -m mpihosts -ppn p program-name Print MPI version. Default Off Print mpirun help message. Default OffStartup script for setting environment on nodes Default Run each process in an xterm window. Default Off Disable-mpi-progress-check Sets the working directory for the node program MPI Over uDAPL Using Other MPI Implementations# modprobe rdmacm # modprobe rdmaucm Using MPD $ mpdboot -f hostsfileMPD Description File I/O in MPI MPI-IO with Romio InfiniPath MPI and Hybrid MPI/OpenMP Applications Using Debuggers MPI ErrorsDebugging MPI Programs InfiniPath MPI Limitations No ports available on /dev/ipath Benchmark Programs Benchmark 1 Measuring MPI Latency Between Two Nodes Benchmark 2 Measuring MPI Bandwidth Between Two Nodes $ mpirun -np 2 -ppn 1 -m mpihosts osulatency Benchmark 3 Messaging Rate Microbenchmarks $ mpirun -np 8 ./mpimultibw Might produce output like this Benchmark 4 Measuring MPI Latency in Host Rings$ mpirun -np 4 -ppn 1 -m mpihosts mpilatency 100 Page Batch Queuing Script Allocating Resources Generating the mpihosts File $ batchmpirun -np n mympiprogram Clean Termination of MPI Processes Simple Process Management# /sbin/fuser -v /dev/ipath # lsof /dev/ipath # /sbin/fuser -k /dev/ipath Lock Enough Memory on Nodes When Using Slurm# ulimit -l Troubleshooting InfiniPath Adapter Installation Mechanical and Electrical Considerations Table C-1. LED Link and Data Indicators Some HTX Motherboards May Need 2 or More CPUs in Use Incorrect Mtrr Mapping Mtrr Mapping and Write Combining$ ipathpkttest -B Change Setting for Mapping Memory Issue with SuperMicro H8DCE-HTe and QHT7040Incorrect Mtrr Mapping Causes Unexpected Low Bandwidth To check your bandwidth try Software Installation Issues Install Warning with RHEL4U2Mpirun Installation Requires 32-bit Support OpenFabrics Dependencies Installing Newer Drivers from Other Distributions Remove the InfiniPath kernel components with the commandCheck your distribution for the exact RPM name $ rpm -e infinipath-kernel --nodeps Installing for Your Distribution Kernel and Initialization IssuesReload all modules by using this command as root These are used for SLES, SUSE, and Fedora Kernel Needs CONFIGPCIMSI=y Pcimsiquirk Driver Load Fails Due to Unsupported Kernel InfiniPath Interrupts Not WorkingNormal output will like similar to this $ grep ibipath /proc/interrupts OpenFabrics Load Errors If ibipath Driver Load Fails $ grep -i acpi /proc/cmdline MPI Job Failures Due to Initialization Problems InfiniPath ibipath Initialization Failure OpenFabrics Issues Load and Configure IPoIB Before Loading SDPSystem Administration Troubleshooting Stop OpenSM Before Stopping/Restarting InfiniPath Performance Issues Mvapich Performance IssuesInfiniPath MPI Troubleshooting Broken Intermediate Link $ mpirun $ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osulatency Compiler/Linker Mismatch Compiler Can’t Find Include, Module or Library Files$ export MPICHCC=gcc $ mpicc mpiworld.c $ mpicc myprogram.c Compiling on Development Nodes Specifying the Run-time Library Path$ mpicc myprogram.c -I/path/to/devel/include $ mpif90 myprogramf90.f90 -I/path/to/devel/include Run Time Errors With Different MPI Implementations # /etc/ldconfig Troubleshooting InfiniPath MPI Troubleshooting When compiling, use descriptive names for the object files Using MPI.mod FilesHP-MPI mpirun and executable used together Extending MPI Modules Troubleshooting InfiniPath MPI Troubleshooting Messages from the InfiniPath Library Error Messages Generated by mpirun$ mpirun -m ~/tmp/sm -np 2 -mpilatency 1000 Troubleshooting InfiniPath MPI Troubleshooting MPI Messages There is no route to a valid host Following indicates an unknown host$ mpirun -np 2 -m ~/tmp/q mpilatency 100 One program on one node died There is no route to any host$ mpirun -np 2 -m ~/tmp/q mpilatency 100000 Driver and Link Error Messages Reported by MPI Programs $ mpirun -np ~/tmp/q -q 60 mpilatency 1000000 MPI Stats These two commands perform the same functions Useful Programs and Files for DebuggingCheck Cluster Homogeneity with ipathcheckout Restarting InfiniPath Summary of Useful Programs and Files Shell script that gathers status Example contents are Modprobe Ibstatus Ipathcheckout Ipathcontrol Ipathbug-helper To check the contents of an RPM, use these commands Option-qwill query and --qawill query all# lsmod egrep ’ipathibrdmafindex’ $ mpirun -np 2 -m /tmp/id1 -d0x101 mpilatency 1 Table C-3. statusstr File Command strings can also be used. Its format is as follows Example contents for QLogic-built drivers References for MPI Books for Learning MPI ProgrammingReference and Source for Slurm InfiniBand OpenFabrics Clusters Rocks Glossary GID LID Mtrr SDP Glossary IB6054601-00 D Index Configuration of on Suse and Sles OpenFabrics Configuration