QLogic HCA and QLogic Ofed Software Install Guide
Document Revision History
Preliminary
Preliminary
Removed phrase and Transmission Control
Moved table InfiniPath and OpenFabrics
Ipathether Configuration Setup Gener
Ates Error
Removed note OpenFabrics programs 32-bit
Preliminary
Table of Contents
Software Installation
Installation Verification and Additional Settings
Write Combining
List of Figures
Preliminary
How this Guide is Organized
Introduction
Who Should Read this Guide
Overview
Conventions Used in this Guide
Interoperability
Typographical Conventions
Meaning
Product documentation includes
Documentation
Support Headquarters
Contact Information
Preliminary
Features
Feature Overview
Support
Other Changes
Description
QLogic Adapter Model Numbers
Ofed SRP
Software Components
Hardware Installation
Step-by-Step Installation Checklist
Software Installation
Preliminary
Preliminary
Hardware Installation Requirements
Hardware Installation
Hardware
Adapter Models and Related Platforms
Run ipath control
Form Factors
Product Number Description
Cabling and Switches
QLogic InfiniBand Cables
Configuring the Bios
Verify the Package Contents
Safety with Electricity
List of the Package Contents
Unpacking Information
Hardware Installation Unpacking Information
QLogic QLE7280 with IBA7220 Asic
Asic
Unpacking the QLogic Adapter
Dual Adapter Installation
PCIe Slot in a Typical Motherboard
QLogic PCIe HCA Assembly with Riser Card
Assembled PCIe HCA with Riser
Hardware Installation for QHT7140 with HTX Riser
HTX Slot
QLogic QHT7140 Adapter with Riser Card
Assembled QHT7140 with Riser
Hardware Installation for the QHT7140 Without an HTX Riser
Cabling the Adapter to the InfiniBand Switch
Switch Configuration and Monitoring
Completing the Installation
Cluster Setup
Software Installation
Types of Nodes in a Cluster Environment
Supported Linux Distributions
InfiniPath/OpenFabrics Supported Distributions and Kernels
Distribution Identifiers
Distribution InfiniPath/OpenFabrics Supported Kernels
Distribution Identifiers
Compiler Support
Setting Up Your Environment
Rhel5
Sles10
Required OS Packages
Choose the Appropriate Download Files
OS Distribution Required Packages
Specific Component Requirements
Available Packages for QLogic Ofed 1.4 Release
Package Description Installation Documentation
Install InfiniPath and OpenFab
Kit on
Install QLogicIB-Basic with the Installer Tool
You will see a screen similar to this
Ofed SDP
Then type n to proceed to the next screen
Next screen contains these choices
Configure Ofed IP over IB IPV4 addresses now? n
# ./INSTALL --32bit
Install Options
Install -Clists all the available components, which include
Command Meaning
Additional component names allowed for -Eand -Doptions
About rpm Installation
Using rpm to Install InfiniPath and OpenFabrics
Become root, then
RPM Organization
Next, install all non-prefixed RPMs
Install QLogic Ofed User-level Software with the rpm Command
Finally, install the prefixed RPMs in /usr/mpi/qlogic
Software Installation
Install Frontend and Compute Nodes
Install QLogic Ofed Using Rocks
Rocks Installation on an Existing Frontend Node
Install FastFabric Software CD/ISO Image
Install QLogic Ofed Using a Platform OCS Kit
Then use the following command for each node
Use the following command to rebuild the entire cluster
Installed Layout
Install Additional Software
Installing Lustre
MPI include files are
Removing Software Packages
Uninstalling Using the Installer Tool
Uninstalling InfiniPath and OpenFabrics RPMs
Qlc is a part of all the InfiniPath package names
Uninstalling Software with Rocks or Platform OCS
Downgrading RPMs
Preliminary
InfiniPath and OpenFabrics Driver Overview
Configuring Drivers and Services
OpenFabrics Drivers and Services Configuration and Startup
To verify the configuration, type
Configuring the IPoIB Network Interface
Output from this command will be similar to this
Type
OpenSM
Command to disable it on reboot is
Etc/init.d/opensmd
You can stop opensmd again like this
To use Ofed SRP, follow these steps
Guid
Output, look for lines similar to these
Configuring and Administering the Vnic Interface
# ibqlgcvnicquery
Preliminary
Etc/infiniband/qlgcvnic.cfg
Create the Vnic interfaces using the configuration file
RX CSUM=TRUE HEARTBEAT=100
Preliminary
To restart the QLogic Vnic driver, run the following command
This information is collected from
MPI over uDAPL
On every node, type the following command as root
Other Configuration Changing the MTU Size
Etc/infiniband/openib.conf
Managing the InfiniPath Driver
To enable the driver, use the command as root
Configure InfiniPath Driver State
Start, Stop or Restart InfiniPath
Device files are
Unloading the Driver/Modules Manually
Further Information on Configuring and Loading Drivers
Sequence of commands to restart the driver are as follows
Preliminary
Adapter and Other Settings
Installation Verification and Additional Settings
LED Link and Data Indicators
Hostname1 Hostname2
Customer Acceptance Utility
This will also show link speed
Ipath checkout Options
Preliminary
Hardware Issues
Installation Troubleshooting
Node Spontaneously Reboots
Some HTX Motherboards May Need Two or More CPUs in Use
Software Installation Issues
Bios Settings
Enable Advanced Configuration and Power Interface Acpi
Issue with Supermicro H8DCE-HTe and QHT7040
# yum install kernel-devel
Resolving Conflicts
Mpirun Installation Requires 32-bit Support
Lockable Memory Error on Initial Installation of InfiniPath
Configuration Issues
Outdated ipathether Configuration Setup Generates Error
Check your distribution for the exact RPM name
Eth2 error fetching interface information Device not found
Verify Write Combining is Working
Write Combining
Introduction
Mtrr Mapping and Write Combining
Edit Bios Settings to Fix Mtrr Issues
PAT and Write Combining
Use the ipathmtrr Script to Fix Mtrr Issues
Preliminary
Configuration File Name Description
Configuration Files
Table C-1. Configuration Files
Sample and Template Files Description
Also installed in /etc/infiniband
With the Ofed documentation, or
Qlgcvnictools subdirectory
Different Nodes May Use Different RPMs
Package Descriptions
Package Names with the QLogicIB-Basic Download
InfiniPath and OpenFabrics RPMs
InfiniPath RPM Version Numbers and Identifiers
OpenFabrics RPM Names
InfiniPath RPMs
Documentation RPMs
Table D-1. Documentation/RPMs
Table D-2. InfiniPath/RPMs
Table D-3. InfiniPath-Devel/RPMs
OpenFabrics RPMs
Table D-4. InfiniPath-MPI/RPMs
Table D-5. OpenFabrics/RPMs
Dapl-utils-xxx.x8664.rpm
Dapl-xxx.x8664.rpm
Ibsim-xxx.x8664.rpm
Ibutils-xxx.x8664.rpm
Libibverbs-utils-xxx.x8664.rpm
Libibverbs-xxx.x8664.rpm
Libipathverbs-xxx.x86 64.rpm
Librdmacm-xxx.x86 64.rpm
Table D-6. OpenFabrics-Devel/RPMs
Table D-7. OpenSM/RPM
Table D-8. OpenSM-Devel/RPM
Other HCAs
Table D-9. Other HCAs/RPMs
Table D-10. Other HCAs-Devel/RPMs
Table D-11. OtherMPIs/RPMs
Other MPIs
Openmpigccqlc-xxx.yyy.x8664.rpm
Openmpigcc-xxx.x8664.rpm
Openmpiintelqlc-xxx.yyy.x8664.rpm a
Openmpipathscaleqlc-xxx.yyy.x8664.rpm
Preliminary
Index
Form factors for HCAs Hardware installation overview
Module
Hardware requirements
HTX motherboards may required two or more CPUs A-1
Mpi-selector 5-15,5-18
Skip=LIST 7-3Software
VNIC, configuration
Page
Preliminary