QLogic HCA and QLogic Ofed Software Install Guide
Document Revision History
Preliminary
Preliminary
Removed phrase and Transmission Control
Moved table InfiniPath and OpenFabrics
Ipathether Configuration Setup Gener
Ates Error
Removed note OpenFabrics programs 32-bit
Preliminary
Table of Contents
Software Installation
Installation Verification and Additional Settings
Write Combining
List of Figures
Preliminary
How this Guide is Organized
Introduction
Who Should Read this Guide
Overview
Meaning
Interoperability
Conventions Used in this Guide
Typographical Conventions
Product documentation includes
Documentation
Support Headquarters
Contact Information
Preliminary
Features
Feature Overview
Support
Other Changes
Description
QLogic Adapter Model Numbers
Ofed SRP
Software Components
Hardware Installation
Step-by-Step Installation Checklist
Software Installation
Preliminary
Preliminary
Adapter Models and Related Platforms
Hardware Installation
Hardware Installation Requirements
Hardware
Run ipath control
Form Factors
Product Number Description
Cabling and Switches
QLogic InfiniBand Cables
Configuring the Bios
Unpacking Information
Safety with Electricity
Verify the Package Contents
List of the Package Contents
Hardware Installation Unpacking Information
QLogic QLE7280 with IBA7220 Asic
Asic
Unpacking the QLogic Adapter
Dual Adapter Installation
PCIe Slot in a Typical Motherboard
QLogic PCIe HCA Assembly with Riser Card
Assembled PCIe HCA with Riser
Hardware Installation for QHT7140 with HTX Riser
HTX Slot
QLogic QHT7140 Adapter with Riser Card
Assembled QHT7140 with Riser
Hardware Installation for the QHT7140 Without an HTX Riser
Cabling the Adapter to the InfiniBand Switch
Switch Configuration and Monitoring
Completing the Installation
Supported Linux Distributions
Software Installation
Cluster Setup
Types of Nodes in a Cluster Environment
Distribution Identifiers
Distribution Identifiers
InfiniPath/OpenFabrics Supported Distributions and Kernels
Distribution InfiniPath/OpenFabrics Supported Kernels
Sles10
Setting Up Your Environment
Compiler Support
Rhel5
Specific Component Requirements
Choose the Appropriate Download Files
Required OS Packages
OS Distribution Required Packages
Available Packages for QLogic Ofed 1.4 Release
Package Description Installation Documentation
Install InfiniPath and OpenFab
Kit on
Install QLogicIB-Basic with the Installer Tool
You will see a screen similar to this
Ofed SDP
Then type n to proceed to the next screen
Next screen contains these choices
Configure Ofed IP over IB IPV4 addresses now? n
# ./INSTALL --32bit
Additional component names allowed for -Eand -Doptions
Install -Clists all the available components, which include
Install Options
Command Meaning
About rpm Installation
Using rpm to Install InfiniPath and OpenFabrics
Become root, then
RPM Organization
Next, install all non-prefixed RPMs
Install QLogic Ofed User-level Software with the rpm Command
Finally, install the prefixed RPMs in /usr/mpi/qlogic
Software Installation
Install Frontend and Compute Nodes
Install QLogic Ofed Using Rocks
Rocks Installation on an Existing Frontend Node
Use the following command to rebuild the entire cluster
Install QLogic Ofed Using a Platform OCS Kit
Install FastFabric Software CD/ISO Image
Then use the following command for each node
MPI include files are
Install Additional Software
Installed Layout
Installing Lustre
Removing Software Packages
Uninstalling Using the Installer Tool
Uninstalling InfiniPath and OpenFabrics RPMs
Qlc is a part of all the InfiniPath package names
Uninstalling Software with Rocks or Platform OCS
Downgrading RPMs
Preliminary
InfiniPath and OpenFabrics Driver Overview
Configuring Drivers and Services
OpenFabrics Drivers and Services Configuration and Startup
Type
Configuring the IPoIB Network Interface
To verify the configuration, type
Output from this command will be similar to this
OpenSM
Command to disable it on reboot is
Etc/init.d/opensmd
You can stop opensmd again like this
To use Ofed SRP, follow these steps
Guid
Output, look for lines similar to these
Configuring and Administering the Vnic Interface
# ibqlgcvnicquery
Preliminary
Etc/infiniband/qlgcvnic.cfg
Create the Vnic interfaces using the configuration file
RX CSUM=TRUE HEARTBEAT=100
Preliminary
To restart the QLogic Vnic driver, run the following command
This information is collected from
MPI over uDAPL
On every node, type the following command as root
Other Configuration Changing the MTU Size
Etc/infiniband/openib.conf
Managing the InfiniPath Driver
Device files are
Configure InfiniPath Driver State
To enable the driver, use the command as root
Start, Stop or Restart InfiniPath
Unloading the Driver/Modules Manually
Further Information on Configuring and Loading Drivers
Sequence of commands to restart the driver are as follows
Preliminary
Adapter and Other Settings
Installation Verification and Additional Settings
LED Link and Data Indicators
Hostname1 Hostname2
Customer Acceptance Utility
This will also show link speed
Ipath checkout Options
Preliminary
Some HTX Motherboards May Need Two or More CPUs in Use
Installation Troubleshooting
Hardware Issues
Node Spontaneously Reboots
Issue with Supermicro H8DCE-HTe and QHT7040
Bios Settings
Software Installation Issues
Enable Advanced Configuration and Power Interface Acpi
# yum install kernel-devel
Resolving Conflicts
Mpirun Installation Requires 32-bit Support
Check your distribution for the exact RPM name
Configuration Issues
Lockable Memory Error on Initial Installation of InfiniPath
Outdated ipathether Configuration Setup Generates Error
Eth2 error fetching interface information Device not found
Verify Write Combining is Working
Write Combining
Introduction
Mtrr Mapping and Write Combining
Edit Bios Settings to Fix Mtrr Issues
PAT and Write Combining
Use the ipathmtrr Script to Fix Mtrr Issues
Preliminary
Configuration File Name Description
Configuration Files
Table C-1. Configuration Files
Qlgcvnictools subdirectory
Also installed in /etc/infiniband
Sample and Template Files Description
With the Ofed documentation, or
Different Nodes May Use Different RPMs
Package Descriptions
Package Names with the QLogicIB-Basic Download
InfiniPath and OpenFabrics RPMs
InfiniPath RPM Version Numbers and Identifiers
OpenFabrics RPM Names
Table D-2. InfiniPath/RPMs
Documentation RPMs
InfiniPath RPMs
Table D-1. Documentation/RPMs
Table D-5. OpenFabrics/RPMs
OpenFabrics RPMs
Table D-3. InfiniPath-Devel/RPMs
Table D-4. InfiniPath-MPI/RPMs
Ibutils-xxx.x8664.rpm
Dapl-xxx.x8664.rpm
Dapl-utils-xxx.x8664.rpm
Ibsim-xxx.x8664.rpm
Librdmacm-xxx.x86 64.rpm
Libibverbs-xxx.x8664.rpm
Libibverbs-utils-xxx.x8664.rpm
Libipathverbs-xxx.x86 64.rpm
Table D-6. OpenFabrics-Devel/RPMs
Table D-7. OpenSM/RPM
Table D-10. Other HCAs-Devel/RPMs
Other HCAs
Table D-8. OpenSM-Devel/RPM
Table D-9. Other HCAs/RPMs
Table D-11. OtherMPIs/RPMs
Other MPIs
Openmpipathscaleqlc-xxx.yyy.x8664.rpm
Openmpigcc-xxx.x8664.rpm
Openmpigccqlc-xxx.yyy.x8664.rpm
Openmpiintelqlc-xxx.yyy.x8664.rpm a
Preliminary
Index
Form factors for HCAs Hardware installation overview
Module
Hardware requirements
HTX motherboards may required two or more CPUs A-1
Mpi-selector 5-15,5-18
Skip=LIST 7-3Software
VNIC, configuration
Page
Preliminary