QLogic QLE7280 with IBA7220 Asic | Q-Logic IB0056101-00 G.02, OFED

A

Hardware Installation Unpacking Information

PCI Express

Edge Connectors

InfiniBand

Connector

Face Plate

IBA7220 ASIC

Figure 4-1. QLogic QLE7280 with IBA7220 ASIC

PCI Express Riser Card. Not supplied; shown for reference.

PCI Express

Edge Connectors

IBA6120 ASIC

InfiniBand

Connector

Face Plate

Figure 4-2. QLogic QLE7140 Card with Riser, Top View

IB0056101-00 G.02

Preliminary

4-7

Image 35

Q-Logic IB0056101-00 G.02, OFED manual QLogic QLE7280 with IBA7220 Asic

Contents

QLogic HCA and QLogic Ofed Software Install Guide Document Revision History Preliminary Preliminary Removed phrase and Transmission Control Moved table InfiniPath and OpenFabrics Ipathether Configuration Setup GenerAtes Error Removed note OpenFabrics programs 32-bit Preliminary Table of Contents Software Installation Installation Verification and Additional Settings Write Combining List of Figures Preliminary How this Guide is Organized IntroductionWho Should Read this Guide Overview Meaning InteroperabilityConventions Used in this Guide Typographical Conventions Product documentation includes Documentation Support Headquarters Contact Information Preliminary Features Feature Overview Support Other Changes Description QLogic Adapter Model Numbers „ Ofed SRP Software Components Hardware Installation Step-by-Step Installation Checklist Software Installation Preliminary Preliminary Adapter Models and Related Platforms Hardware InstallationHardware Installation Requirements Hardware Run ipath control Form Factors Product Number Description Cabling and SwitchesQLogic InfiniBand Cables Configuring the Bios Unpacking Information Safety with ElectricityVerify the Package Contents List of the Package Contents Hardware Installation Unpacking Information QLogic QLE7280 with IBA7220 Asic Asic Unpacking the QLogic Adapter Dual Adapter Installation PCIe Slot in a Typical Motherboard QLogic PCIe HCA Assembly with Riser Card Assembled PCIe HCA with Riser Hardware Installation for QHT7140 with HTX Riser HTX Slot QLogic QHT7140 Adapter with Riser Card Assembled QHT7140 with Riser Hardware Installation for the QHT7140 Without an HTX Riser Cabling the Adapter to the InfiniBand Switch Switch Configuration and Monitoring Completing the Installation Supported Linux Distributions Software InstallationCluster Setup Types of Nodes in a Cluster Environment Distribution Identifiers Distribution IdentifiersInfiniPath/OpenFabrics Supported Distributions and Kernels Distribution InfiniPath/OpenFabrics Supported Kernels Sles10 Setting Up Your EnvironmentCompiler Support Rhel5 Specific Component Requirements Choose the Appropriate Download FilesRequired OS Packages OS Distribution Required Packages Available Packages for QLogic Ofed 1.4 Release Package Description Installation Documentation Install InfiniPath and OpenFab Kit on Install QLogicIB-Basic with the Installer Tool You will see a screen similar to this Ofed SDP Then type n to proceed to the next screen Next screen contains these choices Configure Ofed IP over IB IPV4 addresses now? n # ./INSTALL --32bit Additional component names allowed for -Eand -Doptions Install -Clists all the available components, which includeInstall Options Command Meaning About rpm Installation Using rpm to Install InfiniPath and OpenFabrics Become root, then RPM Organization Next, install all non-prefixed RPMs Install QLogic Ofed User-level Software with the rpm Command Finally, install the prefixed RPMs in /usr/mpi/qlogic Software Installation Install Frontend and Compute Nodes Install QLogic Ofed Using Rocks Rocks Installation on an Existing Frontend Node Use the following command to rebuild the entire cluster Install QLogic Ofed Using a Platform OCS KitInstall FastFabric Software CD/ISO Image Then use the following command for each node MPI include files are Install Additional SoftwareInstalled Layout Installing Lustre Removing Software Packages Uninstalling Using the Installer ToolUninstalling InfiniPath and OpenFabrics RPMs Qlc is a part of all the InfiniPath package names Uninstalling Software with Rocks or Platform OCSDowngrading RPMs Preliminary InfiniPath and OpenFabrics Driver Overview Configuring Drivers and ServicesOpenFabrics Drivers and Services Configuration and Startup Type Configuring the IPoIB Network InterfaceTo verify the configuration, type Output from this command will be similar to this OpenSM Command to disable it on reboot is Etc/init.d/opensmd You can stop opensmd again like thisTo use Ofed SRP, follow these steps Guid Output, look for lines similar to these Configuring and Administering the Vnic Interface # ibqlgcvnicquery Preliminary Etc/infiniband/qlgcvnic.cfg Create the Vnic interfaces using the configuration file RX CSUM=TRUE HEARTBEAT=100 Preliminary To restart the QLogic Vnic driver, run the following command This information is collected from MPI over uDAPL On every node, type the following command as root Other Configuration Changing the MTU Size Etc/infiniband/openib.conf Managing the InfiniPath Driver Device files are Configure InfiniPath Driver StateTo enable the driver, use the command as root Start, Stop or Restart InfiniPath Unloading the Driver/Modules Manually Further Information on Configuring and Loading DriversSequence of commands to restart the driver are as follows Preliminary Adapter and Other Settings Installation Verification and Additional SettingsLED Link and Data Indicators Hostname1 Hostname2 Customer Acceptance UtilityThis will also show link speed Ipath checkout Options Preliminary Some HTX Motherboards May Need Two or More CPUs in Use Installation TroubleshootingHardware Issues Node Spontaneously Reboots Issue with Supermicro H8DCE-HTe and QHT7040 Bios SettingsSoftware Installation Issues Enable Advanced Configuration and Power Interface Acpi # yum install kernel-devel Resolving Conflicts Mpirun Installation Requires 32-bit Support Check your distribution for the exact RPM name Configuration IssuesLockable Memory Error on Initial Installation of InfiniPath Outdated ipathether Configuration Setup Generates Error Eth2 error fetching interface information Device not found Verify Write Combining is Working Write CombiningIntroduction Mtrr Mapping and Write Combining Edit Bios Settings to Fix Mtrr IssuesPAT and Write Combining Use the ipathmtrr Script to Fix Mtrr Issues Preliminary Configuration File Name Description Configuration FilesTable C-1. Configuration Files Qlgcvnictools subdirectory Also installed in /etc/infinibandSample and Template Files Description With the Ofed documentation, or Different Nodes May Use Different RPMs Package DescriptionsPackage Names with the QLogicIB-Basic Download InfiniPath and OpenFabrics RPMs InfiniPath RPM Version Numbers and IdentifiersOpenFabrics RPM Names Table D-2. InfiniPath/RPMs Documentation RPMsInfiniPath RPMs Table D-1. Documentation/RPMs Table D-5. OpenFabrics/RPMs OpenFabrics RPMsTable D-3. InfiniPath-Devel/RPMs Table D-4. InfiniPath-MPI/RPMs Ibutils-xxx.x8664.rpm Dapl-xxx.x8664.rpmDapl-utils-xxx.x8664.rpm Ibsim-xxx.x8664.rpm Librdmacm-xxx.x86 64.rpm Libibverbs-xxx.x8664.rpmLibibverbs-utils-xxx.x8664.rpm Libipathverbs-xxx.x86 64.rpm Table D-6. OpenFabrics-Devel/RPMs Table D-7. OpenSM/RPM Table D-10. Other HCAs-Devel/RPMs Other HCAsTable D-8. OpenSM-Devel/RPM Table D-9. Other HCAs/RPMs Table D-11. OtherMPIs/RPMs Other MPIs Openmpipathscaleqlc-xxx.yyy.x8664.rpm Openmpigcc-xxx.x8664.rpmOpenmpigccqlc-xxx.yyy.x8664.rpm Openmpiintelqlc-xxx.yyy.x8664.rpm a Preliminary Index Form factors for HCAs Hardware installation overview Module Hardware requirementsHTX motherboards may required two or more CPUs A-1 Mpi-selector 5-15,5-18 Skip=LIST 7-3Software VNIC, configurationPage Preliminary