Q-Logic IB0056101-00 G.02, OFED instruction

QLogic HCA and QLogic OFED Software Install Guide

QLogic OFED Version 1.4

IB0056101-00 G.02

Preliminary

Image 1

Q-Logic IB0056101-00 G.02, OFED manual QLogic HCA and QLogic Ofed Software Install Guide

Contents

QLogic HCA and QLogic Ofed Software Install Guide Document Revision History Preliminary Preliminary Removed phrase and Transmission Control Ates Error Ipathether Configuration Setup GenerMoved table InfiniPath and OpenFabrics Removed note OpenFabrics programs 32-bit Preliminary Table of Contents Software Installation Installation Verification and Additional Settings Write Combining List of Figures Preliminary Who Should Read this Guide IntroductionHow this Guide is Organized Overview Conventions Used in this Guide InteroperabilityTypographical Conventions Meaning Product documentation includes Documentation Support Headquarters Contact Information Preliminary Features Feature Overview Support Other Changes Description QLogic Adapter Model Numbers „ Ofed SRP Software Components Hardware Installation Step-by-Step Installation Checklist Software Installation Preliminary Preliminary Hardware Installation Requirements Hardware InstallationHardware Adapter Models and Related Platforms Run ipath control Form Factors QLogic InfiniBand Cables Cabling and SwitchesProduct Number Description Configuring the Bios Verify the Package Contents Safety with ElectricityList of the Package Contents Unpacking Information Hardware Installation Unpacking Information QLogic QLE7280 with IBA7220 Asic Asic Unpacking the QLogic Adapter Dual Adapter Installation PCIe Slot in a Typical Motherboard QLogic PCIe HCA Assembly with Riser Card Assembled PCIe HCA with Riser Hardware Installation for QHT7140 with HTX Riser HTX Slot QLogic QHT7140 Adapter with Riser Card Assembled QHT7140 with Riser Hardware Installation for the QHT7140 Without an HTX Riser Cabling the Adapter to the InfiniBand Switch Switch Configuration and Monitoring Completing the Installation Cluster Setup Software InstallationTypes of Nodes in a Cluster Environment Supported Linux Distributions InfiniPath/OpenFabrics Supported Distributions and Kernels Distribution IdentifiersDistribution InfiniPath/OpenFabrics Supported Kernels Distribution Identifiers Compiler Support Setting Up Your EnvironmentRhel5 Sles10 Required OS Packages Choose the Appropriate Download FilesOS Distribution Required Packages Specific Component Requirements Available Packages for QLogic Ofed 1.4 Release Package Description Installation Documentation Install InfiniPath and OpenFab Kit on Install QLogicIB-Basic with the Installer Tool You will see a screen similar to this Ofed SDP Then type n to proceed to the next screen Next screen contains these choices Configure Ofed IP over IB IPV4 addresses now? n # ./INSTALL --32bit Install Options Install -Clists all the available components, which includeCommand Meaning Additional component names allowed for -Eand -Doptions About rpm Installation Using rpm to Install InfiniPath and OpenFabrics Become root, then RPM Organization Next, install all non-prefixed RPMs Install QLogic Ofed User-level Software with the rpm Command Finally, install the prefixed RPMs in /usr/mpi/qlogic Software Installation Install Frontend and Compute Nodes Install QLogic Ofed Using Rocks Rocks Installation on an Existing Frontend Node Install FastFabric Software CD/ISO Image Install QLogic Ofed Using a Platform OCS KitThen use the following command for each node Use the following command to rebuild the entire cluster Installed Layout Install Additional SoftwareInstalling Lustre MPI include files are Uninstalling InfiniPath and OpenFabrics RPMs Uninstalling Using the Installer ToolRemoving Software Packages Downgrading RPMs Uninstalling Software with Rocks or Platform OCSQlc is a part of all the InfiniPath package names Preliminary OpenFabrics Drivers and Services Configuration and Startup Configuring Drivers and ServicesInfiniPath and OpenFabrics Driver Overview To verify the configuration, type Configuring the IPoIB Network InterfaceOutput from this command will be similar to this Type OpenSM Command to disable it on reboot is To use Ofed SRP, follow these steps You can stop opensmd again like thisEtc/init.d/opensmd Guid Output, look for lines similar to these Configuring and Administering the Vnic Interface # ibqlgcvnicquery Preliminary Etc/infiniband/qlgcvnic.cfg Create the Vnic interfaces using the configuration file RX CSUM=TRUE HEARTBEAT=100 Preliminary To restart the QLogic Vnic driver, run the following command This information is collected from MPI over uDAPL On every node, type the following command as root Other Configuration Changing the MTU Size Etc/infiniband/openib.conf Managing the InfiniPath Driver To enable the driver, use the command as root Configure InfiniPath Driver StateStart, Stop or Restart InfiniPath Device files are Sequence of commands to restart the driver are as follows Further Information on Configuring and Loading DriversUnloading the Driver/Modules Manually Preliminary LED Link and Data Indicators Installation Verification and Additional SettingsAdapter and Other Settings This will also show link speed Customer Acceptance UtilityHostname1 Hostname2 Ipath checkout Options Preliminary Hardware Issues Installation TroubleshootingNode Spontaneously Reboots Some HTX Motherboards May Need Two or More CPUs in Use Software Installation Issues Bios SettingsEnable Advanced Configuration and Power Interface Acpi Issue with Supermicro H8DCE-HTe and QHT7040 # yum install kernel-devel Resolving Conflicts Mpirun Installation Requires 32-bit Support Lockable Memory Error on Initial Installation of InfiniPath Configuration IssuesOutdated ipathether Configuration Setup Generates Error Check your distribution for the exact RPM name Eth2 error fetching interface information Device not found Introduction Write CombiningVerify Write Combining is Working PAT and Write Combining Edit Bios Settings to Fix Mtrr IssuesMtrr Mapping and Write Combining Use the ipathmtrr Script to Fix Mtrr Issues Preliminary Table C-1. Configuration Files Configuration FilesConfiguration File Name Description Sample and Template Files Description Also installed in /etc/infinibandWith the Ofed documentation, or Qlgcvnictools subdirectory Package Names with the QLogicIB-Basic Download Package DescriptionsDifferent Nodes May Use Different RPMs OpenFabrics RPM Names InfiniPath RPM Version Numbers and IdentifiersInfiniPath and OpenFabrics RPMs InfiniPath RPMs Documentation RPMsTable D-1. Documentation/RPMs Table D-2. InfiniPath/RPMs Table D-3. InfiniPath-Devel/RPMs OpenFabrics RPMsTable D-4. InfiniPath-MPI/RPMs Table D-5. OpenFabrics/RPMs Dapl-utils-xxx.x8664.rpm Dapl-xxx.x8664.rpmIbsim-xxx.x8664.rpm Ibutils-xxx.x8664.rpm Libibverbs-utils-xxx.x8664.rpm Libibverbs-xxx.x8664.rpmLibipathverbs-xxx.x86 64.rpm Librdmacm-xxx.x86 64.rpm Table D-6. OpenFabrics-Devel/RPMs Table D-7. OpenSM/RPM Table D-8. OpenSM-Devel/RPM Other HCAsTable D-9. Other HCAs/RPMs Table D-10. Other HCAs-Devel/RPMs Table D-11. OtherMPIs/RPMs Other MPIs Openmpigccqlc-xxx.yyy.x8664.rpm Openmpigcc-xxx.x8664.rpmOpenmpiintelqlc-xxx.yyy.x8664.rpm a Openmpipathscaleqlc-xxx.yyy.x8664.rpm Preliminary Index Form factors for HCAs Hardware installation overview HTX motherboards may required two or more CPUs A-1 Hardware requirementsModule Mpi-selector 5-15,5-18 Skip=LIST 7-3Software VNIC, configurationPage Preliminary