Q-Logic IB0056101-00 G.02, OFED Preliminary

Draft

Preliminary

© 2005-2009 QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic and the QLogic logo

are registered trademarks of QLogic Corporation. QLA, SANsurfer, InfiniPath, and SilverStorm are trademarks or registered trademarks of QLogic

Corporation. AMD Opteron is a trademark of Advanced Microdevices Inc. BladeCenter and IBM are registered trademarks of International Business

Machines Corporation. DataDirect Networks is a trademark of DataDirect Networks, Inc. EMCORE is a trademark of EMCORE Corporation. HTX

is a trademark of the HyperTransport Technology Consortium. IBM and BladeCenter are registered trademarks of International Business Machines

Corporation. InfiniBand is a trademark and service mark of the InfiniBand Trade Association. Intel is a registered trademark of Intel Corporation.

Linux is a registered trademark of Linus Torvalds. LSI Logic and Engenio are trademarks or registered trademarks of LSI Logic Corporation.

Lustre is a registered trademark of Cluster File Systems, Inc. Mellanox is a registered trademark and ConnectX is a trademark of Mellanox

Technologies, Inc. PathScale is a trademark of PathScale LLC. PCI Express is a registered trademark of PCI-SIG Corporation. Red Hat and

Enterprise Linux are registered trademarks of RedHat, Inc. Supe rmicro is a registered trademark of Super Micro Computer Inc. SUSE is a

registered trademark of Novell Inc. Zarlink is a trademark of Zarlink Semiconductor Inc. All other brand and product names are trademarks or

registered trademarks of their respective owners. Information supplied by QLogic Corporation is believed to be accurate and reliable. QLogic

Corporation assumes no responsibility for any errors in this brochure. QLogic Corporation reserves the right, without notice, to make changes in

product design or specifications.

Corporate Headquarters QLogic Corporation 26650 Aliso Viejo Parkway Aliso Viejo, CA 92656 949.389.6000 www.qlogic.com

Europe Headquarters QLogic (UK) LTD. Quatro House Lyon Way, Frimley Camberley Surrey, GU16 7ER UK +44 (0) 1276 804 670

Contents

Main Document Revision History Page Page web site for updated information on supported compilers. This information is not on the web site. ates Error RPMs to Use for Each Node in a Cluster from Software Installation section to Appendix Page Page Table of Contents 1 Introduction 2 Feature Overview 3 Step-by-Step Installation Checklist 4 Hardware Installation 5 Software Installation 6 Configuring Drivers and Services 7 Installation Verification and Additional Settings A Installation Troubleshooting B Write Combining C Configuration Files D Package Descriptions List of Figures List of Tables Page 1 Who Should Read this Guide How this Guide is Organized Overview Interoperability Conventions Used in this Guide Documentation Contact Information Page 2 Features Other Changes Continued Support Page Software Components 3 Installation Checklist Hardware Installation Software Installation Page Page 4 Hardware Installation Requirements Hardware Form Factors Cabling and Switches Optical Fibre Option Configuring the BIOS Safety with Electricity Unpacking Information Verify the Package Contents List of the Package Contents Page Figure 4-1. QLogic QLE7280 with IBA7220 ASIC Figure 4-2. QLogic QLE7140 Card with Riser, Top View Unpacking the QLogic Adapter Hardware Installation Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser Dual Adapter Installation Installation Steps Page Page Hardware Installation for QHT7140 with HTX Riser Page Page Hardware Installation for QLE7240, QLE7280, and QLE7140 Without a PCI Express Riser Hardware Installation for the QHT7140 Without an HTX Riser Switch Configuration and Monitoring Cabling the Adapter to the InfiniBand Switch Completing the Installation 5 Cluster Setup Types of Nodes in a Cluster Environment Supported Linux Distributions Distribution Identifiers NOTE: Support for RHEL4 U4 and SLES 10.0 has been removed. Table 5-2. Distribution Identifiers Distribution Identifier Used On Compiler Support Setting Up Your Environment Choose the Appropriate Download Files Page Page Install QLogicIB-Basic with the Installer Tool Page Page The next screen contains these choices: Page Page About rpm Installation Using rpm to Install InfiniPath and OpenFabrics Page RPM Organization Install QLogic OFED User-level Software with the rpm Command Page Rebuilding or Reinstalling the kernel-ib Driver with rpm After a Kernel Upgrade Install QLogic OFED Using Rocks Install Frontend and Compute Nodes Rocks Installation on an Existing Frontend Node Install QLogic OFED Using a Platform OCS Kit Install FastFabric Software CD/ISO Image Install Additional Software Installing Lustre Installed Layout Removing Software Packages Uninstalling Using the Installer Tool Uninstalling InfiniPath and OpenFabrics RPMs Uninstalling Software with Rocks or Platform OCS Downgrading RPMs Page 6 Services InfiniPath and OpenFabrics Driver Overview OpenFabrics Drivers and Services Configuration and Startup Configuring the IPoIB Network Interface OpenSM SRP Using QLogic SRP Using OFED SRP In the output, look for lines similar to these: 4. Find the result that corresponds to the target you want, and echo it into the Configuring and Administering the VNIC Interface Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup IB0056101-00 G.02 Preliminary 6-7 #ib_qlgc_vnic_query Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup 6-8 Preliminary IB0056101-00 G.02 Page Page Page Page MPI over uDAPL Other Configuration: Changing the MTU Size Managing the InfiniPath Driver Configure InfiniPath Driver State Start, Stop or Restart InfiniPath Unloading the Driver/Modules Manually Further Information on Configuring and Loading Drivers Page 7 Additional Settings LED Link and Data Indicators Adapter and Other Settings Customer Acceptance Utility Page Page A Troubleshooting Hardware Issues Node Spontaneously Reboots Some HTX Motherboards May Need Two or More CPUs in Use BIOS Settings Enable Advanced Configuration and Power Interface (ACPI) Issue with Supermicro H8DCE-HTe and QHT7040 Software Installation Issues Missing Kernel RPM Errors Page Resolving Conflicts openmpi_gcc Fails to Install Because of Dependency on gfortran (RHEL 4) mpirun Installation Requires 32-bit Support Lockable Memory Error on Initial Installation of InfiniPath Configuration Issues ibsrpdm Command Hangs When Two HCAs are Installed but Only Unit 1 is Connected to the Switch Outdated ipath_ether Configuration Setup Generates Error Page B Introduction Verify Write Combining is Working PAT and Write Combining MTRR Mapping and Write Combining Edit BIOS Settings to Fix MTRR Issues Use the ipath_mtrr Script to Fix MTRR Issues Page C Page D Package Names with the QLogicIB-Basic Download Different Nodes May Use Different RPMs InfiniPath RPM Version Numbers and Identifiers OpenFabrics RPM Names InfiniPath and OpenFabrics RPMs Documentation RPMs InfiniPath RPMs Table D-1. Documentation/RPMs RPM Name Front End Compute Development Table D-2. InfiniPath/RPMs RPM Name Front End Compute Development OpenFabrics RPMs Page Page The development RPMs in TableD-6 are only needed on the nodes where OFED programs are compiled. Table D-6. OpenFabrics-Devel/RPMs Table D-7. OpenSM/RPM Table D-6. OpenFabrics-Devel/RPMs (Continued) Other HCAs Other MPIs Table D-11. OtherMPIs/RPMs Table D-10. Other HCAs-Devel/RPMs (Continued) Table D-11. OtherMPIs/RPMs (Continued) Page Index A B C D I K L M N O P Q S T U V W Page Preliminary