IBM 24R9718 IB Topologies, Server Switch Module and HCA Expansion Cards, Single-Switch Topology

2

Each Server Switch Module includes 14 1x ports to the backplane and 4 4x ports (in the form of 1 4x connector and 1 12x connector) on the front panel. (In the case of the BladeCenter T, the internal ports beyond the first 8 appear in the CLI but do not function.) The Server Switch Module provides fully non-blocking switching for all 18 ports. All internal 1x ports provide 2.5 Gbps connections to the HCA expansion cards on server blades. All external 4x connectors provide 10 Gbps connections to the outside IB network and can autonegotiate connection speed.

Server Switch Modules transmit information to and from BladeCenter Management Modules over Ethernet (via an internal Ethernet switch) to facilitate setup and management. After you set up a Server Switch Module and bring it online, the on-board Topspin Subnet Manager brings distributed intelligence to the IB network.

Topspin Server Switch Modules allow you to perform a field upgrade to load the latest firmware on the Server Switch Module. You can perform a firmware upgrade even following a failed upgrade. CLI and GUI interfaces both support upgrades. For details, refer to the Command Line Interface Reference Guide or InfiniBand User Guide.

Server Switch Module and HCA Expansion

Cards

Within the BladeCenter unit, Server Switch Modules manage traffic to and from HCA expansion cards on the BladeCenter hosts. Each HCA expansion card adds two IB ports to a BladeCenter host. Each HCA port connects via the unit backplane to a particular Server Switch Module bay. The first IB port of each HCA card (ib0) connects to the Server Switch Module in bay 3, and the second IB port of each HCA card (ib1) connects to bay 4. For detailed information on the HCA expansion card, refer to the InfiniBand Host Channel Adapter Expansion Card for IBM eServer BladeCenter User Guide.

Topologies

With the Server Switch Module and HCA expansion cards, you can create a non-redundant, single switch topology or a redundant, dual-switch topology.

Single-Switch Topology

When you populate just one BladeCenter module bay with a Server Switch Module, you create a bi-sectional bandwidth topology. However, this topology does not provide redundant links from the HCA expansion cards to the Server Switch Module. We strongly recommend that you implement a “Dual-Switch Topology” to avoid single points of failure.

NOTE: Specific operating systems on the blade servers may prefer that the single Server Switch Module reside in bay 3, and others may prefer bay 4. The preference depends on which HCA port the OS identifies as ib0. If your single-switch topology does not run traffic, place the module in the alternate bay.

Dual-Switch Topology

To enable IB redundancy on the BladeCenter chassis, you must install one Server Switch Module in each available bay. HCA expansion cards do not support redundant links to a single Server Switch Module bay. When you add a second Server Switch Module to the BladeCenter chassis, each port of each HCA expansion card connects to a Server Switch Module.