IBM L2/3 2.2.3 Gigabit Ethernet path

Chapter 2. IBMEserver BladeCenter overview 9

2.2.3 Gigabit Ethernet path

Figure 2-5 on page 10 illustrates the Gigabit Ethernet path. Each processor blade has a

minimum of two and a maximum of four EtherLAN interfaces. In particular, the BladeCenter

HS20 processor blade has two serializer/deserializer SERDES-based Gb Ethernet interfaces,

one for each midplane connector. With a daughter card installed, two more network interfaces

can be added. Each switch module (SW Module) receives one LAN input from each

processor blade, for a total of 14 inputs.

The following partial listing illustrates the routing:

򐂰Processor blade 1 LAN 1→ Switch Module 1 input INT1

򐂰Processor blade 1 LAN 2→ Switch Module 2 input INT1

򐂰Processor blade 1 LAN 3→ Expansion Sw itch Module 3 input INT1

򐂰Processor blade 1 LAN 4→ Expansion Sw itch Module 4 input INT1

򐂰Processor blade 2 LAN 1→ Switch Module 1 input INT2

򐂰Processor blade 2 LAN 2→ Switch Module 2 input INT2

򐂰Processor blade 2 LAN 3→ Expansion Sw itch Module 3 input INT2

򐂰Processor blade 2 LAN 4→ Expansion Sw itch Module 4 input INT2

On processor blade, LAN 1 and LAN 2 are the on-board SERDES Gbit Ethernet interfaces,

and are routed to Switch Module 1 and Switch Module 2, respectively, for every processor

blade. LAN 3 and LAN 4 go to the Expansion Switch Modules 3 and 4, respectively, and are

only to be used when a daughter card is installed. Unless a daughter card is installed in one

or more processor blades, there is no need for Switch Modules 3 and 4. Further, the switch

modules have to be compatible with the LAN interface generated by the processor blade. If a

Fibre Channel daughter card is installed in a BladeCenter HS20 processor blade, Switch

Modules 3 and 4 must also be Fibre Channel-based, and any daughter cards installed in the

remaining BladeCenter HS20 processor blades must be Fibre Channel.

Note: On the Nortel Networks L2/3 GbESM, the internal Ethernet ports on the switch are

referred to as MGT1 and MGT2. For more information beyond this generic illustration, see

Chapter 4, “Nortel Networks Layer 2/3 GbE Switch Module architecture” on page21.

Contents

Main Page Page Page Contents Page Page Page Notices Trademarks Preface The team that wrote this Redpaper Become a published author Comments welcome Page Page Page overview 2.1 The IBM Eserver BladeCenter product family 2.1.1 IBM Eserver BladeCenter storage solutions 2.1.2 IBM Eserver BladeCenter system management Management foundation Advanced server management Deployment and update management 2.2 IBM Eserver BladeCenter architecture 2.2.1 The midplane Chapter 2. IBMEserver BladeCenter overview 7 ^ Midplane Lower Section IBM Figure 2-3 Internal picture of the upper and lower midplane of the BladeCenter chassis Figure 2-2 Midplane view ^ IBM BladeCenter - Management Module Ethernet Interface 2.2.3 Gigabit Ethernet path 10 2.3 IBM Eserver HS20 architecture ^ Midplane (Lower Section) IBM BladeCenter - Gigabit Ethernet path 8843 HS20 Block Diagram Servicing the IBM ^ HS20 (M/T 8843) and Blade Storage Expansion-II Option 2.4 Stand-alone configuration tools Page Page Page 3.1 Product description Page 3.2 Value proposition Product strength Leadership features and function Competitive advantage 3.3 Supported hardware Product shipment group Page Page 4.1 Nortel GbESM architecture overview Internal Layer 2 traffic flow in the Nortel Networks L2/3 GbESM 4.2 Nortel Networks L2/3 GbESM block diagram Chapter 4. Nortel Networks Layer 2/3 GbE Switch Module architecture 25 4.2.1 Nortel Networks L2/3 GbESM ports specific roles Figure 4-6 Connections on the Nortel GbESM IBM eServer BladeCenter Figure 4-5 Connections on the Nortel GbESM Nortel Networks L2/3 GbESM Module INT1 14 INT1 - 14 The management VLAN IP address information is not lost during factory reset Page Page 30 5.1 Nortel Networks L2/3 GbESM management connectivity 5.1.1 Out-of-band management Management Module (Path 1) Serial port (Path 2) 5.1.2 In-band management External Ethernet ports (Path 3A) Internal Ethernet ports (Path 3B) In-band management considerations 5.2 Nortel Networks L2/3 GbESM user interface 5.2.1 IBM Eserver BladeCenter Management Module and I2C 5.2.2 Command-line interface Main Menu commands Global commands Navigation commands Configuration control commands Additional commands Upgrading the firmware Capturing the current configuration Configuring user accounts 5.2.3 Browser Based Interface 5.2.4 SNMP management - IBM Director 5.3 Multiple Nortel Networks L2/3 GbESMs in a BladeCenter Page 6.1 IBM Eserver BladeCenter system 6.1.1 Management Module firmware 6.1.2 Management Module network interface Establishing a physical connection to the Management Module Accessing the Management Module Web interface Configuring the Management Module network interfaces 6.1.3 I/O module management tasks Nortel Networks L2/3 GbESM setup and configuration Enabling Nortel Networks L2/3 GbESM uplink ports through the Management Module Nortel Networks L2/3 GbESM firmware download Determining the level of Nortel switch software Obtaining the latest level of switch software Upgrading the switch software 6.2 Blade server initial configuration 6.2.1 Firmware update UpdateXpress Getting started Firmware update 6.2.2 Operating systems Microsoft Windows 2003 Broadcom driver installation Red Hat Linux Broadcom driver installation 6.2.3 Broadcom Advanced Control Suite installation Page 6.3 Firmware and device drivers used in this example Page Page 7.1 Standards and technologies 7.1.1 VLAN tagging - 802.1Q 7.1.2 Link Aggregation and LACP - 802.3ad and 802.3-2002 channel-group 7.1.3 Spanning Tree - 802.1D, 802.1w, 802.1s 7.1.4 Routing Information Protocol - RFC1058 and RFC2453 7.1.5 Open Shortest Path First (OSPF) - RFC1257, RFC2328, and others 7.1.6 Virtual Router Redundancy Protocol (VRRP) - RFC 3768 7.1.7 Where standards originate and how to get them 7.2 Summary of sample configurations 7.2.1 Basic Layer 2 configuration 7.2.2 Advanced Layer 2 configurations 7.2.3 Layer 3 configuration - static routing static routing 7.2.4 Layer 3 configurations - dynamic routing routing 7.3 Introduction to High Availability 7.3.1 Introduction to trunk failover disabled Configuration X 7.3.2 Introduction to NIC Teaming X 7.3.3 Introduction to VRRP VRRP Priority 7.3.4 Some important rules for ensuring High Availability 7.4 Guidelines for attaching the BladeCenter to a network 7.4.1 Guidelines and comments Cable type selection (cross-over or straight-through) Fiber connections Speed or duplex selection 7.5 Base configurations for examples in this document 7.5.1 Hardware and software used for lab environment IBM Eserver BladeCenter configuration Nortel Networks L2/3 GbESM Cisco 3560G switch 7.5.2 Preconfiguration preparation 7.5.3 Base configuration common to all examples Management Module settings for Nortel GbESMs IP address and VLAN scheme y Blade server configuration with BASP 7.6 Basic Layer 2 entry topology 70 Figure 7-3 Basic Layer 2 topology with 802.1Q tagging and trunk failover 7.6.1 Layer 2 configuration with 802.1Q tagging and trunk failover Figure 7-3 illustrates the first basic Layer 2 topology. Summary of disconnect procedure to be performed for each example Cisco 3560G switch configuration Page Nortel Networks L2/3 GbESM switch configuration Page Page 76 Re-enable the ports Enabling the ports on Core1 and Core2 Page 78 Page 7.6.2 Basic topology conclusions 7.7 Advanced Layer 2 topology sample configurations Page Summary of disconnect procedure, to be performed for each example 7.7.1 Dynamic link aggregation IEEE 802.3ad (LACP) Configure the Nortel GbESMs for IEEE 802.1Q tagged LACP trunk Configure the Cisco Core1 and Core2 for tagged LACP trunks Verify the LACP trunk status 7.7.2 Common Spanning Tree configuration - IEEE 802.1D and PVST Configure the Nortel GbESM_1 and GbESM_2 for Spanning Tree Groups Configure the Cisco Core1 and Core2 for Per VLAN Spanning Tree Reenable the ports Verify the Spanning Tree status Page 88 Page Design or topology remarks Full configuration snapshots Page 92 Page 94 7.7.3 Rapid Spanning Tree IEEE 802.1w Prepare the Nortel GbESM_1 and GbESM_2 for Rapid Spanning Tree Configure the Cisco Core1 and Core2 for Rapid-PVST Reenable the ports Verify the Spanning Tree status Design remarks Full configuration snapshots 98 Page 100 7.7.4 Multi-Spanning Tree IEEE 802.1s Configurations for use of MST Key configuration commands to verify MSTP Page Page Page 106 Page 7.8 Layer 3 topology sample configurations Implications of using Layer 3 Reasons for choosing a Layer 3 design 7.8.1 Layer 3 sample configuration with static routing and VRRP Merits of this configuration Summary of disconnect procedure to be performed for each example Summary of IP Addressing used in this sample Table7-1 summarizes the IP addresses used in this sample. Figure 7-6 also illustrates these. Configure VLANs and subnets for the uplink ports Page 114 VRRP configuration Configuration for GbESMs Configuration of Hot-standby for VRRP Configuration on Core switches to work with VRRP Configuration on server blades to work with VRRP Reconnect procedure when configuration is ready Verifying operation of the configuration 118 Page 120 Page 122 Page 124 7.8.2 Dynamic routing options OSPF/RIP Merits of this configuration 126 Summary of IP Addressing used in this sample Figure 7-7 Layer 3 Sample Configuration (Dynamic routing) Table7-2 summarizes the IP addresses in use. Figure 7-7 also shows these. Table 7-2 IP Addresses for Layer 3 Sample Configuration (Dynamic routing) Configure additional VLANs and subnets for the uplink ports 128 Page Configuring RIP Configuring OSPF Verifying operation of this configuration GbESM commands to verify RIP 132 Cisco commands to verify RIP GbESM commands to verify OSPF 134 Page 136 Cisco commands to verify OSPF Page 138 Page 140 Page 142 7.9 Configuration for Extreme switches 144 Page 146 Page 148 Page Page Page 8.1 SOL overview 8.2 General rules to establish an SOL connection 8.3 Configuring SOL for use with the Nortel GbESM 8.4 SOL use during Nortel Networks L2/3 GbESM experiments Page Page 9.1 Basic rules and unique symptoms 9.1.1 Basic rules both sides of the connection. This is an important best practice for any switch-to-switch 9.2 Nortel Networks L2/3 GbESM troubleshooting methodology 9.2.1 General comments on troubleshooting 9.3 Systematic approach 9.3.1 Problem definition 9.3.2 Data collection Test connectivity Details from affected users Nortel GbESM (Alteon OS) switch dumps Network logs 9.3.3 Data analysis 9.3.4 Action plan creation 9.3.5 Action plan implementation 9.3.6 Observation of results 9.3.7 Problem resolution 9.4 Troubleshooting tools Ping Traceroute Telnet/SSH Difference tool TFTP server Syslog server Web load generator Protocol analyzer Page Page 10.1 Placing a call to IBM 10.2 Online services 10.3 Ordering information Page Page Abbreviations and acronyms 170 Related publications IBM Redbooks Other publications Online resources Page How to get IBM Redbooks Help from IBM Page Page Redpaper Back cover INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter