Large HP XC Cluster of Server Blades | HP XC System Software 3.2.1

Figure 2-2 Interconnection Diagram for a Medium Sized HP XC Cluster of Server Blades

GigE Switch

Enclosure 1			Enclosure 2		. . .	Enclosure 32
Switch 1			Switch 1			Switch 1



Enclosure 1			Enclosure 2			Enclosure 3
Server Blades			Server Blades			Server Blades
NIC1			NIC1			NIC1

May be any

ProCurve managed

GigE switch

(smaller systems may use ProCurve 2824)

ProCurve 2848

			ProCurve 2650

				. . .
Enclosure 1					Enclosure 32
	OA				OA
	OA				OA
Ext Link			Enclosure 2		Ext Link
			OA
			Ext Link

Smaller systems

may use ProCurve 2626

					Enclosure 32
					Enclosure 32
						GigE switch

				Enclosure 4
			Secondary OA Ext Link				Primary OA Ext Link
			Secondary OA Ext Link				Primary OA Ext Link
			GigE Switch
			GigE Switch


				Enclosure 3
			Secondary OA Ext Link				Primary OA Ext Link
			GigE Switch


				Enclosure 2
			Secondary OA Ext Link				Primary OA Ext Link

			GigE Switch
			GigE Switch


				Enclosure 1
			Secondary OA Ext Link				Primary OA Ext Link

GigE Switch

ProCurve 2650

Enclosure 32

OA

Ext Link

Large HP XC Cluster of Server Blades

Figure 2-3 (page 37) illustrates a large HP XC cluster of eight regions, with 32 enclosures per region.

There is a Gigabit Ethernet switch for the HP XC system that is connected to a Gigabit Ethernet switch in each region.

The Gigabit Ethernet switch in a region is connected to each enclosure's Gigabit Ethernet switch in bay 1 and to a ProCurve 2650 switch. The ProCurve 2650 switch is connected to the Primary Onboard Administrator External Link of each enclosure.

36 Cabling Server Blades

Image 36

HP XC System Software 3.2.1 manual Large HP XC Cluster of Server Blades

Contents

HP XC System Software Hardware Preparation Guide Page Table of Contents Preparing Individual Nodes 139 141143 147 List of Figures Server Blade Insight DisplayPage ILO Settings for HP ProLiant DL380 G4 and G5 Nodes List of Tables 137 Page About This Document Intended AudienceNew and Changed Information in This Edition Typographic Conventions HP XC and Related HP Products Information Ctrl+x HP XC Program Development Environment Related Information Https//computing.llnl.gov/linux/slurm/documentation.html Http//supermon.sourceforge.net Linux Administration Unleashed, by Thomas Schenk, et al Manpages HP Encourages Your CommentsPage Hardware and Network Overview Supported Cluster PlatformsSupported Processor Architectures and Hardware Models Supported Processor Architectures and Hardware Models Supported Server Blade Combinations Supported HP ProLiant Server Blade Models Server Blade Enclosure Components HP BladeSystem c7000 Enclosure Server Blade Enclosure Components General Configuration Guidelines HP BladeSystem c3000 Enclosure HP BladeSystem c3000 Enclosure Front and Rear Views HP BladeSystem c3000 Enclosure Bay Locations Front View HP BladeSystem c-Class Onboard Administrator HP BladeSystem c3000 Enclosure Bay Locations Rear View Server Blade Mezzanine Cards Server Blade Interconnect ModulesInsight Display Supported Console Management Devices Supported Console Management DevicesCP3000 CP3000BL Administration Network Overview CP4000CP4000BL CP6000 Administration Network Console Branch Interconnect Network Large-Scale Systems Supported Interconnects Cabling Server Blades Blade Enclosure OverviewNetwork Overview Small HP XC Cluster of Server Blades Medium Sized HP XC Cluster of Server Blades Large HP XC Cluster of Server Blades Cabling for the Administration Network GigE Switch Administration Network Connections Cabling for the Console Network Configuring a Gigabit Ethernet Interconnect Cabling for the Interconnect Network Configuring an InfiniBand Interconnect Gigabit Ethernet Interconnect Connections Configuring the External Network Option Cabling for the External Network Ethernet PCI Cards Ethernet Creating VLANs Cfg Making Node and Switch Connections Trunking and Switch ChoicesCabinets Switches Specialized Switch Use Administrator Passwords on ProCurve Switches Switch Port ConnectionsSupported Switch Models Node and Switch Connections on a Typical System Super Root Switch Wire Trunking Root Administration Switch ProCurve 2848 Root Administration Switch Root Console Switches ProCurve 2824 Root Administration Switch ProCurve 2650 Switch ProCurve 2626 Switch Branch Administration Switches 10 ProCurve 2610-24 Root Console Switch Branch Console Switches 11 ProCurve 2848 Branch Administration Switch 13 ProCurve 2650 Branch Console Switch Interconnect Connections Administration Network Interconnect Connections QsNet Interconnect ConnectionsGigabit Ethernet Interconnect Connections Myrinet Interconnect Connections InfiniBand Interconnect Connections Preparing Individual Nodes Firmware Requirements and DependenciesFirmware Dependencies Switches Interconnect Ethernet Port Connections on the Head Node General Hardware Preparations for All Cluster PlatformsEthernet Ports on the Head Node Setting the Onboard Administrator Password Preparing HP ProLiant DL140 G2 and G3 Nodes HP ProLiant DL140 G2 and DL140 G3 Server Rear View Bios Settings for HP ProLiant DL140 G2 Nodes Bios Settings for HP ProLiant DL140 G3 Nodes Means disabledExcept the head node Node Preparing HP ProLiant DL160 G5 Nodes HP ProLiant DL160 G5 Server Rear View Bios Settings for HP ProLiant DL160 G5 Nodes Value to Preparing HP ProLiant DL360 G4 Nodes Otherwise, set this value to ILO Settings for HP ProLiant DL360 G4 Nodes Bios Settings for HP ProLiant DL360 G4 Nodes Preparing HP ProLiant DL360 G5 Nodes HP ProLiant DL360 G5 Server Rear View ILO Settings for HP ProLiant DL360 G5 Nodes Bios Settings for HP ProLiant DL360 G5 Nodes Preparing HP ProLiant DL380 G4 and G5 Nodes HP ProLiant DL380 G4 Server Rear View 10 iLO Settings for HP ProLiant DL380 G4 and G5 Nodes 11 Bios Settings for HP ProLiant DL380 G4 Nodes 12 Bios Settings for HP ProLiant DL380 G5 Nodes 12 Bios Settings for HP ProLiant DL380 G5 Nodes Preparing HP ProLiant DL580 G4 Nodes HP ProLiant DL580 G4 Server Rear View 13 iLO Settings for HP ProLiant DL580 G4 Nodes 14 Bios Settings for HP ProLiant DL580 G4 Nodes Preparing HP ProLiant DL580 G5 Nodes 15 iLO Settings for HP ProLiant DL580 G5 Nodes HP ProLiant DL580 G5 Server Rear View 16 Bios Settings for HP ProLiant DL580 G5 Nodes Preparing HP xw8200 and xw8400 Workstations 17 Bios Settings for xw8200 Workstations Set the following boot order on all nodes exceptOn the head node, set the boot order so that CD-ROM is listed before the hard disk 18 Bios Settings for xw8400 Workstations Controller settings are set to EnabledPreparing HP xw8600 Workstations Head node 19 Bios Settings for xw8600 Workstations 20 Boot Order for HP ProLiant Server Blades Preparing the Hardware for CP3000BL Systems Select Enable HP Static High Performance Mode Preparing Individual Nodes Preparing the Hardware for CP4000 AMD Opteron Systems Preparing HP ProLiant DL145 Nodes 21 Bios Settings for HP ProLiant DL145 Nodes Enabled for all nodes except the headDisabled for the head node Nodes except the head node CD-ROM Preparing HP ProLiant DL145 G2 and DL145 G3 Nodes 12 HP ProLiant DL145 G2 Server Rear View 22 Bios Settings for HP ProLiant DL145 G2 Nodes 13 HP ProLiant DL145 G3 Server Rear View Set the following boot order on On Board for Ethernet 1All nodes except the head node Removable Devices 23 Bios Settings for HP ProLiant DL145 G3 Nodes CD-ROM DriveMBA v9.0.6 Slot Preparing HP ProLiant DL165 G5 Nodes 14 HP ProLiant DL165 G5 Server Rear View 24 Bios Settings for HP ProLiant DL165 G5 Nodes Preparing HP ProLiant DL365 Nodes 25 iLO Settings for HP ProLiant DL365 Nodes 15 HP ProLiant DL365 Server Rear View 26 Rbsu Settings for HP ProLiant DL365 Nodes Enabled all nodes except the head nodeDisabled head node only Preparing HP ProLiant DL365 G5 Nodes 16 HP ProLiant DL365 G5 Server Rear View 27 iLO Settings for HP ProLiant DL365 G5 Nodes 28 Rbsu Settings for HP ProLiant DL365 G5 Nodes Preparing HP ProLiant DL385 and DL385 G2 Nodes 17 HP ProLiant DL385 Server Rear View 29 iLO Settings for HP ProLiant DL385 Nodes 18 HP ProLiant DL385 G2 Server Rear View 30 iLO Settings for HP ProLiant DL385 G2 Nodes Is configurable 31 Rbsu Settings for HP ProLiant DL385 Nodes 32 Rbsu Settings for HP ProLiant DL385 G2 Nodes Preparing HP ProLiant DL385 G5 Nodes 33 iLO Settings for HP ProLiant DL385 G2 Nodes 19 HP ProLiant DL385 G5 Server Rear View Preparing HP ProLiant DL585 and DL585 G2 Nodes 20 HP ProLiant DL585 Server Rear View 34 iLO Settings for HP ProLiant DL585 Nodes 35 iLO Settings for HP ProLiant DL585 G2 Nodes Preparing Individual Nodes 36 Rbsu Settings for HP ProLiant DL585 Nodes 37 Rbsu Settings for HP ProLiant DL585 G2 NodesSet the following boot order on all Nodes except the head node Set the following boot order on the head node Preparing HP ProLiant DL585 G5 Nodes 38 iLO Settings for HP ProLiant DL585 G5 Nodes 22 HP ProLiant DL585 G5 Server Rear View 39 Rbsu Settings for HP ProLiant DL585 G5 Nodes Preparing HP ProLiant DL785 G5 Nodes 23 HP ProLiant DL785 G5 Server Rear View 40 iLO Settings for HP ProLiant DL785 G5 Nodes 41 Rbsu Settings for HP ProLiant DL785 G5 Nodes Preparing HP xw9300 and xw9400 Workstations 24 xw9300 Workstation Rear View 42 Setup Utility Settings for xw9300 Workstations 43 Setup Utility Settings for xw9400 WorkstationsSet the boot order on the head node as follows Preparing Individual Nodes 44 Boot Order for HP ProLiant Server Blades Preparing the Hardware for CP4000BL Systems Select Enable OS Control Mode 45 Additional Bios Setting for HP ProLiant BL685c Nodes Setting Static IP Addresses on Integrity Servers About the EFI Boot Manager User InterfacePreparing the Hardware for CP6000 Intel Itanium Systems Preparing HP Integrity rx1620 and rx2600 Nodes   MP Main Menu appears Preparing HP Integrity rx2620 Nodes 28 HP Integrity rx2620 Server Rear View Select Add Boot Entry Preparing HP Integrity rx2660 Nodes 29 HP Integrity rx2660 Server Rear View Select Add Boot Entry Preparing HP Integrity rx4640 Nodes 30 HP Integrity rx4640 Server Rear View MP Local Preparing HP Integrity rx8620 Nodes 31 HP Integrity rx8620 Core IO Board Connections Preparing the Hardware for CP6000 Intel Itanium Systems Read-only use Ecf to attach to console Exit the Boot Option Maintenance menu Preparing the Hardware for CP6000BL Systems Mpcm pc -on -nc Preparing Individual Nodes Troubleshooting ILO2 Devices1 iLO2 Devices Can Become Unresponsive 140 Establishing a Connection Through a Serial Port 142 Server Blade Configuration Examples InfiniBand Interconnect With Full-Height Server Blades Server Blade Configuration Examples InfiniBand Interconnect With Mixed Height Server Blades InfiniBand Interconnect With Mixed Height Server Blades 146 Glossary Network Availability set Fcfs Ipmi LVS PXE SVA Index Slurm Onboard administrator defined Vlan 157 XCHWP-321c