IBM 9077 D, E

318 IBM 9077 SP Switch Router: Get Connected to the SP Switch

config_netstat 70

csconfig 79

dev1config 87, 95

Eannotator 82

Efence 51, 99

Efence -autojoin 99

Eprimary 51

Estart 51, 99, 101

Eunfence 51, 99, 101

gratm 116, 121

grcard 97, 98, 101, 109

grconslog 79

grf_update 301

grifconfig 94

grreset 95, 97, 99, 109

grrmb 109

grrt 118, 132

grsite 96

grsite --perm 77, 96

grsnapshot 97

grstat 99, 119, 133

grwrite 79, 107, 109

grwrite -vn 96

ifconfig 10, 94

iflash 77

lsdev 168

maint 116, 131

no 95

pax 77

perspectives 3, 52

ping 97, 160

ping -P grid 97, 98

reboot 79

route 10

SDRGetObjects switch_responds 92

smit

annotator 91

delete_extadapter 47

delete_extnode 46

enter_extadapter 47, 90

enter_extnode 44, 89

list_extadapter 50

list_extnode 49

manage_extnode 48, 92

sphardware 52

splstdata 84

spmon 3

sysctl 301

xterm -e telnet 72

command line interface

see CLI

communications bus 28

Control Workstation

see CWS

crosspoint switch 13, 16, 27, 37, 38, 39

CWS 31, 43, 59, 61, 64, 65, 69, 74, 75, 76, 81,

84, 88, 99, 102, 165, 184, 193, 203, 204, 205, 224,

225

D

DAS 121, 123, 129, 130

data forwarding 19

datagrams 19

dedicated router 15

dependent node 3, 4, 5, 8, 41, 43, 44, 51, 60, 61,

64, 65, 69, 82, 101, 102

architecture 3

DIX Ethernet 144

DMA engine 29, 30, 31

DNS 41, 64, 65, 204

domain name service

see DNS

downtime 7

dual ring 125

dual-speed 105

dump profile 107, 108

dynamic routing 31, 33, 66, 96, 210

dynamic routing protocol 66

E

EGP 16, 23

Ethernet 6, 34, 39, 66

card 107, 109

hostname 81

hub 75, 76, 79, 102, 157

media card 161

twisted-pair 75, 76, 79, 102

extended node 92

extension node 3, 6, 45, 46, 47, 89

adapter 6

Exterior Gateway Protocol

see EGP

exterior protocols 20

external routes 22

external switch module 125

Contents

Main Page Page Page Contents Page Page Page Page Page Figures Page Tabl es Page Preface The Team That Wrote This Redbook Page Comments Welcome Page Page Page Chapter 1. Dependent Node dependent node 1.1 Dependent Node Architecture 1.2 Limitations of the Dependent Node Page Chapter 2. Router Node 2.1 Overview extension node extension node adapter 2.1.1 Motivation Page 2.1.2 Design Objectives SDR IP gateway router Internet Protocol Page Page 2.1.4 Routing without the GRF SP Switch . . . Internet/Intranet . . . Internet/Intranet LAN/WAN hop IP Switch Control Board ..... ........ ..... ...... ...... 2.1.6 Overview of Supported Routing Protocols ..... ..... ..... ..... ......... 2.1.7 Media Adapters At-a-Glance 2.1.8 Benefits of the GRF nonblocking crossbar 2.1.9 Price Comparison 2.2 GRF Software 2.2.1 IP Protocol Address Resolution Protocol 2.2.2 Supported Routing Protocols autonomous systems Exterior routing protocols Interior routing protocols interior gateway protocols routed RFC backbone inter-area routing intra-area routing mrouted Internet Group Management Protocol Connectionless Network Protocol InterDomain Routing Protocol Border Gateway Protocol 2.2.3 Filtering 2.2.4 System Management Simple Network Management Protocol 2.3 GRF Hardware 2.3.1 GRF Block Diagram Page 2.3.2 GRF Features Page Quick Branch Routing Technology communications bus (Combus) Page Rx Buffer Tx Buffer Rx Buffer CPU QBRT Tx Buffer CPU QBRT 2.3.3 IP Switch and Control Board ATE Page Page Page 2.3.4 Memory Guidelines for the IP Switch Control Board 2.3.5 Characteristics of GRF Media Cards 2.3.6 SP Switch Router Adapter Page 2.3.7 Media Card Performance 2.3.8 Other Media Cards 2.3.9 GRF Operating Environment 2.4 PSSP Enhancements 2.4.1 SDR Enhancements Page Page dependent standard Do not change this value for any reason Do not change this value for any reason 2.4.2 New Commands Page Page Page Page Page Page Page 2.4.3 Enhanced Commands Page Page Page Page Switch Responds Page 2.4.5 SP Extension Node SNMP Manager 2.4.6 Dependent Node MIB 2.4.7 Coexistence coexistence 2.4.8 Partitioning 2.5 Planning for the GRF Partition B Partition A Cross-partition communication through the SP Switch SDR Page 2.6 Planning for the Dependent Node 2.7 Conclusion Part 2. Scenarios Page Chapter 3. Installation and Configuration RS/6000 SP: Administration Guide Version 2 Release 4 GRF 400/1600 Getting Started RS/6000 SP: Administration Guide Version 2 Release 4, GRF Configuration Guide 1.4 3.1 Initial Configuration 3.2 Pre-Installation Assumptions GRF 400/1600 Getting Started V.14, Page 3.2.1 Order of Information Terminal Settings 9600 baud No parity Eight data bits One stop bit 3.3 Installing an SP Switch Router Adapter Card 3.3.1 Installation Overview GRF 400/ 1600 Getting Started 1.4, 3.3.2 Installing the PCMCIA Spinning Disk /dev/wd3a /var/log ufs rw 0 2 # PCMCIA slot A Page 3.4 Attaching SP Switch Router Cables 3.4.1 Ethernet Cable 3.4.2 SP Switch Cable 3.4.3 Procedure for Connecting Cards to the SP Switch 3.5 Configuration Required on the SP System 3.5.1 Determining the Switch Connection for a Dependent Nod e RS/6000 SP: Maintenance Information, Volume 1, Installation and Customer Engineer Operations RS/6000 SP: Administration Guide Version 2 Release 4, Frame n+3 Frame n+2 Installation and Configur ation 84 Figure 33. Node Numbering for an SP System 3.5.2 Procedure to Get the Jack Number 2. From the SP CWS, determine the switch_node_number by entering SDRGetObjects DependentNode node_number==n switch_node_number, where n # SDRGetObjects DependentNode node_numb er node_number 4 # # SDRGetObjects DependentNode node_numb er==4 switch_n ode_number switch_node_number 3 # 3.6 Multiple Frames for Multiple System Connections 3.7 Step-by-Step Media Card Configuration 3.7.1 Configuration Files and Their Uses GRF Reference Guide 1.4 RS/6000 SP: Administration Guide Version 2 Release 4, after 3.8 Step 1. Check SNMP in the SP Switch Router S ystem must not remove GRF Configuration Guide 1.4, 3.9 Put SNMP Changes into Effect 3.10 Step 2. Assign IP Addresses 3.10.1 Method 1: Use SP SNMP Manager - Recommended RS/6000 SP: Administration Guide Version 2 Release 4, Page Page Command: Use SDRGetObjects switch_responds to see the actual status of the SP Switch: Before the extended node can be successfully Eunfenced, more work has to be done. 3.10.2 Method 2: Edit /etc/grifconfig.conf - Optional Interface name Internet address Netmask Broadcast or destination address Argument field Default MTU values 3.10.3 Putting grifconfig.conf Additions into Effect 3.11 Step 3. Change Profile Settings GRF 400/1600 Configuration Guide 1.4, 3.12 Step 4. Run dev1config 3.13 Step 5. Reset SP Switch Router System to Install Files 3.13.1 Saving Configuration Files 3.14 Verify an SP Switch Router Adapter Card on the Router 3.14.1 Verify Media Card Operation Using ping 3.14.2 Check Media Card Status Using grcard 3.14.3 Reset Media Card Using grreset GRF Configuration Guide 1.4, , RS/6000 SP: Command Reference Version 2 Release 4, 3.14.4 Using grstat to Display GRF Statistics 3.15 Bringing the SP Switch Router Adapter Card Online with the SP RS/6000 SP: Installation and Migration Guide Version 2 Release 4, RS/6000 SP: Administration Guide Version 2 Release 4, RS/6000 SP: Diagnosis and Messages Guide Version 2 Release 4, 3.15.1 Checking Connectivity to the SP System RS/6000 SP Page Page Chapter 4. Configuration of IP-Forwarding Media Cards GRF Configuration Guide 1.4 4.1 Ethernet 10/100Base-T Configuration 4.1.1 Physical and Logical Interfaces Interface Name 4.1.2 Configuration File and Profile Overview GRF Configuration Guide 1.4, 4.1.3 Installing Configurations or Changes 4.1.4 Assign IP Addresses - grifconfig.conf 4.1.5 Specify Ethernet Card Parameters Set the Negotiation or Transfer Rate ports/ether 4.1.6 Some maint Commands for the Ethernet Media Cards 4.2 ATM OC-3c Configuration 4.2.1 Physical and Logical ATM Interfaces Virtual Circuits Virtual P a th s VCIs Permanent Virtual Circuits Switched Virtual Circuits Verifying an ATM Configuration 4.2.2 Installing Configurations or Changes 4.2.3 Configuration Files and Profiles 4.2.4 Assign IP Addresses - grifconfig.conf 4.2.5 Specify ATM Card Parameters GRF Configuration Guide 1.4, only 4.2.6 Configuring PVCs GRF Reference Guide 1.4, 4.2.7 Some maint Commands for the ATM OC-3c Media Card Page 118 4.2.8 Using grrt to Display the Route Table 4.2.9 Using grstat to Display GRF Statistics 4.3 ATM OC-12c Configuration 4.3.1 Physical and Logical ATM Interfaces Virtual Circuits Virtual P a th s 4.3.2 Installing Configurations or Changes 4.3.3 Configuration Files and Profiles GRF Configuration Guide 1.4, 4.4 FDDI Configuration Single Attach (SAS) FDDI Dual Attach (DAS) FDDI Configuring SAS versus DAS Page Optical bypass Dual homing 4.4.1 Separate Networks versus Bridging 4.4.2 Naming the FDDI Interfaces Configuration of IP-Forwarding Media Cards 127 grifconfig.conf Figure 46. Assigning Numbers to FDDI Interfaces 4.4.3 Physical Interface Numbers 4 single attach 2 dual attach 1 dual 2 single attach 2 single 1 dual attach grifconfig.conf 4.4.4 GRF Interface Name 4.4.5 Configuration Files and Profiles 4.4.6 Assign IP Addresses - grifconfig.conf 4.4.7 Specify FDDI Card Parameters GRF Configuration Guide 1.4, 4.4.8 Installing Configurations or Changes 4.4.9 Some maint Commands for the FDDI Media Card 132 4.4.10 Using grrt to Display the Route Table 4.4.11 Using grstat to Display GRF Statistics 4.5 HIPPI Configuration 4.5.1 Introduction to HIPPI Page 00 Source Routing Page It will reject the connection if the W bit is set on. unless HIPPI does not bridge! GRF Configuration Guide 1.4, 4.5.2 HIPPI Configuration Options 4.5.3 Physical and Logical Interfaces Interface Name 4.5.4 Configuration Files and Profiles 4.5.5 Installing Configurations or Changes 4.5.6 Some maint Commands for the HIPPI Media Card 4.6 Configuring Bridging 4.6.1 GRF Bridging Implementation 4.6.2 Simultaneous Routing and Bridging other than 4.6.3 Configuration Options 4.6.4 Interoperability FDDI Spanning tree ATM OC - 3c Ethernet 4.6.8 Spamming initial 4.6.9 Bridging Components 4.6.10 Management Tools GRF Configuration Guide 1.4, Page 4.6.11 Configuration File and Profile Overview GRF Configuration Guide 1.4, required Page required all specific Configuration in gratm.conf Page Page only 4.6.12 Bridging ATM 4.6.13 Bridging FDDI 4.6.14 Bridging Ethernet Page Chapter 5. Single RS/6000 SP and Single SP Switch Router 5.1 Single SP Partition and Single SP Switch Router Adapter Card 5.1.1 SP Switch - Ethernet Connection SP Switch SP node F50 Page Page Page 5.1.2 SP Switch - FDDI Connection SP2 Page Page Page 5.1.3 SP Switch - ATM Connection SP21 Page Page Page 172 On the chosen nodes in SP21, ping the ATM interface of the F50: Page 5.1.4 SP Switch - FDDI Connection (Distinct FDDI Networks) SP21 Page Page Page Page SP21 bridge group Page Note: , does not work GRF Configuration Guide 1.4, Page Page 5.1.5 SP Switch - FDDI Connection in an ADSM Environment ADSM c lients SP21 SP2 Page 5.2 Single SP Partition and Multiple SP Switch Router Adapter Cards 5.2.1 Configuration of a Dual SP Switch Router Connection Note: This configuration does not work. Page 5.2.2 Complex Configuration Page Page Page Page Figure 63 on page 196 shows the IP traffic flow when issuing the node 1 node 6 node 1 node 10 5.2.3 Recovery Procedure for an SP Switch Adapter Card Failure 5.3 Multiple SP Partition and Multiple SP Switch Router Ada pter Cards and Note: All other subnetting does not work. Page Page Page Page Chapter 6. Multiple RS/6000 SPs and One SP Switch Router 6.1 RS/6000 SP Switch - RS/6000 SP Switch Connection both twice Note: Page Page 6.2 Sharing Network Resources Page Chapter 7. Multiple RS/6000 SPs and Multiple GRFs 7.1 ATM OC-3c Backbone Connection 7.1.1 ATM OC-3c Backbone - Using One Port Page Page Page Page 7.1.2 ATM OC-3c Backbone - Using Two Ports Page Page Page Page Page Page 7.2 ATM OC-12c Backbone - One Port Page Page Page Page 7.3 HIPPI Backbon e Co nnec tio n Page Page Page Page Page Appendix A. Laboratory Hardware and Software Configuration A.1 Node and Control Workstation Configuration 234 Page 236 Page 238 Laboratory Hardware and Software Configuration 239 Table 32. Software Levels on CWS and All Nodes Part 1 of 14 240 Table 33. Software Levels on CWS and All Nodes Part 2 of 14 Laboratory Hardware and Software Configuration 241 Table 34. Software Levels on CWS and All Nodes Part 3 of 14 242 Table 35. Software Levels on CWS and All Nodes Part 4 of 14 Laboratory Hardware and Software Configuration 243 Table 36. Software Levels on CWS and All Nodes Part 5 of 14 244 Table 37. Software Levels on CWS and All Nodes Part 6 of 14 Laboratory Hardware and Software Configuration 245 Table 38. Software Levels on CWS and All Nodes Part 7 of 14 246 Table 39. Software Levels on CWS and All Nodes Part 8 of 14 Laboratory Hardware and Software Configuration 247 Table 40. Software Levels on CWS and All Nodes Part 9 of 14 248 Table 41. Software Levels on CWS and All Nodes Part 10 of14 Laboratory Hardware and Software Configuration 249 Table 42. Software Levels on CWS and All Nodes Part 11 of 14 250 Table 43. Software Levels on CWS and All Nodes Part 12 of 14 Laboratory Hardware and Software Configuration 251 Table 44. Software Levels on CWS and All Nodes Part 13 of 14 252 Table 45. Software Levels on CWS and All Nodes Part 14 of 14 Laboratory Hardware and Software Configuration 253 Table 46. Network Options of CWS and All Nodes Part 1 of 3 254 Table 47. Network Options of CWS and All Nodes Part 2 of 3 A.2 SP Switch Pool Size Settings A.3 7025-F50 Configuration Laboratory Hardware and Software Configuration 257 Table 50. Network Options of 7025-F50 Part 2 of 3 A.4 SP IP Switch Router Configuration Page Page Appendix B. GRF Configuration Files B.1 /root/.profile 262 GRF Configuration Files 263 B.2 /etc/Release 264 B.3 /etc/bridged.conf GRF Configuration Files 265 266 GRF Configuration Files 267 B.4 /etc/fstab B.5 /etc/grarp.conf This file holds ARP information for distant interfaces that do not support inverse ARP. 268 B.6 /etc/gratm.conf In this file, the parameters for ATM (OC-3c or OC-12c) are set. GRF Configuration Files 269 270 GRF Configuration Files 271 272 GRF Configuration Files 273 274 B.7 /etc/grclean.conf GRF Configuration Files 275 B.8 /etc/grclean.logs.conf 276 GRF Configuration Files 277 B.9 /etc/grdev1.conf This file normally gets updated automatically by the SNMP daemon running on the CWS. 278 GRF Configuration Files 279 280 GRF Configuration Files 281 282 B.10 /etc/grifconfig.conf GRF Configuration Files 283 284 B.11 /etc/grlamap.conf GRF Configuration Files 285 B.12 /etc/grroute.conf 286 B.13 /etc/hosts The file /etc/hosts contains the correlation between IP addresses and hostnames. B.14 /etc/inetd.conf This file is included here just for curiosity in case you are interested in why ftp to the GRF is not possible. B.15 /etc/motd B.16 /etc/rc.local Use this file for local modifications that are to be carried out during boot. 288 B.17 /etc/snmpd.conf GRF Configuration Files 289 290 GRF Configuration Files 291 B.18 /etc/syslog.conf 292 B.19 /etc/ttys GRF Configuration Files 293 294 Appendix C. Hardware and Software Information C.1 The Front Panel of the SP Switch Router Adapter Card - Operational 296 C.2 SP Switch Router Adapter Media Card LEDs C.3 SP Switch Router Adapter Media Card - Bootup Table 54 shows the settings for the Switch Router Adapter Media Card LEDs during bootup. 298 C.4 Connectors and Receptacles for Different Media Figure 73 on page 299 is taken from the Redbook C.5 Chip Interconnection on the TBS Board RS/6000 SP: Problem Determination Guide , SG24-4778-00, Chapter 4.2 "Reviewing Switch Boards". C.6 Updating Router Software only the IBM version of the code will work on the SP Switch Router Page 302 Page Page Appendix D. Special Notices Page Page Page Appendix E. Related Publications E.1 International Technical Support Organization Publications RS/6000 SP: Installation and Migration Guide Version 2 Release 4 RS/6000 SP: Administration Guide Version 2 Release 4, RS/6000 SP: Problem Determination Guide, Page How to Get ITSO Redbooks How IBM Employees Can Get ITSO Redbooks 312 How Customers Can Get ITSO Redbooks IBM Redbook Order Form Page List of Abbreviations 316 Index Symbols Numerics A B C D E F G H I J K L M N Q R S T U V W ITSO Redbook Evaluation