IBM 990 2.1.5 Connectivity

Chapter 2. System structure and design 31

Draft Document for Review April 7, 2004 6:15 pm 6947ch02.fm

Figure 2-6 Three-book system ring structure

Figure 2-7 Four-book system ring structure

2.1.5 Connectivity

STI connections to I/O cages and ICB-4 links are driven from the Memory Bus Adapters

(MBAs) that are located on a separate card in the book. Figure2-8 shows the location of the

STI connectors and the MBA card.

STI

MBA

STI

STI STI

STI

MBA

STI

STI STI

4 x 2 GB/S 4 x 2 GB/S

STI

MBA

STI

STI STI

4 x 2 GB/S

MBA Card

Ring Structure

PU PUPUPUPU PU PU PU PU PU PU PU

Jumper Book

STI

MBA

STI

STI STI

STI

MBA

STI

STI STI

4 x 2 GB/S 4 x 2 GB/S

STI

MBA

STI

STI STI

4 x 2 GB/S

MBA Card

PU PUPUPUPU PU PU PU PU PU PU PU

STI

MBA

STI

STI STI

STI

MBA

STI

STI STI

4 x 2 GB/S 4 x 2 GB/S

STI

MBA

STI

STI STI

4 x 2 GB/S

PU PUPUPUPU PU PU PU PU PU PU PU

Memory Up to 64GB

Level 2 Cache 32MB

MCM

MCM

Memory Up to 64GB

Memory Up to 64GB

Level 2 Cache 32MB

Level 2 Cache 32MB

MCM

MCM

MCM

MCM

Ring Structure

STI

MBA

STI

STI STI

STI

MBA

STI

STI STI

4 x 2.5 GB/S 4 x 2.5 GB/S

STI

MBA

STI

STI STI

4 x 2.5 GB/S

MBA Card

STI

MBA

STI

STI STI

STI

MBA

STI

STI STI

4 x 2.5 GB/S 4 x 2.5 GB/S

STI

MBA

STI

STI STI

4 x 2.5 GB/S

MBA Card

STI

MBA

STI

STI STI

STI

MBA

STI

STI STI

4 x 2.5 GB/S 4 x 2.5 GB/S

STI

MBA

STI

STI STI

4 x 2.5 GB/S

MBA Card

STI

MBA

STI

STI STI

STI

MBA

STI

STI STI

4 x 2.5 GB/S 4 x 2.5 GB/S

STI

MBA

STI

STI STI

4 x 2.5 GB/S

MBA Card

Memory Up to 64GB

Level 2 Cache 32MB

PU PUPUPU PU PU PU PU PU PU PU PU

MCM

MCM

PU PUPUPU PU PU PU PU PUPU PU PUPU PUPUPU PU PU PU PUPU PU PU PU PU PUPUPU PU PU PU PU PUPU PU PU

Memory Up to 64GB

Level 2 Cache 32MB

MCM

MCM

Memory Up to 64GB

Level 2 Cache 32MB

MCM

MCM

Level 2 Cache 32MB

Memory Up to 64GB

MCM

MCM

Contents

Main Page Page Page Page Page Contents Page Page Page Notices 6947spec.fm x Trademarks Preface The team that wrote this redbook Become a published author Comments welcome Page Page 1.1 Introduction 1.2 z990 models General rules Model upgrade paths Model downgrades Concurrent Processor Unit (PU) conversions 1.3 System functions and features 1.3.1 Processor MCM technology 1.3.2 Memory 1.3.3 Self-Timed Interconnect (STI) 1.3.4 Channel Subsystem (CSS) 1.3.5 Physical Channel IDs (PCHIDs) and CHPID Mapping Tool 1.3.6 Spanned channels 1.3.7 I/O connectivity I/O cage Z frame Cargo A frame Up to 1024 ESCON channels Up to 120 FICON Express channels CEC 1st I/O 3rd I/O FICON CTC function FICON Cascaded Directors OSA-Express Gigabit Ethernet 1000BASE-T Ethernet OSA-Express Integrated Console Controller (OSA-ICC): Checksum Offload for Linux and z/OS when QDIO mode: 1.3.8 Cryptographic CP Assist for cryptographic function PCI Cryptographic Accelerator feature (PCICA) PCI X-Cryptographic Coprocessor (PCIXCC) feature 1.3.9 Parallel Sysplex support ISC-3 ICB-2 (Integrated Cluster Bus 2) ICB-3 (Integrated Cluster Bus 3) ICB-4 (Integrated Cluster Bus 4) Internal Coupling (IC) System-Managed CF Structure Duplexing 1.3.10 Intelligent Resource Director (IRD) Channel Subsystem Priority Queuing Dynamic Channel Path Management LPAR CPU Management 1.3.11 Hardware consoles 1.3.12 Concurrent upgrades Capacity Upgrade on Demand (CUoD) Customer Initiated Upgrade (CIU) On/Off Capacity Upgrade on Demand (On/Off CoD) Capacity BackUp (CBU) 1.3.13 Performance 1.3.14 Reliability, Availability, Serviceability (RAS) 1.3.15 Software Traditional database/transaction workloads UNIX System Services zSeries File System (zFS) zSeries Plat for m Linux on zSeries (and Linux for S/390) 1.3.16 Software support Compatibility and exploitation Compatibility support Exploitation support OS/390 & z/OS Software pricing 1.3.17 Summary Page 2.1 System structure 2.1.1 Book concept STI Slots books MCM Memory cards Power Cooling Hybrid cooling system 2.1.2 Models 2.1.3 Memory Memory Sizes Page Memory sparing Memory upgrades Book replacement and memory 2.1.4 Ring Topology Page Chapter 2. System structure and design 31 Draft Document for Review April 7, 2004 6:15 pm 6947ch02.fm PU PUPUPU PU PU PU PUPU PU PU PU PU PUPUPU PU PU PU PU PUPU PU PU PU PUPUPU PU PU PU PU PU PU PU PU 2.1.5 Connectivity MBA Card PU PUPUPU PU PU PU PU PUPU PU PU Book upgrade STI connectors Book front view Book replacement and connectivity 2.1.6 Frames and cages A frame Z frame I/O cages 2.1.7 The MCM Page The dual-core PU chips share the path to the SC chip (L2 control) and the clock chip (CLK). 2.1.9 Summary Table2-2 summarizes all aspects of the z990 system structure. 2.2 System design 2.2.1 Design highlights 2.2.2 Book design Page Dual External Time Reference 2.2.3 Processor Unit design Superscalar processor Asymmetric mirroring for error detection Compression Unit on a chip CP Assist for Cryptographic Function I-Unit E-Unit I-Unit B-Unit E-Unit Processor Branch History Table (BHT) With BHT: Without BHT: IEEE Floating Point Translation Lookaside Buffer Instruction fetching and instruction decode Instruction fetching Instruction decoding 2.2.4 Processor unit functions Central Processors Integrated Facilities for Linux cannot Internal Coupling Facilities Dynamic ICF Expansion Dynamic Coupling Facility Dispatching zSeries Application Assist Processors zAAPs and LPAR definitions Purpose of a zAAP Software support System Assist Processors Optional additional orderable SAPs Optionally assignable SAPs Reserved processors Processor unit characterization Transparent CP, IFL, ICF, zAAP, and SAP sparing Application preservation Dynamic SAP sparing and reassignment Sparing rules 2.2.5 Memory design Memory allocation Central storage (CS) Expanded storage (ES) LPAR single storage pool Hardware System Area (HSA) 2.2.6 Modes of operation Logical Partitioning overview Page Reserved Logically Partitioned Mode Dynamic Add/Delete of a logical partition name 2.2.7 Model configurations Upgrades Page PU conversions Capacity Backup (CBU) Software model MSU values Hardware Management Console and Support Elements Dual External Time Reference 2.2.8 Storage operations ESA/390 mode z/Architecture mode will not ESA/390 architecture mode ESA/390 TPF mode Coupling Facility mode Linux Only mode 2.2.9 Reserved storage 2.2.10 LPAR storage granularity 2.2.11 LPAR Dynamic Storage Reconfiguration (DSR) 2.2.12 I/O subsystem 2.2.13 Channel Subsystem Logical Channel Subsystem (LCSS) Physical Channel ID (PCHID) Page 3.1 Overview High bandwidth Wide connectivity Cryptographic functions Concurrent I/O upgrades 3.2 I/O cages Z Frame A Frame CEC Cage 1st I/O Cage (standard) 2nd I/O Cage (optional) Board 76 3.2.1 Self-Timed Interconnect (STI) 3.2.2 STIs and I/O cage connections eSTI-M card I/O Cage CEC Cage STI-2 Extender card STI-3 Extender card 3.2.3 Balancing I/O connections STI links balancing across books and MBAs I/O Cage 1 I/O Cage 2 STIs I/O Cage 1 I/O Cage 2 STI Rebalance feature (Feature Code 2400) STIs STIs I/O port balancing across MBAs and books 3.3 I/O and cryptographic feature cards 3.3.1 I/O feature cards I/O feature cards no longer supported 3.3.2 Cryptographic feature cards Cryptographic feature card no longer supported 3.3.3 Physical Channel IDs (PCHIDs) ... PCHIDs I/O Cage 1 - Front Page Page z990 CHIPD Mapping Tool (CMT) 3.4 Connectivity 3.4.1 I/O and cryptographic features support and configuration rules 6947ch03.fm 90 Spanned and Shared channels I/O features cables and connectors 3.4.2 ESCON channel z990 16-port ESCON feature ESCON channel port enablement feature 16-port ESCON channel sparing Fiber Quick Connect (FQC) for ESCON Quick Connect 3.4.3 FICON channel FICON Express features FICON Express LX feature FICON Express SX feature FICON channel in Fibre Channel Protocol (FCP) mode z990 adapter interruptions enhancement for FCP z990 FCP SCSI IPL feature enabler (FC 9904) z990 FCP concurrent patch 3.4.4 OSA-Express adapter OSA-Express GbE LX (feature code 1364) OSA-Express GbE SX (feature code 1365) OSA-Express GbE LX (feature code 2364, upgrade only) an upgrade OSA-Express GbE SX (feature code 2365, upgrade only) an upgrade OSA-Express 1000BASE-T Ethernet (feature code 1366) OSA-Express Fast Ethernet (feature code 2366, upgrade only) forward on an upgrade OSA-Express Token Ring (feature code 2367) Checksum offload for IPv4 packets when in QDIO mode z990 adapter interruptions enhancement for QDIO HiperSockets function 3.4.5 Coupling Facility links z990 coupling link features ISC-3 link RPQ 8P2197: Extended distance option ICB-4 link ICB-3 link ICB-2 link IC links 3.4.6 External Time Reference (ETR) feature 3.4.7 Cryptographic features PCIX Cryptographic Coprocessor (PCIXCC) feature PCI Cryptographic Accelerator (PCICA) feature Page Page 4.1 Multiple Logical Channel Subsystem (LCSS) 4.1.1 Logical Channel Subsystem structure Logical Channel Subsystem (LCSS) Multiple Image Facility (MIF) Logical partition number Logical partition identifier MIF Image ID (MIFID) Logical partition name Dynamic Addition or Deletion of a logical partition name 4.1.2 Physical Channel ID (PCHID) 4.1.3 Channel spanning 4.2 LCSS configuration management IBM Configurator for e-business (e-Config) Hardware Configuration Dialog (HCD) IBM z990 CHPID Mapping Tool (CMT) 4.2.1 z990 configuration management 4.3 LCSS-related numbers HCD Reports IODF IOCP Deck Page Page 5.1 Cryptographic function support 5.1.1 Cryptographic Synchronous functions 5.1.2 Cryptographic Asynchronous functions Page 5.2 z990 Cryptographic processors 5.2.1 CP Assist for Cryptographic Function (CPACF) 5.2.2 PCIX Cryptographic Coprocessor (PCIXCC) 5.2.3 PCI Cryptographic Accelerator (PCICA) feature 5.3 Cryptographic hardware features 5.3.1 PCIX Cryptographic Coprocessor feature PCIXCC Feature 5.3.2 The PCICA feature 5.3.3 Configuration rules 5.3.4 z990 cryptographic feature codes 5.3.5 TKE workstation feature 5.4 Cryptographic features comparison 5.5 Software requirements Page Page Page Page 6.1 Operating system support 6.2 z/OS software support Exploitation 6.2.1 Compatibility support for z/OS z990 Compatibility for selected OS/390, and z/OS releases Compatibility support requirements 6947ch06.fm ICF CF Compatibility support restrictions G5, G6, z800, or z900 CSS 0 ICF ICF 6.2.2 Exploitation support for z/OS z/OS V1.4 z990 Exploitation Support feature z/OS V1.5 Support Planned z/OS V1.6 support Dynamic addition and deletion of a logical partition name 24 processors within a single logical partition zSeries Application Assist Processor (zAAP) Page Option 1: Java dispatching by priority (honor_priority=yes) Option 2: Java discretionary crossover (crossover=yes) Option 3: No Java crossover (crossover=no) 6.2.3 HCD support 6.2.4 Automation changes 6.2.5 SMF support 6.2.6 RMF support 6.2.7 ICKDSF requirements 6.2.8 ICSF support 6.2.9 Additional exploitation support considerations SMF Standalone dump Automati on Page EREP Extended Channel Measurement Block Dynamic activates for hardware changes Dynamic CHPID management Greater than 15 logical partitions 6.3 z/VM software support 6.4 z/VSE and VSE/ESA software support 6.5 TPF software support 6.6 Linux software support 6.7 Summary of software requirements 6.7.1 Summary of z/OS and OS/390 software requirements 6947ch06.fm 148 6.7.2 Summary of z/VM, z/VSE, VSE/ESA, TPF, and Linux Software Requirements Page 6.8 Workload License Charges 6.9 Concurrent upgrades considerations Processor Identification Channel to Channel links Page 7.1 Parallel Sysplex IBM z990 7.1.1 Parallel Sysplex described IBM z900 Model 2xx IBM z990 Continuous (application) availability High capacity Dynamic workload balancing Systems management Resource sharing Single system image 7.1.2 Parallel Sysplex summary 7.2 Sysplex and Coupling Facility considerations 7.2.1 Sysplex configurations and Sysplex Timer considerations Message Time Ordering External Reference ID Page Page 7.2.2 Coupling Facility and CFCC considerations 7.2.3 CFCC enhanced patch apply Page 7.2.4 Coupling Facility link connectivity Peer mode links 7.2.5 Coupling Facility Resource Manager (CFRM) policy considerations 7.2.6 ICF processor assignments Page 7.2.7 Dynamic CF dispatching and dynamic ICF expansion 7.3 System-managed CF structure duplexing 7.3.1 Benefits z/OS CF CF HMC 7.3.2 CF Structure Duplexing 7.3.3 Configuration planning z/OS ICF ICF z/OS z800/z900/z990/z890/G5/G6 z800/z900/z990/z890/G5/G6 7.4 Geographically Dispersed Parallel Sysplex 7.4.1 GDPS/PPRC Site 1 Site 2 GDPS/PPRC HyperSwap Remote copy (PPRC) Planned HyperSwap Unplanned HyperSwap P S application UCB PPRC application UCB GDPS/PPRC Management for Open Systems LUNs GDPS/PPRC over Fiber Channel links GDPS FlashCopy V2 support Business Continuity for Linux guests GDPS/PPRC Cross-site extended distance for Parallel Sysplex 7.4.2 GDPS/XRC Site 1 Site 2 CPC CPC 100 km Route A Route B 7.4.3 GDPS and Capacity Backup (CBU) Site 1 Site 2 7.5 Intelligent Resource Director IRD overview 7.5.1 LPAR CPU management Value of CPU management 7.5.2 Dynamic Channel Path Management Value of Dynamic Channel Path Management 7.5.3 Channel Subsystem Priority Queueing Value of Channel Subsystem Priority Queueing 7.5.4 WLM and Channel Subsystem priority 7.5.5 Special considerations and restrictions Unique LPAR cluster names Disabling Dynamic Channel Path Management Automatic I/O interface reset System automation - I/O operations 7.5.6 References Page Page 8.1 Concurrent upgrades Licensed Internal Code (LIC)-based upgrades Concurrent hardware installation upgrades Concurrent PU conversions Model upgrades Planned upgrades Unplanned upgrades Capacity upgrade functions 8.2 Capacity Upgrade on Demand (CUoD) concurrent and permanent capacity growth. Page CUoD for processors CUoD + 5 CPs (+ 1 Book) CUoD + 1 CP + 2 IFLs CUoD for memory Page Chapter 8. Capacity upgrades 193 The new amount of installed memory cannot cause the storage granularity or increment to Figure 8-3 CUoD for I/O LIC-CC upgrade example CUoD for I/O CUoD for I/O can add, concurrently, more I/O ports to a z990 server by either: Enabling additional ports on the already installed I/O cards via LIC-CC. Installing additional I/O cards on an already installed I/O cages slots. Note: I/O cages cannot be installed concurrently. 8.3 Customer Initiated Upgrade (CIU) CIU Registration and Agreed Contract for CIU Internet Internet Order and fulfillment process Page Activation 8.4 On/Off Capacity on Demand (On/Off CoD) Page Initiation Activation/Deactivation Termi nati on Upgrade Capability during On/Off CoD Repair capability during On/Off CoD Monitoring 8.5 Capacity BackUp (CBU) Page Activation/deactivation of CBU CBU activation Image upgrades CBU deactivation CBU testing Capacity BackUp operation example Automatic CBU enablement for GDPS Workload Transfer 8.6 Nondisruptive upgrades Processors Memory I/O PCI Cryptographic coprocessors 8.6.1 Upgrade scenarios after Shared logical partitions upgrade Book 0 Book 0 Book 1 LP1 LP2 2084-A08 Model 307 Dedicated and shared logical partitions upgrade LP3 Figure 8-12 Dedicated and shared logical partitions upgrade example Book 0 Book 1 LP1 LP2 Shared partitions and zAAP upgrade Chapter 8. Capacity upgrades 213 Figure 8-13 Shared logical partitions and zAAP upgrade example LP1 has eight shared (SHR) logical CPs, two reserved CPs and four reserved zAAPs LP2 has two shared (SHR) logical CPs, one reserved CP and three reserved zAAPs Dedicated, shared partitions and IFL upgrade Book 0 Book 1 LP1 LP2 2084-B16 Model 309 + 3 zAAPs Book 0 Book 1 Dedicated, shared partitions and ICF upgrade Book 0 Book 1 LP3 (redefined with IFLs) LP3 (not activated) LP1 LP2 8.6.2 Planning for nondisruptive upgrades Reasons for disruptive upgrades Book 0 Book 1 LP3 LP1 LP2 Recommendations to avoid disruptive upgrades partitions Considerations when installing additional books 8.7 Capacity planning considerations 8.7.1 Balanced system design Additional performance improvements for e-business Multi-book structure 8.7.2 Superscalar processors ... PU PU ... PU PU Book 0 Book 1 8.7.3 Integrated hardware and system assists zSeries Application Assist Processors (zAAPs) Cryptographic function on every Processor Unit (PU) Secure encrypted transactions with higher performance Continued cryptographic support for the e-business environment 8.8 Capacity measurements 8.8.1 Large Systems Performance Reference (LSPR) Internal Throughput Rate (ITR) and ITR Ratio (ITRR) LSPR workloads prior to z990 LSPR workloads for z990 Page Page New predefined z/OS workload mixes Page Page 9.1 Introduction 9.1.1 Power and cooling requirements 9.1.2 Power consumption 9.1.3 Internal Battery Feature 9.1.4 Emergency power-off 9.2 Weights 6947ch09.fm 232 9.3 Dimensions With service clearance, 5.45 square meters (58.69 square feet) are needed. Configuration Weight in kg (lb) without IBF Weight in kg (lb) with IBF Frames Width mm (in) Depth mm (in) Height mm (in) A (HMC) Introduction LAN LAN zSeries z990 Hardware Management Console LAN HMC Notes on wiring with multiple adapters: Token ring only wiring scenario Token-Ring LAN Additional token ring only wiring scenario Ethernet only - one-path wiring scenario Additional connections to the Ethernet LAN Ethernet only - two-path wiring scenario Additional connections to the Ethernet LAN Token ring and Ethernet wiring scenario Additional connections to the Token Ring LAN Remote operations Support Element z990 HMC enhancements z990 HMC Integrated 3270 Console z990 HMC Integrated ASCII Console support Optional strict password rules supported Enhanced logging facilities Increased Console tasks performed log Customizable console data mirroring SNA Operations Management for Operations Automation Page B Fiber optic cabling services from IBM Summary Page Related publications IBM Redbooks Other publications Online resources How to get IBM Redbooks Glossary 254 Page 256 initial program load (IPL). LAN. input/output (I/O). input/output configuration data set (IOCDS). local area network (LAN). Logical Channel Subsystem (LCSS). A defined subset logical control unit (LCU) logical partition (LPAR). logical switch number (LSN). 258 original equipment manufacturers information (OEMI). parallel channel. path group path. Small Computer System Interface (SCSI). spanning subchannel. subsystem. SWCH. Page Conditional Text Settings (ONLY!) to the book files. IBM eServer zSeries 990 Technical Guide IBM eServer zSeries 990 Technical Guide IBM eServer zSeries 990 Technical Guide Page Index Numerics A B C E F G H I M N O P 6947IX.fm Q R S T U Page INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION IBM eServer zSeries 990 Technical Guide Back cover