Contents
IBM TotalStorage DS8000 Series
Page
IBM TotalStorage DS8000 Series Concepts and Architecture
First Edition April
Contents
Page
Benefits of virtualization
Vi DS8000 Series Concepts and Architecture
Summary
Viii DS8000 Series Concepts and Architecture
Page
Page
Index
Xii DS8000 Series Concepts and Architecture
Copyright License
Trademarks
Dfsort
Preface
Team that wrote this redbook
Xvi DS8000 Series Concepts and Architecture
Preface
Xviii DS8000 Series Concepts and Architecture
Become a published author
Comments welcome
Xx DS8000 Series Concepts and Architecture
Part 1 Introduction
DS8000 Series Concepts and Architecture
Introduction to the DS8000 series
Infrastructure Simplification
DS8000, a member of the TotalStorage DS family
Overview of the DS8000 series
Business Continuity
DS8000 Base frame
Hardware overview
Storage Hardware Management Console S-HMC for the DS8000
Storage capacity
Storage system logical partitions LPARs
IBM Standby Capacity on Demand offering for the DS8000
Supported environments
Resiliency Family for Business Continuity
IBM TotalStorage FlashCopy
IBM TotalStorage Metro Mirror Synchronous Pprc
Interoperability
Service and setup
Three-site solution
IBM TotalStorage Global Mirror Asynchronous Pprc
Positioning
Common set of functions
DS8000 compared to ESS
DS8000 compared to DS6000
IBM TotalStorage DS Command-Line Interface DS CLI
Common management functions
IBM TotalStorage DS Storage Manager
Scalability and configuration flexibility
Future directions of storage system LPARs
DS Open application programming interface
IBM TotalStorage Multipath Subsystem Device Driver SDD
Sequential Prefetching in Adaptive Replacement Cache Sarc
Performance
Performance for zSeries
Summary
16 DS8000 Series Concepts and Architecture
Part 2 Architecture
18 DS8000 Series Concepts and Architecture
Components
Frames
Base frame
Expansion frame
Rack operator panel
Architecture
Rack operator panel
SAN fabric
Server-based SMP design
Cache management
Components
Processor complex
Randomseq
Components
Rear view
Service processor and Spcn
Front view
Processor memory
RIO-G
2 I/O enclosures
Disk subsystem
Rear view
Device adapters
Disk enclosures
DS8000 device adapter
Non-switched FC-AL drawbacks
DS8000 disk enclosure
Switched FC-AL advantages
11 DS8000 disk enclosure
DS8000 switched FC-AL implementation
12 Disk enclosure switched connections
Expansion
Arrays and spares
14 DS8000 switched loop layout
Arrays across loops
Host adapters
AAL benefits
DDMs
Escon distances
Ficon and Fibre Channel protocol host adapters
Control units and logical paths
Remote Mirror and Copy with Escon
Fibre Channel supported servers
Power and cooling
Rack Power Control cards RPC
Fibre Channel distances
Disk enclosure power and cooling
Primary power supplies
Processor and I/O enclosure power supplies
Battery backup assemblies
Ethernet switches
42 DS8000 Series Concepts and Architecture
Storage system LPARs Logical partitions
Partitioning concepts
Introduction to logical partitioning
Virtualization Engine technology
Partitions
Building block
Physical partitioning Ppar
Logical partitioning Lpar
Micro-Partitioning
Software and hardware fault isolation
Dynamic logical partitioning
Virtual I/O
Production and test environments
Why Logically Partition?
Server consolidation
Application isolation
DS8000 and Lpar
Lpar and storage facility images
Increased flexibility of resource allocation
2 DS8300 Lpar implementation
DS8300 Model 9A2 Lpar and storage facility image
Storage facility image hardware components
DS8300 Lpar resource allocation
Storage Facility Image Processor complex
Processor and memory allocations
4 DS8300 Model 9A2 configuration options
RIO-G interconnect separation
DS8300 example configuration
Model conversion
Lpar security through Power Hypervisor Phyp
Model conversions regarding Lpar functionality
Lpar and Copy Services
Lpar protection Power Hypervisor
Lpar benefits
FlashCopy
Remote mirroring
Storage system LPARs Logical partitions
Addressing capabilities with storage facility images
Example of storage facility images in the DS8300
DS8300 addressing capabilities
60 DS8000 Series Concepts and Architecture
RAS
Storage unit
Naming
Storage complex
Storage facility image
Processor complex RAS
Processor complex
Reliability, availability, and serviceability
Permanent monitoring
Memory reliability, fault tolerance, and integrity
Self-healing
+1 redundancy
Hypervisor Storage image independence
Fault masking
Concurrent Maintenance
Resource deallocation
2 I/O enclosure
Server RAS
RIO-G a self-healing interconnect
Metadata checks
Server failover and failback
Data flow
Failover
Normal data flow
NVS recovery after complete power loss
Failback
Host connection availability
Single or multiple path
Enclosure
SAN/FICON/ESCON switches
Multi-pathing software
Open systems host connection
ZSeries host connection
Switched disk connections
Disk path redundancy
RAID-5 theory
RAID-5 overview
RAID-10 overview
RAID-5 implementation in the DS8000
Spare creation
RAID-10 theory
RAID-10 implementation in the DS8000
Predictive Failure Analysis PFA
Floating spares
Hot plugable DDMs
Disk scrubbing
Battery backup units
Rack power control card RPC
Rack cooling fans
Power control of the DS8000
Building power loss
Power fluctuation protection
Emergency power off EPO
Microcode updates
Installation process
Management console
HMC considerations
Different code versions across storage images
Virtualization concepts
Virtualization definition
Storage system virtualization
Abstraction layers for disk virtualization
Storage Facility virtualization
Array sites
Physical layer as the base for virtualization
Arrays
Array site
Ranks
Creation of an array
Extent pools
Forming an FB rank with 1 GB extents
Server0
Logical volumes
CKD volumes
Fixed Block LUNs
Volume with
Allocation and deletion of LUNs/CKD volumes
ISeries LUNs
Logical subsystems LSS
DB2
Volume access
Address groups
Volume group
Host attachment
11 Host attachments and volume groups
Summary of the virtualization hierarchy
Placement of data
12 Virtualization hierarchy
13 Optimal placement of data
Benefits of virtualization
101
102 DS8000 Series Concepts and Architecture
IBM TotalStorage DS8000 model overview and scalability
103
DS8000 highlights
Model naming conventions
2 DS8100 Model
Maximum configuration for the Model
3 DS8300 Models 922 and 9A2
IBM TotalStorage DS8000 model overview and scalability
Model comparison
Large and scalable capacity
Designed for scalability
Scalability for capacity
DS8000 model comparison
Future plan
Scalability for performance
Adding DDMs
Comparison of models for capacity
Way I/O controllers
Linear-scalable architecture
RIO-G
Benefit of the DS8000 for scalability
Model upgrades
Model conversions
114 DS8000 Series Concepts and Architecture
Copy Services
115
Introduction to Copy Services
Copy Services functions
Point-in-Time Copy FlashCopy
Read from the source volume
Read from the target volume
Write to the source volume
FlashCopy options
Point-in-Time Copy function authorization
Refresh target volume also known as Incremental FlashCopy
No background copy option
Data Set FlashCopy
Incremental FlashCopy
Multiple Relationship FlashCopy
Data Set FlashCopy
What is Consistency Group FlashCopy?
Consistency Group FlashCopy
FlashCopy to Pprc Primary Volume
FlashCopy Source Volume FlashCopy Target Volume
Persistent FlashCopy
Inband Commands over Remote Mirror link
Remote Mirror and Copy Peer-to-Peer Remote Copy
Metro Mirror Synchronous Pprc
Global Copy PPRC-XD
Metro Mirror
Global Mirror Asynchronous Pprc
Global Copy
How Global Mirror works
Global Mirror
Global Mirror How it works
OS Global Mirror XRC
Secondary server
11 z/OS Global Mirror
Comparison of the Remote Mirror and Copy functions
12 z/OS Metro/Global Mirror
Global Copy PPRC-XD
What is a Consistency Group?
What is data consistency?
How does Consistency Group keep data consistency?
1st
LSS13
Interfaces for Copy Services
Storage Hardware Management Console S-HMC
16 DS8000 Copy Services network components
DS Storage Manager Web-based interface
DS Command-Line Interface DS CLI
Tip What is changed from the ESS CLI?
DS Open application programming Interface API
Interoperability with ESS
Future Plans
140 DS8000 Series Concepts and Architecture
Part 3 Planning and configuration
141
142 DS8000 Series Concepts and Architecture
Installation planning
143
Packaged dimensions and weight for DS8000 models
General considerations
Delivery requirements
Installation site preparation
Floor and space requirements
145
Installing on raised or nonraised floors
Meeting floor load requirements
Calculating space requirements
DS8000 dimensions
Power requirements
Power control
147
Power outlet requirements
Input voltage requirements
Environmental requirements
Power connector requirements
Power consumption and environmental information
Fans and air intake areas
Host attachment
Attaching to open systems hosts
Cooling the storage complex
ESCON-attached S/390 and zSeries hosts
FICON-attached S/390 and zSeries hosts
151
Where to get the updated information for host attachment
Host systems attachment
SAN Fabric products
Channel extension technology products
Network and SAN requirements
HMC network requirements
153
Remote power control requirements
Remote support connection requirements
SAN requirements
155
156 DS8000 Series Concepts and Architecture
Configuration planning
157
Configuration planning overview
Storage Hardware Management Console S-HMC
S-HMC and Ethernet switches
External S-HMC
Remote services
HMC software components
DS Storage Manager
161
Required ports
HMC network topology
Dial-up connection
Secure high-speed connection
163
Remote access
Call home
Security mechanism 2 Public key encryption
HMC security considerations
Security mechanism 1 Console must initiate session
165
Security mechanism 3 Login security
FTP Offload option
HMC user management
DS8000 licensed functions
Operating environment license OEL required feature
Licensed function indicators
167
Point-in-Time Copy function 2244 Model PTC
Optional features
Operating environment license feature codes
License scope for each DS8000 licensed function
Remote Mirror and Copy functions 2244 Model RMC
Point-in-Time Copy PTC feature codes
Remote Mirror and Copy RMC feature codes
Remote Mirror for z/OS 2244 Model RMZ
Parallel Access Volumes PAV feature codes
Parallel Access Volumes 2244 Model PAV
Remote Mirror for zSeries RMZ feature codes
Ordering licensed functions
171
User authorize to FlashCopy 25 TB of CKD data
Solution
Disk storage feature activation
173
Scenarios for managing licensing
Logical configurations
Capacity planning
Adding storage capacity to an existing licensed function
175
CKD RAID rank capacity
Sparing rules
FB RAID rank capacity
Sparing examples
177
Sparing Example 2 RAID-10
179
11 Sparing example 3 RAID-5 Different capacity, same RPM
Sparing Example 4 RAID-5
IBM Standby Capacity on Demand Standby CoD
Capacity and well-balanced configuration
181
DDM to DA Mapping -- 2-way
Data migration planning
183
Software packages
Basic commands
Operating system mirroring
Remote copy technologies
Migration services and appliances
6 z/OS data migration methods
185
Planning for performance
15 Different data migration methods
Size of cache storage
Parallel Access Volumes z/OS only
Disk Magic
Number of host ports/channels
Hot spot avoidance
DS Storage Manager logical configuration
189
Configuration hierarchy, terminology, and concepts
Storage configuration terminology
PSeries1 PSeries2
Ranks
Array sites
Arrays
Extent pools
Extent Pool
Logical volumes
Volume groups
Volume Group
Host System C
Address groups
Extent Pool
Extent Pool
Planning
Raw or physical DDM layer
Array site layer
Array layer
Rank layer
Logical Configuration flow
Logical volume layer
Extent pool layer
Introducing the GUI and logical configuration panels
Connecting to the DS8000
10 Entering the URL using the TCP/IP address for the S-HMC
Welcome panel
Real-time Manager configuration
12 The Welcome panel
Copy Services
Simulated Manager configuration
Creating and defining the users and passwords
Log
Using the help panels information center
15 User administration panel
Navigating the GUI
17 View of the information center
19 View of the storage complexes in the work area
20 View of the Storage Complexes section
Radio buttons and check boxes
21 Storage unit view of the pull-down
Logical configuration process
Configuring a storage complex
Configuring the storage unit
25 The General storage unit information panel
Configuration advancement steps
28 Specify I/O adapter configuration panel
Configuring the logical host systems
29 Create host systems, screen
31 View of the General host information panel
33 Define host ports panel, with updated host information
Creating arrays from array sites
Click Apply assignment and OK
36 The Definition method panel
Creating extent pools
38 The Add arrays to ranks panel with FB selected
Creating FB volumes from extents
39 The Definition method panel
42 The Define volume properties panel
41 The Select extent pool panel
Creating volume groups
Click Volume Groups
44 The Define volume group properties filled out
Deleting LUNs and recovering space in the extent pool
Assigning LUNs to the hosts
Click Host Systems
Creating CKD LCUs
Creating CKD volumes
Click Volumes → zSeries
Click Finish
Displaying the storage unit Wwnn
Click Storage Units
49 View of the Wwnn in the General panel
230 DS8000 Series Concepts and Architecture
231
Introduction
Functionality
Installation methods
Supported environments
Command flow
ESS Copy Services command flow using ESS Copy Services CLI
DS CLI command flow
Secure sockets
TCP/IP ports
DS6000 command flow
DS8000 split network
Command flow for the DS6000
ESS CLI co-existence
Storage management
CLI co-existence
Example 11-1 Using DS CLI via a single command
Command modes
Single command mode
User security
Example 11-4 Using DS CLI in interactive mode
Interactive mode
Script mode
Example 11-3 Creating a DS CLI script
Syntax conventions
Lists all available DS CLI commands
Lists all DS CLI commands with syntax information
User assistance
Man pages
Example 11-8 Use of the help -lcommand
Return codes
DS CLI return codes
Example 11-11 Example of a configuration script
Usage examples
Example 11-10 Return code examples
Which CLI to use based on what hardware you have installed
Mixed device environments and migration
Determining the saved tasks to be migrated
DS CLI migration example
Migration tasks
Migration considerations
A portion of the tasks listed by using the GUI
Collecting the task details
Converting the saved task to a DS CLI command
Using the GUI to get the contents of a FlashCopy task
Example 11-14 Using interactive dscli mode without profiles
Converting a FlashCopy task to DS CLI
Setting up a profile
Using DS CLI commands via a single command or script
Procedure to create an encrypted password file
Creating a user ID for use only with ESS
Issuing a DS CLI command and scripting it
Example 11-18 Establishing a FlashCopy with a single command
Example 11-20 Using script mode
Example 11-19 Creating an executable file
252 DS8000 Series Concepts and Architecture
Performance considerations
253
What is the challenge?
Speed gap between server and disk storage
New and enhanced functions
Where do we start?
255
Arrays across loops
Switch from Escon to Ficon ports
SSA backend interconnection
Pprc over Fibre Channel links
FC-AL shortcomings
Fibre Channel switched disk interconnection at the back end
How does the DS8000 address the challenge?
How the DS8000 series overcomes FC-AL shortcomings
Memory
259
Fibre Channel device adapter
New four-port host adapters
POWER5 Heart of the DS8000 dual cluster design
261
RIO-G Interconnect
RIO-G Interconnect
263
Performance and sizing considerations for open systems
Vertical growth and scalability
Cache size considerations for open systems
Workload characteristics
Data placement in the DS8000
265
LVM striping
266 DS8000 Series Concepts and Architecture
Stripe size
267
Determining where to attach the host
Determining the number of paths to a LUN
Subsystem Device Driver SDD Dynamic I/O load-balancing
Performance and sizing considerations for z/OS
Connect to zSeries hosts
269
Performance potential in z/OS environments
12 DS8100 frontend connectivity example partial view
Appropriate DS8000 size in z/OS environments
Processor memory size considerations for z/OS environments
271
Or zSeries channel consolidation
Disk array sizing considerations for z/OS environments
273
Configuration recommendations for z/OS
Configure one extent pool for each single rank
Minimize the number of extent pools
275
Extent pool0 Extent pool1
Plan for a reasonable number of extent pools
277
15 Mix of extent pools
278 DS8000 Series Concepts and Architecture
279
280 DS8000 Series Concepts and Architecture
ZSeries software enhancements
281
Software enhancements for the DS8000
13.2 z/OS enhancements
Scalability support
Benefits of the scalability enhancements
Large Volume Support LVS
Read availability mask support
283
Initial Program Load IPL enhancements
13.2.5 DS8000 definition to host software
Read control unit and device recognition for DS8000
New performance statistics
285
Listdata Count
287
Listdata Counts report of DS8000
Listdata Status
Resource Management Facility RMF
289
Migration considerations
13.3 z/VM enhancements
13.4 z/VSE enhancements
Coexistence considerations
TPF enhancements
291
292 DS8000 Series Concepts and Architecture
Data migration in zSeries environments
293
Define migration objectives in z/OS environments
Consolidate storage subsystems
Consolidate logical volumes
Considerations for new logical volume size
295
Dynamic Parallel Access Volumes required for large volumes
Example 14-1 Data Classes with EF attribute
Keep source and target volume size at the current size
297
Data migration based on physical migration
Summary of data migration objectives
Physical migration with DFSMSdss and other storage software
Software- and hardware-based data migration
Data migration with Piper for z/OS
299
Piper hardware
Data migration with z/OS Global Mirror
301
Hardware- or microcode-based migration
Bridge from Escon to Ficon with Metro/Global Copy
Data migration with Metro Mirror or Global Copy
303
Pprcopy DDNAMEDD02 Query
305
Example 14-3 All data is replicated
491
Example 14-5 All data is replicated
Data Set Services Utility
Data migration based on logical migration
307
Hierarchical Storage Manager, DFSMShsm
System utilities
309
SMS Storage Groups migration source environment
Example 14-8 Select SMS storage group in Scds
Storage Group Application Selection
Alter Volume Statuses Pool Copy Pool Backup only
311
Example 14-12 Confirmation about SMS volume status change
ALL Volumes Altered
313
Combine physical and logical data migration
14.5 z/VM and VSE/ESA data migration
Summary of data migration
315
316 DS8000 Series Concepts and Architecture
317
318 DS8000 Series Concepts and Architecture
Open systems support and software
319
Where to look for updated and detailed information
Open systems support
Supported operating systems and servers
DS8000 Host Systems Attachment Guide
DS8000 Interoperability Matrix
IBM HBA Search Tool
TotalStorage Proven program
Emulex Corporation
Differences to the ESS
Platform and operating system vendors’ pages
Atto
Boot support
Additional supported configurations RPQ
Subsystem Device Driver
Other multipathing solutions
DS CLI
IBM TotalStorage Productivity Center
IBM TotalStorage Productivity Center
MDM main panel
Device Manager
Sample Device Manager view
TPC for Disk
Sample screenshot of TPC for Disk
Global Mirror Utility
TPC for Replication
Enterprise Remote Copy Management Facility eRCMF
332 DS8000 Series Concepts and Architecture
Data migration in the open Systems environment
333
Tor
Comparison of migration methods
Host operating system-based migration
Basic copy commands
Copy raw devices
Migration using volume management software
Online copy and synchronization with rsync
Backup and restore
Migration using LVM mirroring
Migration using backup and restore
Subsystem-based data migration
Metro Mirror and Global Copy
Combination of Metro Mirror and Global Copy
IBM Piper migration
Piper migration
IBM migration services
Other migration applications
Appendix A. Open systems operating systems specifics
343
Planning
Data placement considerations
DS8000 Host Systems Attachment Guide
Capacity planning considerations
Unix performance monitoring tools
Iostat
Example A-1 AIX iostat output
System Activity Report SAR
Example A-2 SUN Solaris iostat output
Vmstat
Example A-3 SAR Sample Output
Finding the World Wide Port Names
Other publications
AIX host attachment scripts
Example A-4 Finding Fibre Channel adapter WWN
Managing multiple paths
Determine the installed SDD level
Useful SDD commands
Subsystem device driver SDD
Example A-7 lsvpcfg command
Multipath I/O Mpio
Determine the installed Sddpcm level
Useful Mpio commands
Example A-8 lspath command result
LVM striping
LVM configuration
AIX access methods for I/O
LVM mirroring
Direct I/O
Boot device support
AIX on IBM iSeries
Concurrent I/O
Filemon
Monitoring I/O performance
Iostat
Example A-9 Filemon output file
IBM Mpio FC
Linux
Linux with zSeries and ESS Essentials
Existing reference material
Implementing Linux with IBM Disk Storage
Getting Started with zSeries Fibre Channel Protocol
Important Linux issues
Some Linux Scsi basics
Table A-1 Major numbers and special device files
Missing device files
Example A-10 Create new special device files for Scsi disks
Managing multiple paths
Limited number of Scsi devices
Scsi device assignment changes
RedHat Enterprise Linux RH-EL multiple LUN support
Fibre Channel disks discovered before internal Scsi disks
Adding FC disks dynamically
Example A-12 Sample /etc/modules.conf
Example A-13 SCSi disks attached at system start time
Linux on IBM iSeries
Gaps in the LUN sequence
Troubleshooting and monitoring
/proc pseudo file system
Example A-16 Sample /proc/scsi/scsi file
Performance monitoring with iostat
Generic Scsi tools
Example A-17 Sample /proc/scsi/qla2300/x
HBA and operating system settings
Microsoft Windows 2000/2003
SDD for Windows
Windows Server 2003 VDS support
Figure A-1 Microsoft VDS Architecture
HP OpenVMS
FC port configuration
Volume Shadow Copy Service
Geographically Dispersed Sites
Volume configuration
Configurations
Command Console LUN
OpenVMS volume shadowing
Appendix A. Open systems operating systems specifics
372 DS8000 Series Concepts and Architecture
Appendix B. Using DS8000 with iSeries
373
Supported environment
Logical volume sizes
Table B-1 OS/400 logical volume sizes
Hardware
Protected versus unprotected volumes
Changing LUN protection
Adding volumes to iSeries configuration
Using 5250 interface
Figure B-3 Work with Disk Configuration menu
Adding volumes to an Independent Auxiliary Storage Pool
Figure B-5 Confirm Add Units
Figure B-7 iSeries Navigator Signon to iSeries window
Figure B-9 SST Signon
Figure B-11 New disk pool welcome
Figure B-13 Confirm disk pool configuration
Figure B-15 Choose the disks to add to the Disk Pool
Figure B-17 New Disk Pool Summary
Figure B-20 New Disk Pool shown on iSeries Navigator
Multipath
Avoiding single points of failure
Configuring multipath
Figure B-22 Single points of failure
Adding multipath volumes to iSeries using 5250 interface
Figure B-24 Example of multipath with iSeries
Figure B-25 Adding multipath volumes to an ASP
Adding volumes to iSeries using iSeries Navigator
Figure B-27 Adding a multipath volume
Figure B-28 New Disk Pool shown on iSeries Navigator
Managing multipath volumes using iSeries Navigator
Figure B-30 Example of multipath logical units
Appendix B. Using DS8000 with iSeries
Figure B-32 Multipath logical unit properties
Multipath rules for multiple iSeries systems or partitions
Figure B-33 Multipath connections
Sizing guidelines
Changing from single path to multipath
Planning for arrays and DDMs
Cache
Number of iSeries Fibre Channel adapters
Size and number of LUNs
Table B-2 Capacity per I/O Adapter
Recommended number of ranks
Sharing ranks between iSeries and other servers
Table B-3 Disk operations per second per RAID rank
OS/400 mirroring
Connecting via SAN switches
Migration
Metro Mirror and Global Copy
Figure B-35 Using Metro Mirror to migrate from ESS to DS8000
OS/400 data migration
Figure B-36 Ending allocation for existing disk units
Copy Services for iSeries
FlashCopy
Remote Mirror and Copy
ISeries toolkit for Copy Services
AIX on IBM iSeries
Linux on IBM iSeries
406 DS8000 Series Concepts and Architecture
Appendix C. Service and support offerings
407
IBM Implementation Services for TotalStorage disk systems
IBM Web sites for service offerings
IBM service offerings
IBM Implementation Services for TotalStorage Copy Functions
IBM Geographically Dispersed Parallel Sysplex Gdps
Enterprise Remote Copy Management Facility eRCMF
IBM Operational Support Services Support Line
IBM eServer iSeries Copy Services
Appendix C. Service and support offerings
412 DS8000 Series Concepts and Architecture
IBM Redbooks
Other publications
413
Online resources
How to get IBM Redbooks
Help from IBM
416 DS8000 Series Concepts and Architecture
Index
417
DS CLI
OEL
Escon Ficon
RH-EL
BBU
Spcn
VSE
IBM TotalStorage DS8000 Series Concepts and Architecture
Page
Page
IBM TotalStorage DS8000 Series Concepts Architecture