IBM DS8000 manual Online resources

Page 436

Online resources

These Web sites and URLs are also relevant as further information sources:

IBM Disk Storage Feature Activation (DSFA) Web site at http://www.ibm.com/storage/dsfa

The PSP information can be found at: http://www-1.ibm.com/servers/resourcelink/svc03100.nsf?OpenDatabase

Documentation for the DS8000: http://www.ibm.com/servers/storage/support/disk/2107.html

Supported servers for the DS8000: http://www.storage.ibm.com/hardsoft/products/DS8000/supserver.htm

The interoperability matrix: http://www.ibm.com/servers/storage/disk/DS8000/interop.html

Fibre Channel host bus adapter firmware and driver level matrix: http://knowledge.storage.ibm.com/HBA/HBASearchTool

ATTO:

http://www.attotech.com/

Emulex:

http://www.emulex.com/ts/dds.html

JNI:

http://www.jni.com/OEM/oem.cfm?ID=4

QLogic:

http://www.qlogic.com/support/oem_detail_all.asp?oemid=22

IBM:

http://www.ibm.com/storage/ibmsan/products/sanfabric.html

CNT (INRANGE): http://www.cnt.com/ibm/

McDATA:

http://www.mcdata.com/ibm/

Cisco:

http://www.cisco.com/go/ibm/storage

CIENA:

http://www.ciena.com/products/transport/shorthaul/cn2000/index.asp

CNT:

http://www.cnt.com/ibm/

Nortel:

http://www.nortelnetworks.com/

ADVA:

http://www.advaoptical.com/

414DS8000 Series: Concepts and Architecture

Image 436

Contents IBM TotalStorage DS8000 Series Page IBM TotalStorage DS8000 Series Concepts and Architecture First Edition April Contents Page Benefits of virtualization Vi DS8000 Series Concepts and Architecture Summary Viii DS8000 Series Concepts and Architecture Page Page Index Xii DS8000 Series Concepts and Architecture Copyright License Trademarks DfsortPreface Team that wrote this redbookXvi DS8000 Series Concepts and Architecture Preface Xviii DS8000 Series Concepts and Architecture Become a published author Comments welcomeXx DS8000 Series Concepts and Architecture Part 1 Introduction DS8000 Series Concepts and Architecture Introduction to the DS8000 series DS8000, a member of the TotalStorage DS family Overview of the DS8000 seriesInfrastructure Simplification Business ContinuityDS8000 Base frame Hardware overview Storage capacity Storage system logical partitions LPARsStorage Hardware Management Console S-HMC for the DS8000 IBM Standby Capacity on Demand offering for the DS8000Supported environments Resiliency Family for Business ContinuityIBM TotalStorage FlashCopy IBM TotalStorage Metro Mirror Synchronous PprcService and setup Three-site solutionInteroperability IBM TotalStorage Global Mirror Asynchronous PprcCommon set of functions PositioningDS8000 compared to ESS IBM TotalStorage DS Command-Line Interface DS CLI Common management functionsDS8000 compared to DS6000 IBM TotalStorage DS Storage ManagerFuture directions of storage system LPARs Scalability and configuration flexibilityDS Open application programming interface Sequential Prefetching in Adaptive Replacement Cache Sarc PerformanceIBM TotalStorage Multipath Subsystem Device Driver SDD Performance for zSeriesSummary 16 DS8000 Series Concepts and Architecture Part 2 Architecture 18 DS8000 Series Concepts and Architecture Components Frames Base frameExpansion frame Rack operator panelArchitecture Rack operator panelSAN fabric Server-based SMP design Cache managementComponents Processor complex RandomseqComponents Service processor and Spcn Front viewRear view Processor memoryRIO-G 2 I/O enclosuresRear view Disk subsystemDevice adapters Disk enclosures DS8000 device adapterNon-switched FC-AL drawbacks DS8000 disk enclosureSwitched FC-AL advantages 11 DS8000 disk enclosureDS8000 switched FC-AL implementation 12 Disk enclosure switched connectionsExpansion Arrays and spares14 DS8000 switched loop layout Arrays across loopsAAL benefits Host adaptersDDMs Ficon and Fibre Channel protocol host adapters Control units and logical pathsEscon distances Remote Mirror and Copy with EsconPower and cooling Rack Power Control cards RPCFibre Channel supported servers Fibre Channel distancesPrimary power supplies Processor and I/O enclosure power suppliesDisk enclosure power and cooling Battery backup assembliesEthernet switches 42 DS8000 Series Concepts and Architecture Storage system LPARs Logical partitions Introduction to logical partitioning Virtualization Engine technologyPartitioning concepts PartitionsPhysical partitioning Ppar Building blockLogical partitioning Lpar Software and hardware fault isolation Dynamic logical partitioningMicro-Partitioning Virtual I/OWhy Logically Partition? Server consolidationProduction and test environments Application isolationLpar and storage facility images DS8000 and LparIncreased flexibility of resource allocation 2 DS8300 Lpar implementation DS8300 Model 9A2 Lpar and storage facility imageStorage facility image hardware components DS8300 Lpar resource allocationStorage Facility Image Processor complex Processor and memory allocations4 DS8300 Model 9A2 configuration options RIO-G interconnect separationDS8300 example configuration Model conversionLpar security through Power Hypervisor Phyp Model conversions regarding Lpar functionalityLpar and Copy Services Lpar protection Power HypervisorFlashCopy Lpar benefitsRemote mirroring Storage system LPARs Logical partitions Addressing capabilities with storage facility images Example of storage facility images in the DS8300DS8300 addressing capabilities 60 DS8000 Series Concepts and Architecture RAS Naming Storage complexStorage unit Storage facility imageProcessor complex RAS Processor complexReliability, availability, and serviceability Permanent monitoringSelf-healing Memory reliability, fault tolerance, and integrity+1 redundancy Fault masking Concurrent MaintenanceHypervisor Storage image independence Resource deallocationServer RAS RIO-G a self-healing interconnect2 I/O enclosure Metadata checksServer failover and failback Data flowFailover Normal data flowNVS recovery after complete power loss FailbackHost connection availability Single or multiple pathEnclosure SAN/FICON/ESCON switches Multi-pathing softwareOpen systems host connection ZSeries host connectionSwitched disk connections Disk path redundancyRAID-5 overview RAID-10 overviewRAID-5 theory RAID-5 implementation in the DS8000RAID-10 theory Spare creationRAID-10 implementation in the DS8000 Floating spares Predictive Failure Analysis PFAHot plugable DDMs Battery backup units Rack power control card RPCDisk scrubbing Rack cooling fansBuilding power loss Power fluctuation protectionPower control of the DS8000 Emergency power off EPOMicrocode updates Installation processHMC considerations Management consoleDifferent code versions across storage images Virtualization concepts Virtualization definition Storage system virtualizationAbstraction layers for disk virtualization Storage Facility virtualizationArray sites Physical layer as the base for virtualizationArrays Array siteRanks Creation of an arrayExtent pools Forming an FB rank with 1 GB extentsServer0 CKD volumes Logical volumesFixed Block LUNs Volume with Allocation and deletion of LUNs/CKD volumes ISeries LUNsLogical subsystems LSS DB2 Volume access Address groupsVolume group Host attachment11 Host attachments and volume groups Summary of the virtualization hierarchyPlacement of data 12 Virtualization hierarchy13 Optimal placement of data Benefits of virtualization101 102 DS8000 Series Concepts and Architecture IBM TotalStorage DS8000 model overview and scalability 103DS8000 highlights Model naming conventions2 DS8100 Model Maximum configuration for the Model 3 DS8300 Models 922 and 9A2IBM TotalStorage DS8000 model overview and scalability Model comparison Designed for scalability Scalability for capacityLarge and scalable capacity DS8000 model comparisonScalability for performance Adding DDMsFuture plan Comparison of models for capacityWay I/O controllers Linear-scalable architectureRIO-G Benefit of the DS8000 for scalabilityModel upgrades Model conversions114 DS8000 Series Concepts and Architecture Copy Services 115Copy Services functions Introduction to Copy ServicesPoint-in-Time Copy FlashCopy Read from the target volume Read from the source volumeWrite to the source volume Point-in-Time Copy function authorization Refresh target volume also known as Incremental FlashCopyFlashCopy options No background copy optionData Set FlashCopy Incremental FlashCopyMultiple Relationship FlashCopy Data Set FlashCopyWhat is Consistency Group FlashCopy? Consistency Group FlashCopyFlashCopy to Pprc Primary Volume FlashCopy Source Volume FlashCopy Target VolumeInband Commands over Remote Mirror link Remote Mirror and Copy Peer-to-Peer Remote CopyPersistent FlashCopy Metro Mirror Synchronous PprcGlobal Copy PPRC-XD Metro MirrorGlobal Mirror Asynchronous Pprc Global CopyHow Global Mirror works Global MirrorGlobal Mirror How it works OS Global Mirror XRC Secondary server 11 z/OS Global MirrorComparison of the Remote Mirror and Copy functions 12 z/OS Metro/Global MirrorGlobal Copy PPRC-XD What is a Consistency Group? What is data consistency?How does Consistency Group keep data consistency? 1st LSS13 Interfaces for Copy Services Storage Hardware Management Console S-HMC16 DS8000 Copy Services network components DS Storage Manager Web-based interfaceTip What is changed from the ESS CLI? DS Command-Line Interface DS CLIDS Open application programming Interface API Interoperability with ESS Future Plans140 DS8000 Series Concepts and Architecture Part 3 Planning and configuration 141142 DS8000 Series Concepts and Architecture Installation planning 143General considerations Packaged dimensions and weight for DS8000 modelsDelivery requirements Floor and space requirements Installation site preparation145 Meeting floor load requirements Installing on raised or nonraised floorsCalculating space requirements Power requirements Power controlDS8000 dimensions 147Power outlet requirements Input voltage requirementsPower connector requirements Power consumption and environmental informationEnvironmental requirements Fans and air intake areasAttaching to open systems hosts Host attachmentCooling the storage complex FICON-attached S/390 and zSeries hosts ESCON-attached S/390 and zSeries hosts151 Host systems attachment Where to get the updated information for host attachmentSAN Fabric products Network and SAN requirements HMC network requirementsChannel extension technology products 153Remote support connection requirements Remote power control requirementsSAN requirements 155 156 DS8000 Series Concepts and Architecture Configuration planning 157Configuration planning overview Storage Hardware Management Console S-HMCS-HMC and Ethernet switches External S-HMCHMC software components Remote servicesDS Storage Manager 161 Required portsHMC network topology Secure high-speed connection Dial-up connection163 Remote access Call homeHMC security considerations Security mechanism 1 Console must initiate sessionSecurity mechanism 2 Public key encryption 165FTP Offload option Security mechanism 3 Login securityHMC user management Operating environment license OEL required feature Licensed function indicatorsDS8000 licensed functions 167Optional features Operating environment license feature codesPoint-in-Time Copy function 2244 Model PTC License scope for each DS8000 licensed functionPoint-in-Time Copy PTC feature codes Remote Mirror and Copy RMC feature codesRemote Mirror and Copy functions 2244 Model RMC Remote Mirror for z/OS 2244 Model RMZParallel Access Volumes 2244 Model PAV Remote Mirror for zSeries RMZ feature codesParallel Access Volumes PAV feature codes Ordering licensed functions171 User authorize to FlashCopy 25 TB of CKD dataSolution Disk storage feature activation 173Logical configurations Capacity planningScenarios for managing licensing Adding storage capacity to an existing licensed function175 CKD RAID rank capacitySparing rules FB RAID rank capacitySparing examples 177Sparing Example 2 RAID-10 179 11 Sparing example 3 RAID-5 Different capacity, same RPMSparing Example 4 RAID-5 IBM Standby Capacity on Demand Standby CoDCapacity and well-balanced configuration 181DDM to DA Mapping -- 2-way Data migration planning 183Basic commands Operating system mirroringSoftware packages Remote copy technologies6 z/OS data migration methods Migration services and appliances185 Planning for performance 15 Different data migration methodsParallel Access Volumes z/OS only Disk MagicSize of cache storage Number of host ports/channelsHot spot avoidance DS Storage Manager logical configuration 189Configuration hierarchy, terminology, and concepts Storage configuration terminologyPSeries1 PSeries2 Array sites ArraysRanks Extent poolsExtent Pool Logical volumes Volume groupsVolume Group Host System C Address groupsExtent Pool Extent Pool Planning Array site layer Raw or physical DDM layerArray layer Logical Configuration flow Logical volume layerRank layer Extent pool layerIntroducing the GUI and logical configuration panels Connecting to the DS800010 Entering the URL using the TCP/IP address for the S-HMC Welcome panelReal-time Manager configuration 12 The Welcome panelCopy Services Simulated Manager configurationCreating and defining the users and passwords LogUsing the help panels information center 15 User administration panelNavigating the GUI 17 View of the information center19 View of the storage complexes in the work area 20 View of the Storage Complexes sectionRadio buttons and check boxes 21 Storage unit view of the pull-downLogical configuration process Configuring a storage complexConfiguring the storage unit 25 The General storage unit information panel Configuration advancement steps 28 Specify I/O adapter configuration panel Configuring the logical host systems 29 Create host systems, screen31 View of the General host information panel 33 Define host ports panel, with updated host information Creating arrays from array sites Click Apply assignment and OK36 The Definition method panel Creating extent pools 38 The Add arrays to ranks panel with FB selectedCreating FB volumes from extents 39 The Definition method panel42 The Define volume properties panel 41 The Select extent pool panelCreating volume groups Click Volume Groups44 The Define volume group properties filled out Assigning LUNs to the hosts Deleting LUNs and recovering space in the extent poolClick Host Systems Creating CKD volumes Click Volumes → zSeriesCreating CKD LCUs Click FinishDisplaying the storage unit Wwnn Click Storage Units49 View of the Wwnn in the General panel 230 DS8000 Series Concepts and Architecture 231 Introduction FunctionalityInstallation methods Supported environmentsCommand flow ESS Copy Services command flow using ESS Copy Services CLISecure sockets DS CLI command flowTCP/IP ports DS6000 command flow DS8000 split networkCommand flow for the DS6000 ESS CLI co-existenceStorage management CLI co-existenceCommand modes Single command modeExample 11-1 Using DS CLI via a single command User securityInteractive mode Script modeExample 11-4 Using DS CLI in interactive mode Example 11-3 Creating a DS CLI scriptLists all available DS CLI commands Lists all DS CLI commands with syntax informationSyntax conventions User assistanceExample 11-8 Use of the help -lcommand Return codesMan pages DS CLI return codesUsage examples Example 11-11 Example of a configuration scriptExample 11-10 Return code examples Which CLI to use based on what hardware you have installed Mixed device environments and migrationDS CLI migration example Migration tasksDetermining the saved tasks to be migrated Migration considerationsA portion of the tasks listed by using the GUI Collecting the task detailsConverting the saved task to a DS CLI command Using the GUI to get the contents of a FlashCopy taskExample 11-14 Using interactive dscli mode without profiles Converting a FlashCopy task to DS CLIUsing DS CLI commands via a single command or script Procedure to create an encrypted password fileSetting up a profile Creating a user ID for use only with ESSIssuing a DS CLI command and scripting it Example 11-18 Establishing a FlashCopy with a single commandExample 11-20 Using script mode Example 11-19 Creating an executable file252 DS8000 Series Concepts and Architecture Performance considerations 253Speed gap between server and disk storage What is the challenge?New and enhanced functions Where do we start? 255Switch from Escon to Ficon ports SSA backend interconnectionArrays across loops Pprc over Fibre Channel linksFibre Channel switched disk interconnection at the back end How does the DS8000 address the challenge?FC-AL shortcomings How the DS8000 series overcomes FC-AL shortcomingsMemory 259 Fibre Channel device adapter New four-port host adaptersPOWER5 Heart of the DS8000 dual cluster design 261RIO-G Interconnect RIO-G Interconnect 263Performance and sizing considerations for open systems Vertical growth and scalabilityWorkload characteristics Data placement in the DS8000Cache size considerations for open systems 265LVM striping 266 DS8000 Series Concepts and ArchitectureStripe size 267Determining the number of paths to a LUN Determining where to attach the hostSubsystem Device Driver SDD Dynamic I/O load-balancing Connect to zSeries hosts Performance and sizing considerations for z/OS269 Performance potential in z/OS environments 12 DS8100 frontend connectivity example partial viewProcessor memory size considerations for z/OS environments Appropriate DS8000 size in z/OS environments271 Or zSeries channel consolidation Disk array sizing considerations for z/OS environments 273Configuration recommendations for z/OS Configure one extent pool for each single rankMinimize the number of extent pools 275Extent pool0 Extent pool1 Plan for a reasonable number of extent pools277 15 Mix of extent pools278 DS8000 Series Concepts and Architecture 279 280 DS8000 Series Concepts and Architecture ZSeries software enhancements 28113.2 z/OS enhancements Software enhancements for the DS8000Scalability support Large Volume Support LVS Read availability mask supportBenefits of the scalability enhancements 28313.2.5 DS8000 definition to host software Initial Program Load IPL enhancementsRead control unit and device recognition for DS8000 New performance statistics 285Listdata Count 287 Listdata Counts report of DS8000Listdata Status Resource Management Facility RMF 28913.3 z/VM enhancements 13.4 z/VSE enhancementsMigration considerations Coexistence considerationsTPF enhancements 291292 DS8000 Series Concepts and Architecture Data migration in zSeries environments 293Define migration objectives in z/OS environments Consolidate storage subsystemsConsiderations for new logical volume size Consolidate logical volumes295 Dynamic Parallel Access Volumes required for large volumes Example 14-1 Data Classes with EF attributeKeep source and target volume size at the current size 297Summary of data migration objectives Data migration based on physical migrationPhysical migration with DFSMSdss and other storage software Data migration with Piper for z/OS Software- and hardware-based data migration299 Piper hardware Data migration with z/OS Global Mirror 301Hardware- or microcode-based migration Bridge from Escon to Ficon with Metro/Global CopyData migration with Metro Mirror or Global Copy 303Pprcopy DDNAMEDD02 Query 305 Example 14-3 All data is replicated491 Example 14-5 All data is replicatedData migration based on logical migration Data Set Services Utility307 Hierarchical Storage Manager, DFSMShsm System utilities309 SMS Storage Groups migration source environmentExample 14-8 Select SMS storage group in Scds Storage Group Application SelectionAlter Volume Statuses Pool Copy Pool Backup only 311Example 14-12 Confirmation about SMS volume status change ALL Volumes Altered313 Combine physical and logical data migration Summary of data migration 14.5 z/VM and VSE/ESA data migration315 316 DS8000 Series Concepts and Architecture 317 318 DS8000 Series Concepts and Architecture Open systems support and software 319Open systems support Where to look for updated and detailed informationSupported operating systems and servers DS8000 Interoperability Matrix IBM HBA Search ToolDS8000 Host Systems Attachment Guide TotalStorage Proven programDifferences to the ESS Platform and operating system vendors’ pagesEmulex Corporation AttoBoot support Additional supported configurations RPQSubsystem Device Driver Other multipathing solutions DS CLIIBM TotalStorage Productivity Center IBM TotalStorage Productivity CenterMDM main panel Device Manager Sample Device Manager viewTPC for Disk Sample screenshot of TPC for DiskGlobal Mirror Utility TPC for ReplicationEnterprise Remote Copy Management Facility eRCMF 332 DS8000 Series Concepts and Architecture Data migration in the open Systems environment 333Tor Comparison of migration methods Host operating system-based migrationBasic copy commands Copy raw devicesMigration using volume management software Online copy and synchronization with rsyncBackup and restore Migration using LVM mirroringMigration using backup and restore Subsystem-based data migrationMetro Mirror and Global Copy Combination of Metro Mirror and Global CopyIBM Piper migration Piper migrationIBM migration services Other migration applicationsAppendix A. Open systems operating systems specifics 343Data placement considerations DS8000 Host Systems Attachment GuidePlanning Capacity planning considerationsIostat Unix performance monitoring toolsExample A-1 AIX iostat output System Activity Report SAR Example A-2 SUN Solaris iostat outputVmstat Example A-3 SAR Sample OutputOther publications AIX host attachment scriptsFinding the World Wide Port Names Example A-4 Finding Fibre Channel adapter WWNDetermine the installed SDD level Useful SDD commandsManaging multiple paths Subsystem device driver SDDExample A-7 lsvpcfg command Multipath I/O MpioUseful Mpio commands Determine the installed Sddpcm levelExample A-8 lspath command result LVM configuration AIX access methods for I/OLVM striping LVM mirroringBoot device support AIX on IBM iSeriesDirect I/O Concurrent I/OMonitoring I/O performance IostatFilemon Example A-9 Filemon output fileIBM Mpio FC Linux Existing reference material Implementing Linux with IBM Disk StorageLinux with zSeries and ESS Essentials Getting Started with zSeries Fibre Channel ProtocolSome Linux Scsi basics Important Linux issuesTable A-1 Major numbers and special device files Missing device files Example A-10 Create new special device files for Scsi disksLimited number of Scsi devices Managing multiple pathsScsi device assignment changes RedHat Enterprise Linux RH-EL multiple LUN support Fibre Channel disks discovered before internal Scsi disksExample A-12 Sample /etc/modules.conf Adding FC disks dynamicallyExample A-13 SCSi disks attached at system start time Linux on IBM iSeries Gaps in the LUN sequence/proc pseudo file system Troubleshooting and monitoringExample A-16 Sample /proc/scsi/scsi file Generic Scsi tools Performance monitoring with iostatExample A-17 Sample /proc/scsi/qla2300/x Microsoft Windows 2000/2003 HBA and operating system settingsSDD for Windows Windows Server 2003 VDS support Figure A-1 Microsoft VDS ArchitectureFC port configuration Volume Shadow Copy ServiceHP OpenVMS Geographically Dispersed SitesVolume configuration ConfigurationsCommand Console LUN OpenVMS volume shadowingAppendix A. Open systems operating systems specifics 372 DS8000 Series Concepts and Architecture Appendix B. Using DS8000 with iSeries 373Logical volume sizes Table B-1 OS/400 logical volume sizesSupported environment HardwareProtected versus unprotected volumes Changing LUN protectionAdding volumes to iSeries configuration Using 5250 interfaceFigure B-3 Work with Disk Configuration menu Adding volumes to an Independent Auxiliary Storage Pool Figure B-5 Confirm Add UnitsFigure B-7 iSeries Navigator Signon to iSeries window Figure B-9 SST Signon Figure B-11 New disk pool welcome Figure B-13 Confirm disk pool configuration Figure B-15 Choose the disks to add to the Disk Pool Figure B-17 New Disk Pool Summary Figure B-20 New Disk Pool shown on iSeries Navigator Multipath Avoiding single points of failureConfiguring multipath Figure B-22 Single points of failureAdding multipath volumes to iSeries using 5250 interface Figure B-24 Example of multipath with iSeriesFigure B-25 Adding multipath volumes to an ASP Adding volumes to iSeries using iSeries Navigator Figure B-27 Adding a multipath volumeFigure B-28 New Disk Pool shown on iSeries Navigator Managing multipath volumes using iSeries Navigator Figure B-30 Example of multipath logical unitsAppendix B. Using DS8000 with iSeries Figure B-32 Multipath logical unit properties Multipath rules for multiple iSeries systems or partitions Figure B-33 Multipath connectionsSizing guidelines Changing from single path to multipathPlanning for arrays and DDMs CacheSize and number of LUNs Number of iSeries Fibre Channel adaptersTable B-2 Capacity per I/O Adapter Sharing ranks between iSeries and other servers Recommended number of ranksTable B-3 Disk operations per second per RAID rank Connecting via SAN switches MigrationOS/400 mirroring Metro Mirror and Global CopyFigure B-35 Using Metro Mirror to migrate from ESS to DS8000 OS/400 data migrationFigure B-36 Ending allocation for existing disk units FlashCopy Copy Services for iSeriesRemote Mirror and Copy ISeries toolkit for Copy Services AIX on IBM iSeriesLinux on IBM iSeries 406 DS8000 Series Concepts and Architecture Appendix C. Service and support offerings 407IBM Web sites for service offerings IBM service offeringsIBM Implementation Services for TotalStorage disk systems IBM Implementation Services for TotalStorage Copy FunctionsIBM Geographically Dispersed Parallel Sysplex Gdps Enterprise Remote Copy Management Facility eRCMFIBM Operational Support Services Support Line IBM eServer iSeries Copy ServicesAppendix C. Service and support offerings 412 DS8000 Series Concepts and Architecture Other publications IBM Redbooks413 Online resources How to get IBM Redbooks Help from IBM416 DS8000 Series Concepts and Architecture Index 417DS CLI OEL Escon Ficon RH-EL BBU Spcn VSE IBM TotalStorage DS8000 Series Concepts and Architecture Page Page IBM TotalStorage DS8000 Series Concepts Architecture

Related manuals

Manual 78 pages 25.99 Kb

Manual 374 pages 22.96 Kb