Recovery and Restart Guide
Page
Recovery and Restart Guide
Copyright IBM Corporation 1982
Contents
Part 3. Implementing recovery
Restart
Programming for recovery 141
Disaster recovery 223
Part 4. Appendixes 241
How to use this book
Preface
What this book is about
Who should read this book
Viii Cics TS for z/OS 4.1 Recovery and Restart Guide
Copyright IBM Corp
Cics TS for z/OS 4.1 Recovery and Restart Guide
Part 1. Cics recovery and restart concepts
Cics TS for z/OS 4.1 Recovery and Restart Guide
Maintaining the integrity of data
Recovery and restart facilities
Logging changes
Minimizing the effect of failures
Role of Cics
Recoverable resources
Cics backward recovery backout
Dynamic transaction backout
Emergency restart backout
Cics forward recovery
Forward recovery of Cics data sets
Cics recovery processing following a communication failure
Failures that require Cics recovery processing
Forward recovery for non-VSAM resources
XCF/MRO partner failures
Cics recovery processing following a transaction failure
Cics recovery processing following a system failure
Recovery and restart facilities
Cics TS for z/OS 4.1 Recovery and Restart Guide
Units of work
Resource recovery in Cics
Shunted units of work
Locks
Active and retained states for locks
Synchronization points
Examples of synchronization points
Units of work and syncpoints
Cics recovery manager
Backout of units of work
DB2
Coordinating updates to local resources
Coordinating updates in distributed units of work
Managing indoubt units of work
Cics system log
Resynchronization after system or connection failure
Information recorded on the system log
System activity keypoints
Forward recovery logs
User journals and automatic journaling
Resource recovery in Cics
Cics TS for z/OS 4.1 Recovery and Restart Guide
Normal shutdown processing
Shutdown and restart recovery
First quiesce stage
Second quiesce stage
Third quiesce stage
Warm keypoints
Shunted units of work at shutdown
Immediate shutdown processing Perform Shutdown Immediate
Flushing journal buffers
Perform Immediate not recommended
Shutdown requested by the operating system
Dfhcesd
Uncontrolled termination
Shutdown assist transaction
Cataloging Cics resources
Global catalog
Local catalog
Effect of problems with the system log
Shutdown initiated by Cics log manager
How the state of the Cics region is reconstructed
DFHRM0403 and DFHRM0404
Overriding the type of start indicator
Warm restart
Emergency restart
About this task
Cold start
Recovery of data during an emergency restart
An initial start of Cics
Dynamic RLS restart
Recovery with Vtam persistent sessions
Running with persistent sessions support
SNPS, single-node persistent sessions
MNPS, multinode persistent sessions
Situations in which sessions are not reestablished
Running without persistent sessions support
Situations in which Vtam does not retain sessions
Regions that do have persistent sessions support
Cics TS for z/OS 4.1 Recovery and Restart Guide
Part 2. Recovery and restart processes
Cics TS for z/OS 4.1 Recovery and Restart Guide
Cics cold start
Starting Cics with the START=COLD parameter
Files
Vsam
Temporary storage
Transient data
Temporary storage data sharing server
Data tables
Resource definitions dynamically installed
Transactions
Journal names and journal models
Library resources
Monitoring and statistics
Committing and cataloging resources installed from the CSD
Terminal control resources
Installable set install
Distributed transaction resources
Dump table
Starting Cics with the START=INITIAL parameter
Cics cold start
Cics TS for z/OS 4.1 Recovery and Restart Guide
Cics warm restart
Rebuilding the Cics state after a normal shutdown
Data set name blocks
Reconnecting to Smsvsam for RLS access
Recreating non-RLS retained locks
TDINTRA=NOEMPTY the default
Trigger levels for Terminal and System only
No autoinstall for programs
Autoinstall for programs
Start requests
CSD-defined resource definitions
Tcam and sequential Bsam devices
Urimap definitions and virtual hosts
Cics TS for z/OS 4.1 Recovery and Restart Guide
Recovering after a Cics failure
Recovering information from the system log
Cics emergency restart
Driving backout processing for in-flight units of work
Other backout processing
Effect of delayed recovery on Pltpi processing
Rebuilding the Cics state after an abnormal termination
RLS restart processing and orphan locks
Reconnecting to Smsvsam for RLS access
Recreating non-RLS retained locks
Start requests
Distributed transaction resources
Cics TS for z/OS 4.1 Recovery and Restart Guide
Cics ARM processing
Automatic restart management
Restrictions
Registering with ARM
Waiting for predecessor subsystems
De-registering from ARM
Before you begin
ARM couple data sets
Failing to register
Cics restart JCL and parameters
Workload policies
Connecting to Vtam
Cics Start options
Covr transaction
Messages associated with automatic restart
Automatic restart of Cics data-sharing servers
Server ARM processing
Server commands for ARM support
Waiting on events during initialization
Server initialization parameters for ARM support
ARMELEMENTNAME=elementname
Unit of work recovery and abend processing
Unit of work recovery
In-flight-failed
Commit-failed
Backout-failed
Transaction backout
Indoubt-failed
Bdam files and Vsam Esds files
Files
Cics data tables
Auxiliary temporary storage
Intrapartition transient data
Start requests
Start with recoverable data Protect
Start with nonrecoverable data no Protect
Start with nonrecoverable data Protect
Restart of started transactions
Exec Cics Cancel requests
Basic mapping support BMS messages
Backout-failed recovery
Error
Transient data
Retrying backout-failed units of work
Disposition of data sets after backout failures
Logical delete not performed
Open error
Smsvsam server failure
DFSMSdss non-BWO backup in progress
Duplicate key error
Smsvsam server recycle during backout
Coupling facility cache structure failure
Lock structure full error
Commit-failed recovery
None of the above
Indoubt failure recovery
Investigating an indoubt failure
Cemt Inquire UOW details of UOW AC0CD65E5D990800
Cemt Inquire UOWENQ-details of a lock associated with a UOW
Recovery from failures associated with the coupling facility
Cache failure support
Rebuilding the lock structure
Lost locks recovery
Notifying Cics of Smsvsam restart
Performing lost locks recovery for failed units of work
Connection failure to a coupling facility cache structure
MVS system recovery and sysplex recovery
Connection failure to a coupling facility lock structure
Transaction abend processing
Exit code
Abnormal termination of a task
Transaction restart
Actions taken at transaction failure
Processing operating system abends and program checks
Unit of work recovery and abend processing
Cics TS for z/OS 4.1 Recovery and Restart Guide
Communication error processing
Terminal error processing
Node error program Dfhznep
Terminal error program Dfhtep
Intersystem communication failures
Part 3. Implementing recovery and restart
Cics TS for z/OS 4.1 Recovery and Restart Guide
Questions relating to recovery requirements
Planning aspects of recovery
Application design considerations
Validate the recovery requirements statement
End user’s standby procedures
Designing the end user’s restart procedure
Communications between application and user
System definitions for recovery-related functions
System recovery table SRT
Resource definitions for recovery
Security
Documentation and test plans
Transient data queues
Temporary storage table
Program list table PLT
Cics TS for z/OS 4.1 Recovery and Restart Guide
Defining system and general log streams
Defining log streams to MVS
Defining system log streams
System log streams
General log streams
Without a Journalmodel definition
Specifying a Journalmodel resource definition
With a Journalmodel definition
Recovery considerations
Model log streams for Cics system logs
Sharing system logger structures between 2 MVS images
Varying the model log stream name
Activity keypointing
About this task
Keeping system log data to a minimum
System log scan during restart
Log-tail deletion
Moving units of work to the secondary log
Avoiding retention periods on the system log
Writing user-recovery data
Retrieving user records from the system log
About this task Procedure
Defining forward recovery log streams
Long-running transactions
Model log streams for Cics general logs
What to do next
Merging data on shared general log streams
Defining the log of logs
Log of logs failure
Reading log streams offline
Effect of daylight saving time changes
Adjusting local time
Time stamping log and journal records
Offline utility program, Dfhjup
Cics TS for z/OS 4.1 Recovery and Restart Guide
Defining recoverability for CICS-managed resources
Recovery for transactions
Defining transaction recovery attributes
DTIMOUTNO1-6800
Indoubt options for distributed transactions
WAITTIME00,00,00dd,hh,mm
Recovery for files
File-owning regions and RLS access
Vsam files
Sharing data sets with batch jobs
Basic direct access method Bdam
Defining files as recoverable resources
Forward recovery
Backward recovery
Vsam files accessed in non-RLS mode
Recoveryall
Vsam files accessed in RLS mode
Inquiring on recovery attributes
File recovery attribute consistency checking non-RLS
Bdam files
CSD data set
Overriding open failures at the Xfcnrec global user exit
Cics responses to file open requests
Implementing forward recovery with user-written utilities
Implementing forward recovery with Cics Vsam Recovery
Recovery for intrapartition transient data
Backward recovery
Logical recovery
Physical recovery
Forward recovery
No recovery
Recovery for extrapartition transient data
Input extrapartition data sets
Recovery for temporary storage
Using post-initialization Pltpi programs
Output extrapartition data sets
Recovery for Web services
Configuring Cics to support persistent messages
Defining local queues in a service provider
Results What to do next
Error processing
Persistent message processing
Defining recoverability for CICS-managed resources
Cics TS for z/OS 4.1 Recovery and Restart Guide
Designing applications for recovery
Programming for recovery
Splitting the application into transactions
Cics TS for z/OS 4.1 Recovery and Restart Guide
Program design
SAA-compatible applications
Dividing transactions into units of work
Conversational processing
Processing dialogs with users
Pseudoconversational processing
Mechanisms for passing data between transactions
Cics recoverable resources
Main storage areas
Designing to avoid transaction deadlocks
Implications of interval control Start requests
Implications of automatic task initiation TD trigger level
Using transient data queues
Terminal paging through BMS
Managing transaction and system failures
Transaction failures
Exec Cics Syncpoint Rollback command
Handle Abend commands
Dynamic transaction backout
Use of the program error program Dfhpep
System failures
Handling abends and program level abend exits
Transaction restart after DTB
Command Information provided
Processing the Ioerr condition
PL/I programs and error handling
Start Transid commands
Locking enqueuing on resources in application programs
Nonrecoverable files
Implicit locking for files
Recoverable files
Cics TS for z/OS 4.1 Recovery and Restart Guide
Implicit enqueuing on recoverable temporary storage queues
Implicit enqueuing on logically recoverable TD destinations
Writeq TD Readq TD Deleteq TD
Implicit enqueuing on DL/I databases with Dbctl
Explicit enqueuing by the application programmer
Direct methods HDAM, Hidam
Sequential methods HSAM, HISAM, Shisam
Possibility of transaction deadlock
Deadlock
Where you can add your own code
User exits for transaction backout
Xrcinpt exit
Xrcinit exit
Xfcbfail global user exit
Xfcldel global user exit
Xfcbover global user exit
Xfcbout global user exit
Coding transaction backout exits
Using a program error program PEP
CICS-supplied PEP
Your own PEP
Omitting the PEP
Cics TS for z/OS 4.1 Recovery and Restart Guide
Resolving retained locks on recoverable resources
Quiescing RLS data sets
RLS quiesce and unquiesce functions
Illustration of the quiesce flow across two Cics regions
MVS1 MVS2 Cics AOR1 Cics AOR2
Non-BWO data set backup start
Other quiesce interface functions
Non-BWO data set backup end
BWO backup start
BWO backup end
Forward recovery complete
Switching from RLS to non-RLS access mode
Lost locks recovery complete
Exception for read-only operations
Quiesce coupling facility cache available
What can prevent a switch to non-RLS access mode?
Investigating which retained locks are held and why
Inquire Dsname
Shcds List subcommands
Resolving retained locks and preserving data integrity
Choosing data availability over data integrity
Cemt command examples
Batch-enabling sample programs
Uowdsnfail
Permitnonrlsupdate subcommand
Denynonrlsupdate subcommand
Special case lost locks
Overriding retained locks
Post-batch processing
Coupling facility data table retained locks
Procedure for moving a data set with retained locks
Moving recoverable data sets that have retained locks
Using the Repro method
Shcds Frsetrr
Using the Export and Import functions
Rebuilding alternate indexes
Forward recovery procedures
Forward recovery of data sets accessed in RLS mode
Recovery of data set with volume still available
Recovery of data set with loss of volume
Issue the Frbind subcommand
Issue the Frresetrr subcommand
Alter the new data set name
Volume recovery procedure using Cfvol Quiesce
Forward recovery procedures
Example of recovery using data set backup
Shcds LISTSUBSYSDS’ADSWA01D’
Vary SMS,SMSVSAM,FORCEDELETELOCKSTRUCTURE
Inquire Uowdsnfail Status Results
Example of recovery using volume backup
Catalog recovery
Above command produced the following message
Procedure for failed RLS mode forward recovery operation
Forward recovery of data sets accessed in non-RLS mode
Close all files
Forward recovery procedures
Uowactioncommit Backout Force
Procedure for failed non-RLS mode forward recovery operation
Cics TS for z/OS 4.1 Recovery and Restart Guide
BWO and backups
Backup-while-open BWO
BWO and concurrent copy
BWO requirements
Component name Full DFSMS/MVS name Previous product Name
Which data sets are eligible for BWO
Hardware requirements
Vsam control interval or control area split
How you request BWO
Specifying BWO using access method services
Results
Specifying BWO on Cics file resource definitions
Removing BWO attributes
Systems administration
Batch jobs
BWO processing
File opening
First file opened in non-RLS mode against a cluster
Back-level data sets
Subsequent files opened when use count is not zero
Subsequent files opened when use count is zero
File closing non-RLS mode
Restriction for Vsam upgrade set
Data set backup and restore
Shutdown and restart
Controlled shutdown
Immediate or uncontrolled shutdown
Vsam access method services
Invalid state changes for BWO attributes
Forward recovery logging
Data set restore
Data sets
Non-SMS managed storage
Recovery point non-RLS mode
Recovering Vsam spheres with AIXs
An assembler program that calls Dfsms callable services
ASM XOPTSCICS,NOEPILOG,SP
DS 8C
MVC
MVC SUCMSG18,SUCTXT1 SUCMSG21,SUCTXT2
Cics TS for z/OS 4.1 Recovery and Restart Guide
Disaster recovery
Why have a disaster recovery plan?
Disaster recovery testing
Tier 0 no off-site data
Six tiers of solutions for off-site recovery
Tier 1 physical removal
Disaster recovery tier 1 physical removal
Tier 3 electronic vaulting
Tier 2 physical removal with hot site
Tier
Tier 0-3 solutions
Summarizes the tier 3 solution
Tier 4 active secondary site
Tier 0
Disaster recovery tier 4 active secondary site
Tier 5 two-site, two-phase commit
Tier 6 minimal to zero data loss
Summarizes the tier 6 solution
Tier 4-6 solutions
Tier 4
Disaster recovery and high availability
Peer-to-peer remote copy Pprc and extended remote copy XRC
Pprc or XRC?
Use Pprc for high value transactions
Remote Recovery Data Facility
Other benefits of Pprc and XRC
Choosing between Rrdf and 3990-6 solutions
Disaster recovery personnel considerations
MVS system logger recovery support
Disaster recovery facilities
Returning to your primary site
Cics Vsam Recovery Qsam copy
Remote Recovery Data Facility support
Remote site recovery for RLS-mode data sets
Cics VR shadowing
Final summary
Part 4. Appendixes
241
Cics TS for z/OS 4.1 Recovery and Restart Guide
243
Trademarks
Bibliography
Cics books for Cics Transaction Server for z/OS
CICSPlex SM books for Cics Transaction Server for z/OS
Administration and Management
Other Cics publications
Accessibility
Cics TS for z/OS 4.1 Recovery and Restart Guide
Airdelay
Index
DL/I
251
Tpend
Readers’ Comments Wed Like to Hear from You
Readers’ Comments Wed Like to Hear from You
Page
SC34-7012-01