Version 4 Release
Recovery and Restart Guide
CICS Transaction Server for z/OS
SC34-7012-01
Page
Version 4 Release
Recovery and Restart Guide
CICS Transaction Server for z/OS
SC34-7012-01
Copyright IBM Corporation 1982
Part 2. Recovery and restart processes
Contents
Part 1. CICS recovery and restart concepts
Chapter 2. Resource recovery in CICS
recovery
Part 3. Implementing recovery and
Chapter 9. Communication error
Chapter 8. Unit of work recovery and
Chapter 17. Forward recovery
Chapter 13. Programming for recovery
Chapter 16. Moving recoverable data
Chapter 18. Backup-while-open BWO
Part 4. Appendixes
Chapter 19. Disaster recovery
Accessibility
Notices
What this book is about
How to use this book
Preface
Who should read this book
viii CICS TS for z/OS 4.1 Recovery and Restart Guide
Changes in CICS Transaction Server for z/OS, Version 4 Release
x CICS TS for z/OS 4.1 Recovery and Restart Guide
Part 1. CICS recovery and restart concepts
2 CICS TS for z/OS 4.1 Recovery and Restart Guide
Logging changes
Chapter 1. Recovery and restart facilities
Maintaining the integrity of data
4 CICS TS for z/OS 4.1 Recovery and Restart Guide
Minimizing the effect of failures
The role of CICS
Recoverable resources
CICS backward recovery backout
Dynamic transaction backout
Emergency restart backout
CICS forward recovery
Forward recovery of CICS data sets
Forward recovery for non-VSAM resources
Failures that require CICS recovery processing
CICS recovery processing following a communication failure
XCF/MRO partner failures
10 CICS TS for z/OS 4.1 Recovery and Restart Guide
CICS recovery processing following a transaction failure
CICS recovery processing following a system failure
v Back out recoverable resources
12 CICS TS for z/OS 4.1 Recovery and Restart Guide
Shunted units of work
Chapter 2. Resource recovery in CICS
Units of work
Locks
Active and retained states for locks
EXEC CICS DISCARD CONNECTION EXEC CICS DISCARD TERMINAL
Synchronization points
EXEC CICS CREATE TERMINAL EXEC CICS CREATE CONNECTION COMPLETE
EXEC CICS CREATE TERMINAL
EXEC CICS DISCARD TERMINAL
Examples of synchronization points
EXEC CICS CREATE CONNECTION COMPLETE EXEC CICS DISCARD CONNECTION
CICS recovery manager
v Coordinating recoverable conversations to remote nodes
FC/RLS
Recovery
Figure 3. CICS recovery manager and resources it works with
Coordinating updates to local resources
Coordinating updates in distributed units of work
Managing indoubt units of work
Information recorded on the system log
Resynchronization after system or connection failure
CICS system log
User journals and automatic journaling
Forward recovery logs
System activity keypoints
Input or output messages from terminals accessed through VTAM
24 CICS TS for z/OS 4.1 Recovery and Restart Guide
First quiesce stage
Chapter 3. Shutdown and restart recovery
Normal shutdown processing
Second quiesce stage
Third quiesce stage
Warm keypoints
Shunted units of work at shutdown
PERFORM IMMEDIATE not recommended
Flushing journal buffers
Immediate shutdown processing PERFORM SHUTDOWN IMMEDIATE
Shutdown requested by the operating system
30 CICS TS for z/OS 4.1 Recovery and Restart Guide
Uncontrolled termination
The shutdown assist transaction
Cataloging CICS resources
Global catalog
Local catalog
Effect of problems with the system log
Shutdown initiated by CICS log manager
DFHRM0403 and DFHRM0404
How the state of the CICS region is reconstructed
DFHRM0402
DFHRM0405
Emergency restart
Overriding the type of start indicator
Warm restart
About this task
An initial start of CICS
Recovery of data during an emergency restart
Cold start
Dynamic RLS restart
SNPS, single-node persistent sessions
Recovery with VTAM persistent sessions
Running with persistent sessions support
MNPS, multinode persistent sessions
Situations in which sessions are not reestablished
SET VTAM FORCECLOSE SET VTAM IMMCLOSE SET VTAM CLOSED
Running without persistent sessions support
Situations in which VTAM does not retain sessions
regions that do have persistent sessions support
42 CICS TS for z/OS 4.1 Recovery and Restart Guide
Part 2. Recovery and restart processes
44 CICS TS for z/OS 4.1 Recovery and Restart Guide
About this task
Chapter 4. CICS cold start
Starting CICS with the START=COLD parameter
Files
VSAM
Temporary storage data sharing server
Temporary storage
Transient data
Data tables
Journal names and journal models
Resource definitions dynamically installed
Transactions
LIBRARY resources
Monitoring and statistics
Committing and cataloging resources installed from the CSD
Single resource install
Terminal control resources
Dump table
Installable set install
Distributed transaction resources
Starting CICS with the START=INITIAL parameter
information saved in the system log from a previous run. The primary and secondary system log streams are purged and CICS begins writing a new system log
52 CICS TS for z/OS 4.1 Recovery and Restart Guide
Chapter 5. CICS warm restart
Rebuilding the CICS state after a normal shutdown
Recreating non-RLS retained locks
Reconnecting to SMSVSAM for RLS access
Data set name blocks
Files
Trigger levels for TERMINAL and SYSTEM only
TDINTRA=NOEMPTY the default
TDINTRA=EMPTY
Temporary storage
LIBRARY resources
No autoinstall for programs
Transactions
Programs
Monitoring and statistics
Autoinstall for programs
Start requests
Journal names and journal models
CSD-defined resource definitions
TCAM and sequential BSAM devices
Terminal control resources
Distributed transaction resources
URIMAP definitions and virtual hosts
v Different TCT from last run. CICS installs the TCT only, and does not apply the warm keypoint information, effectively making this a cold start for these devices
60 CICS TS for z/OS 4.1 Recovery and Restart Guide
Chapter 6. CICS emergency restart
Recovering after a CICS failure
Recovering information from the system log
Driving backout processing for in-flight units of work
Rebuilding the CICS state after an abnormal termination
Effect of delayed recovery on PLTPI processing
Other backout processing
62 CICS TS for z/OS 4.1 Recovery and Restart Guide
Recreating non-RLS retained locks
Reconnecting to SMSVSAM for RLS access
RLS restart processing and orphan locks
Temporary storage
Terminal control resources
64 CICS TS for z/OS 4.1 Recovery and Restart Guide
Start requests
CSD-defined resource definitions
TCAM and sequential BSAM devices
is successful, but CICS abnormally terminates before the catalog can be updated, CICS recovers the information from the forward recovery records on the system log
Distributed transaction resources
66 CICS TS for z/OS 4.1 Recovery and Restart Guide
Restrictions
Chapter 7. Automatic restart management
CICS ARM processing
De-registering from ARM
Registering with ARM
Waiting for predecessor subsystems
Before you begin
CICS restart JCL and parameters
Failing to register
ARM couple data sets
Chapter 7. Automatic restart management
CICS START options
Connecting to VTAM
Workload policies
Automatic restart of CICS data-sharing servers
The COVR transaction
Messages associated with automatic restart
Server ARM processing
Waiting on events during initialization
Server commands for ARM support
CANCEL RESTART=NOYES
Server initialization parameters for ARM support
In-flight-failed
Chapter 8. Unit of work recovery and abend processing
Unit of work recovery
Commit-failed
Indoubt-failed
Transaction backout
Backout-failed
CICS data tables
Files
BDAM files and VSAM ESDS files
Auxiliary temporary storage
START with recoverable data no PROTECT
Intrapartition transient data
START requests
START with nonrecoverable data PROTECT
START with recoverable data PROTECT
START with nonrecoverable data no PROTECT
Restart of started transactions
Basic mapping support BMS messages
EXEC CICS CANCEL requests
Backout-failed recovery
Auxiliary temporary storage
Retrying backout-failed units of work
I/O error
Transient data
Disposition of data sets after backout failures
SMSVSAM server failure
Open error
Logical delete not performed
SMSVSAM server recycle during backout
DFSMSdss non-BWO backup in progress
Duplicate key error
Coupling facility cache structure failure
None of the above
Commit-failed recovery
Lock structure full error
Indoubt failure recovery
Files
Auxiliary temporary storage
Investigating an indoubt failure
Intrapartition transient data
The WAITSTATE of Shunted shows that this UOW has been suspended
We can now see that
Recovery from failures associated with the coupling facility
Cache failure support
Notifying CICS of SMSVSAM restart
Lost locks recovery
Rebuilding the lock structure
About this task
Performing lost locks recovery for failed units of work
90 CICS TS for z/OS 4.1 Recovery and Restart Guide
Connection failure to a coupling facility lock structure
MVS system recovery and sysplex recovery
Connection failure to a coupling facility cache structure
Transaction abend processing
Exit code
Abnormal termination of a task
Transaction restart
Actions taken at transaction failure
Processing operating system abends and program checks
v If a match is not found, CICS is terminated
96 CICS TS for z/OS 4.1 Recovery and Restart Guide
Node error program DFHZNEP
Chapter 9. Communication error processing
Terminal error processing
Terminal error program DFHTEP
Intersystem communication failures
Part 3. Implementing recovery and restart
100 CICS TS for z/OS 4.1 Recovery and Restart Guide
Application design considerations
Chapter 10. Planning aspects of recovery
Questions relating to recovery requirements
Validate the recovery requirements statement
Communications between application and user
Designing the end user’s restart procedure
End user’s standby procedures
About this task
Resource definitions for recovery
System definitions for recovery-related functions
System recovery table SRT
Security
Temporary storage table
Documentation and test plans
Transient data queues
Program list table PLT
v Forecast the exceptional conditions that can be expected
Chapter 11. Defining system and general log streams
recovery purposes only
Forward recovery logging
System logging
System log streams
Defining log streams to MVS
Defining system log streams
General log streams
With a JOURNALMODEL definition
Specifying a JOURNALMODEL resource definition
Without a JOURNALMODEL definition
Recovery considerations
Model log streams for CICS system logs
2-Way Sysplex
About this task
Activity keypointing
Varying the model log stream name
Keeping system log data to a minimum
About this task
About this task
Log-tail deletion
Moving units of work to the secondary log
Retrieving user records from the system log
Writing user-recovery data
Avoiding retention periods on the system log
About this task
Long-running transactions
Defining forward recovery log streams
About this task Procedure
About this task
Model log streams for CICS general logs
What to do next
About this task
Merging data on shared general log streams
Defining the log of logs
About this task
Log of logs failure
Reading log streams offline
About this task
Time stamping log and journal records
Adjusting local time
Effect of daylight saving time changes
About this task
Offline utility program, DFHJUP
122 CICS TS for z/OS 4.1 Recovery and Restart Guide
Defining transaction recovery attributes
Chapter 12. Defining recoverability for CICS-managed resources
Recovery for transactions
RESTARTNOYES
TPURGENOYES
Indoubt options for distributed transactions
SPURGENOYES
ACTIONBACKOUTCOMMIT
VSAM files
Recovery for files
File-owning regions and RLS access
Sharing data sets with batch jobs
Forward recovery
Basic direct access method BDAM
Defining files as recoverable resources
Backward recovery
RECOVERYBACKOUTONLY
VSAM files accessed in non-RLS mode
RECOVERYALL
BACKUPTYPEDYNAMIC
LOGNONEUNDOALL
VSAM files accessed in RLS mode
Inquiring on recovery attributes
NONE
The CSD data set
File recovery attribute consistency checking non-RLS
BDAM files
Overriding open failures at the XFCNREC global user exit
CICS responses to file open requests
About this task
Recovery for intrapartition transient data
Implementing forward recovery with user-written utilities
Implementing forward recovery with CICS VSAM Recovery MVS/ESA
Backward recovery
Logical recovery
Physical recovery
Forward recovery
No recovery
Recovery for extrapartition transient data
Input extrapartition data sets
Output extrapartition data sets
Using post-initialization PLTPI programs
Recovery for temporary storage
Backward recovery
Forward recovery
Recovery for Web services
Configuring CICS to support persistent messages
GROUPTSRECOV PREFIXDF LOCATIONAUXILIARY RECOVERYYES
Results What to do next
Defining local queues in a service provider
Procedure
Procedure
Error processing
Persistent message processing
For example, your recovery transaction could
140 CICS TS for z/OS 4.1 Recovery and Restart Guide
Splitting the application into transactions
Chapter 13. Programming for recovery
Designing applications for recovery
About this task Procedure
Example What to do next Relationships between processing units
142 CICS TS for z/OS 4.1 Recovery and Restart Guide
Dividing transactions into units of work
SAA-compatible applications
Program design
Procedure
Pseudoconversational processing
Processing dialogs with users
Conversational processing
Main storage areas
CICS recoverable resources
Mechanisms for passing data between transactions
Transient data queues
Designing to avoid transaction deadlocks
Temporary storage auxiliary
User files and DL/I and DB2 databases
Implications of interval control START requests
Procedure
Implications of presenting large amounts of data to the user
Using transient data queues
Implications of automatic task initiation TD trigger level
Terminal paging through BMS
About this task
Managing transaction and system failures
Transaction failures
About this task
Dynamic transaction backout
HANDLE ABEND commands
EXEC CICS SYNCPOINT ROLLBACK command
Handling abends and program level abend exits
Use of the program error program DFHPEP
System failures
Transaction restart after DTB
Information provided
Processing the IOERR condition
Command
Locking enqueuing on resources in application programs
START TRANSID commands
PL/I programs and error handling
About this task
Nonrecoverable files
Implicit locking for files
Recoverable files
v READ for UPDATE v WRITE v DELETE
v WRITEQ TD v READQ TD v DELETEQ TD
Implicit enqueuing on logically recoverable TD destinations
Implicit enqueuing on recoverable temporary storage queues
Direct methods HDAM, HIDAM
Implicit enqueuing on DL/I databases with DBCTL
Explicit enqueuing by the application programmer
Sequential methods HSAM, HISAM, SHISAM
Possibility of transaction deadlock
deadlock
About this task
Where you can add your own code
User exits for transaction backout
XFCBFAIL global user exit
XRCINIT exit
XRCINPT exit
Procedure
XFCBOUT global user exit
XFCLDEL global user exit
XFCBOVER global user exit
Coding transaction backout exits
Procedure
Chapter 14. Using a program error program PEP
The CICS-supplied PEP
About this task
Your own PEP
About this task
Omitting the PEP
Chapter 14. Using a program error program PEP
166 CICS TS for z/OS 4.1 Recovery and Restart Guide
About this task
Chapter 15. Resolving retained locks on recoverable resources
Quiescing RLS data sets
About this task
The RLS quiesce and unquiesce functions
Illustration of the quiesce flow across two CICS regions
Chapter 15. Resolving retained locks on recoverable resources
Non-BWO data set backup start
Other quiesce interface functions
BWO backup end
Non-BWO data set backup end
BWO backup start
Forward recovery complete
Exception for read-only operations
Switching from RLS to non-RLS access mode
Lost locks recovery complete
Quiesce coupling facility cache available
What can prevent a switch to non-RLS access mode?
Procedure
Resolving retained locks before opening data sets in non-RLS mode
Investigating which retained locks are held and why
About this task
About this task
INQUIRE DSNAME
INQUIRE UOWDSNFAIL
About this task Procedure
SHCDS LIST subcommands
Resolving retained locks and preserving data integrity
176 CICS TS for z/OS 4.1 Recovery and Restart Guide
Choosing data availability over data integrity
About this task
DFH0BAT1
CEMT command examples
The batch-enabling sample programs
DFH0BAT2
DsnRLS.ACCOUNTS.ESDS.DBASE1
SET DSNAME’RLS.ACCOUNTS.ESDS.DBASE1’ RETRY
STATUS RESULTS - OVERTYPE TO MODIFY
NORMAL
A special case lost locks
The PERMITNONRLSUPDATE subcommand
The DENYNONRLSUPDATE subcommand
Overriding retained locks
Post-batch processing
Coupling facility data table retained locks
Using the REPRO method
Chapter 16. Moving recoverable data sets that have retained locks
Procedure for moving a data set with retained locks
About this task
SHCDS FRUNBIND
SHCDS FRRESETRR
SHCDS FRSETRR
SHCDS FRBIND
About this task
Using the EXPORT and IMPORT functions
Chapter 16. Moving recoverable data sets that have retained locks
Rebuilding alternate indexes
About this task
About this task
Chapter 17. Forward recovery procedures
Forward recovery of data sets accessed in RLS mode
4. Issue FRUNBIND
Recovery of data set with volume still available
3. Issue FRSETRR
5. Restore the backup
11. Issue the FRRESETRR subcommand
Recovery of data set with loss of volume
10. Issue the FRBIND subcommand
9. Alter the new data set name
Volume recovery procedure using CFVOL QUIESCE
1. VARY SMS,CFVOLvolser,QUIESCE
Example of recovery using data set backup
5. We terminated the SMSVSAM servers using the MVS command
194 CICS TS for z/OS 4.1 Recovery and Restart Guide
INQUIRE UOWDSNFAIL DSNRLSADSW.VF01D.BANKACCT
These commands are issued to each CICS AOR that requires access
Example of recovery using volume backup
196 CICS TS for z/OS 4.1 Recovery and Restart Guide
Catalog recovery
3. Restore the backup
Forward recovery of data sets accessed in non-RLS mode
Procedure for failed RLS mode forward recovery operation
4. Run the forward recovery utility
1. Tidy up any outstanding CICS recovery work, as follows
1 Force shunted indoubt units of work using SET DSNAME
Procedure for failed non-RLS mode forward recovery operation
202 CICS TS for z/OS 4.1 Recovery and Restart Guide
BWO and concurrent copy
Chapter 18. Backup-while-open BWO
BWO and backups
Full DFSMS/MVS name
BWO requirements
Component name
Previous product
VSAM control interval or control area split
Hardware requirements
Which data sets are eligible for BWO
Results
Specifying BWO using access method services
How you request BWO
TYPECICS
Specifying BWO on CICS file resource definitions
About this task
Batch jobs
Systems administration
Removing BWO attributes
Procedure
BWO processing
Back-level data sets
First file opened in non-RLS mode against a cluster
File opening
Subsequent files opened when use count is not zero
Subsequent files opened when use count is zero
File closing non-RLS mode
Restriction for VSAM upgrade set
Controlled shutdown
Data set backup and restore
Shutdown and restart
Immediate or uncontrolled shutdown
VSAM access method services
Invalid state changes for BWO attributes
Data sets
Forward recovery logging
Data set restore
Non-SMS managed storage
Recovery point non-RLS mode
Forward recovery
Recovering VSAM spheres with AIXs
About this task
An assembler program that calls DFSMS callable services
RECOVPTP
DATETIME
DTTENTHS
DATEPACK
BWOFLAGS12,ZEROES
220 CICS TS for z/OS 4.1 Recovery and Restart Guide
PRGCONT
LOAD
END PROG
222 CICS TS for z/OS 4.1 Recovery and Restart Guide
Chapter 19. Disaster recovery
Why have a disaster recovery plan?
Disaster recovery testing
Tier 1 - physical removal
Six tiers of solutions for off-site recovery
Tier 0 no off-site data
The drawbacks are
Tier
Tier 2 - physical removal with hot site
Tier 3 - electronic vaulting
Tier 0-3 solutions
Tier 4 - active secondary site
Tier 0
Figure 22. Disaster recovery tier 4 active secondary site
230 CICS TS for z/OS 4.1 Recovery and Restart Guide
Tier 5 - two-site, two-phase commit
Tier 6 - minimal to zero data loss
Figure 24 summarizes the tier 6 solution
Tier 4-6 solutions
Tier 4
Disaster recovery and high availability
Peer-to-peer remote copy PPRC and extended remote copy XRC
Use PPRC for high value transactions
Use XRC for high volume transactions
PPRC or XRC?
Forward recovery
Remote Recovery Data Facility
Other benefits of PPRC and XRC
236 CICS TS for z/OS 4.1 Recovery and Restart Guide
About this task
Choosing between RRDF and 3990-6 solutions
Disaster recovery personnel considerations
RRDF
Returning to your primary site
Disaster recovery facilities
MVS system logger recovery support
238 CICS TS for z/OS 4.1 Recovery and Restart Guide
Remote site recovery for RLS-mode data sets
CICS VSAM Recovery QSAM copy
Remote Recovery Data Facility support
CICS VR shadowing
Final summary
Part 4. Appendixes
Copyright IBM Corp. 1982
242 CICS TS for z/OS 4.1 Recovery and Restart Guide
Notices
Trademarks
244 CICS TS for z/OS 4.1 Recovery and Restart Guide
Bibliography
Access to CICS
Administration
CICS books for CICS Transaction Server for z/OS
Other CICS publications
Administration and Management
CICSPlex SM books for CICS Transaction Server for z/OS
General
Accessibility
248 CICS TS for z/OS 4.1 Recovery and Restart Guide
Index A
DL/I continued
files continued
locking continued
Page
CICS Transaction Server for z/OS Version 4 Release
Readers’ Comments - Wed Like to Hear from You
Recovery and Restart Guide Publication No. SC34-7012-01
SC34-7012-01
Readers’ Comments - Wed Like to Hear from You
SC34-7012-01
IBM United Kingdom Limited
Page
SC34-7012-01