CICS Transaction Server for z/OS
Recovery and Restart Guide
Version 4 Release
SC34-7012-01
Page
CICS Transaction Server for z/OS
Recovery and Restart Guide
Version 4 Release
SC34-7012-01
Copyright IBM Corporation 1982
Part 1. CICS recovery and restart concepts
Contents
Part 2. Recovery and restart processes
Chapter 2. Resource recovery in CICS
Chapter 9. Communication error
Part 3. Implementing recovery and
recovery
Chapter 8. Unit of work recovery and
Chapter 16. Moving recoverable data
Chapter 13. Programming for recovery
Chapter 17. Forward recovery
Chapter 18. Backup-while-open BWO
Accessibility
Chapter 19. Disaster recovery
Part 4. Appendixes
Notices
Preface
How to use this book
What this book is about
Who should read this book
viii CICS TS for z/OS 4.1 Recovery and Restart Guide
Changes in CICS Transaction Server for z/OS, Version 4 Release
x CICS TS for z/OS 4.1 Recovery and Restart Guide
Part 1. CICS recovery and restart concepts
2 CICS TS for z/OS 4.1 Recovery and Restart Guide
Maintaining the integrity of data
Chapter 1. Recovery and restart facilities
Logging changes
The role of CICS
Minimizing the effect of failures
4 CICS TS for z/OS 4.1 Recovery and Restart Guide
CICS backward recovery backout
Recoverable resources
Emergency restart backout
Dynamic transaction backout
Forward recovery of CICS data sets
CICS forward recovery
CICS recovery processing following a communication failure
Failures that require CICS recovery processing
Forward recovery for non-VSAM resources
XCF/MRO partner failures
CICS recovery processing following a system failure
CICS recovery processing following a transaction failure
10 CICS TS for z/OS 4.1 Recovery and Restart Guide
v Back out recoverable resources
12 CICS TS for z/OS 4.1 Recovery and Restart Guide
Units of work
Chapter 2. Resource recovery in CICS
Shunted units of work
Active and retained states for locks
Locks
EXEC CICS CREATE TERMINAL EXEC CICS CREATE CONNECTION COMPLETE
Synchronization points
EXEC CICS DISCARD CONNECTION EXEC CICS DISCARD TERMINAL
EXEC CICS CREATE TERMINAL
EXEC CICS CREATE CONNECTION COMPLETE EXEC CICS DISCARD CONNECTION
Examples of synchronization points
EXEC CICS DISCARD TERMINAL
CICS recovery manager
Recovery
FC/RLS
v Coordinating recoverable conversations to remote nodes
Figure 3. CICS recovery manager and resources it works with
Coordinating updates to local resources
Managing indoubt units of work
Coordinating updates in distributed units of work
CICS system log
Resynchronization after system or connection failure
Information recorded on the system log
System activity keypoints
Forward recovery logs
User journals and automatic journaling
Input or output messages from terminals accessed through VTAM
24 CICS TS for z/OS 4.1 Recovery and Restart Guide
Normal shutdown processing
Chapter 3. Shutdown and restart recovery
First quiesce stage
Third quiesce stage
Second quiesce stage
Shunted units of work at shutdown
Warm keypoints
Immediate shutdown processing PERFORM SHUTDOWN IMMEDIATE
Flushing journal buffers
PERFORM IMMEDIATE not recommended
Shutdown requested by the operating system
The shutdown assist transaction
Uncontrolled termination
30 CICS TS for z/OS 4.1 Recovery and Restart Guide
Global catalog
Cataloging CICS resources
Local catalog
Shutdown initiated by CICS log manager
Effect of problems with the system log
DFHRM0402
How the state of the CICS region is reconstructed
DFHRM0403 and DFHRM0404
DFHRM0405
Warm restart
Overriding the type of start indicator
Emergency restart
About this task
Cold start
Recovery of data during an emergency restart
An initial start of CICS
Dynamic RLS restart
Running with persistent sessions support
Recovery with VTAM persistent sessions
SNPS, single-node persistent sessions
MNPS, multinode persistent sessions
Situations in which sessions are not reestablished
Situations in which VTAM does not retain sessions
Running without persistent sessions support
SET VTAM FORCECLOSE SET VTAM IMMCLOSE SET VTAM CLOSED
regions that do have persistent sessions support
42 CICS TS for z/OS 4.1 Recovery and Restart Guide
Part 2. Recovery and restart processes
44 CICS TS for z/OS 4.1 Recovery and Restart Guide
Starting CICS with the START=COLD parameter
Chapter 4. CICS cold start
About this task
VSAM
Files
Transient data
Temporary storage
Temporary storage data sharing server
Data tables
Transactions
Resource definitions dynamically installed
Journal names and journal models
LIBRARY resources
Single resource install
Committing and cataloging resources installed from the CSD
Monitoring and statistics
Terminal control resources
Distributed transaction resources
Installable set install
Dump table
Starting CICS with the START=INITIAL parameter
information saved in the system log from a previous run. The primary and secondary system log streams are purged and CICS begins writing a new system log
52 CICS TS for z/OS 4.1 Recovery and Restart Guide
Rebuilding the CICS state after a normal shutdown
Chapter 5. CICS warm restart
Data set name blocks
Reconnecting to SMSVSAM for RLS access
Recreating non-RLS retained locks
Files
TDINTRA=EMPTY
TDINTRA=NOEMPTY the default
Trigger levels for TERMINAL and SYSTEM only
Temporary storage
Transactions
No autoinstall for programs
LIBRARY resources
Programs
Start requests
Autoinstall for programs
Monitoring and statistics
TCAM and sequential BSAM devices
CSD-defined resource definitions
Journal names and journal models
Terminal control resources
v Different TCT from last run. CICS installs the TCT only, and does not apply the warm keypoint information, effectively making this a cold start for these devices
URIMAP definitions and virtual hosts
Distributed transaction resources
60 CICS TS for z/OS 4.1 Recovery and Restart Guide
Recovering information from the system log
Recovering after a CICS failure
Chapter 6. CICS emergency restart
Driving backout processing for in-flight units of work
Other backout processing
Effect of delayed recovery on PLTPI processing
Rebuilding the CICS state after an abnormal termination
62 CICS TS for z/OS 4.1 Recovery and Restart Guide
RLS restart processing and orphan locks
Reconnecting to SMSVSAM for RLS access
Recreating non-RLS retained locks
Temporary storage
Start requests
64 CICS TS for z/OS 4.1 Recovery and Restart Guide
Terminal control resources
CSD-defined resource definitions
Distributed transaction resources
is successful, but CICS abnormally terminates before the catalog can be updated, CICS recovers the information from the forward recovery records on the system log
TCAM and sequential BSAM devices
66 CICS TS for z/OS 4.1 Recovery and Restart Guide
CICS ARM processing
Chapter 7. Automatic restart management
Restrictions
Waiting for predecessor subsystems
Registering with ARM
De-registering from ARM
Before you begin
ARM couple data sets
Failing to register
CICS restart JCL and parameters
Chapter 7. Automatic restart management
Workload policies
Connecting to VTAM
CICS START options
Messages associated with automatic restart
The COVR transaction
Automatic restart of CICS data-sharing servers
Server ARM processing
CANCEL RESTART=NOYES
Server commands for ARM support
Waiting on events during initialization
Server initialization parameters for ARM support
Unit of work recovery
Chapter 8. Unit of work recovery and abend processing
In-flight-failed
Commit-failed
Backout-failed
Transaction backout
Indoubt-failed
BDAM files and VSAM ESDS files
Files
CICS data tables
Intrapartition transient data
START with recoverable data no PROTECT
Auxiliary temporary storage
START requests
START with nonrecoverable data no PROTECT
START with recoverable data PROTECT
START with nonrecoverable data PROTECT
Restart of started transactions
EXEC CICS CANCEL requests
Basic mapping support BMS messages
Auxiliary temporary storage
Backout-failed recovery
Transient data
I/O error
Retrying backout-failed units of work
Disposition of data sets after backout failures
Logical delete not performed
Open error
SMSVSAM server failure
Duplicate key error
DFSMSdss non-BWO backup in progress
SMSVSAM server recycle during backout
Coupling facility cache structure failure
Lock structure full error
Commit-failed recovery
None of the above
Files
Indoubt failure recovery
Intrapartition transient data
Investigating an indoubt failure
Auxiliary temporary storage
The WAITSTATE of Shunted shows that this UOW has been suspended
We can now see that
Cache failure support
Recovery from failures associated with the coupling facility
Rebuilding the lock structure
Lost locks recovery
Notifying CICS of SMSVSAM restart
About this task
90 CICS TS for z/OS 4.1 Recovery and Restart Guide
Performing lost locks recovery for failed units of work
Connection failure to a coupling facility cache structure
MVS system recovery and sysplex recovery
Connection failure to a coupling facility lock structure
Exit code
Transaction abend processing
Transaction restart
Abnormal termination of a task
Processing operating system abends and program checks
Actions taken at transaction failure
v If a match is not found, CICS is terminated
96 CICS TS for z/OS 4.1 Recovery and Restart Guide
Terminal error processing
Chapter 9. Communication error processing
Node error program DFHZNEP
Terminal error program DFHTEP
Intersystem communication failures
Part 3. Implementing recovery and restart
100 CICS TS for z/OS 4.1 Recovery and Restart Guide
Questions relating to recovery requirements
Chapter 10. Planning aspects of recovery
Application design considerations
Validate the recovery requirements statement
End user’s standby procedures
Designing the end user’s restart procedure
Communications between application and user
About this task
System recovery table SRT
System definitions for recovery-related functions
Resource definitions for recovery
Security
Transient data queues
Documentation and test plans
Temporary storage table
Program list table PLT
v Forecast the exceptional conditions that can be expected
Forward recovery logging
recovery purposes only
Chapter 11. Defining system and general log streams
System logging
Defining system log streams
Defining log streams to MVS
System log streams
General log streams
Without a JOURNALMODEL definition
Specifying a JOURNALMODEL resource definition
With a JOURNALMODEL definition
Model log streams for CICS system logs
Recovery considerations
2-Way Sysplex
Varying the model log stream name
Activity keypointing
About this task
About this task
Keeping system log data to a minimum
Moving units of work to the secondary log
Log-tail deletion
About this task
Avoiding retention periods on the system log
Writing user-recovery data
Retrieving user records from the system log
About this task
About this task Procedure
Defining forward recovery log streams
Long-running transactions
About this task
What to do next
Model log streams for CICS general logs
Defining the log of logs
Merging data on shared general log streams
About this task
Reading log streams offline
Log of logs failure
About this task
About this task
Effect of daylight saving time changes
Adjusting local time
Time stamping log and journal records
About this task
Offline utility program, DFHJUP
122 CICS TS for z/OS 4.1 Recovery and Restart Guide
Recovery for transactions
Chapter 12. Defining recoverability for CICS-managed resources
Defining transaction recovery attributes
RESTARTNOYES
SPURGENOYES
Indoubt options for distributed transactions
TPURGENOYES
ACTIONBACKOUTCOMMIT
File-owning regions and RLS access
Recovery for files
VSAM files
Sharing data sets with batch jobs
Defining files as recoverable resources
Basic direct access method BDAM
Forward recovery
Backward recovery
RECOVERYALL
VSAM files accessed in non-RLS mode
RECOVERYBACKOUTONLY
BACKUPTYPEDYNAMIC
Inquiring on recovery attributes
VSAM files accessed in RLS mode
LOGNONEUNDOALL
NONE
BDAM files
File recovery attribute consistency checking non-RLS
The CSD data set
Overriding open failures at the XFCNREC global user exit
About this task
CICS responses to file open requests
Implementing forward recovery with CICS VSAM Recovery MVS/ESA
Implementing forward recovery with user-written utilities
Recovery for intrapartition transient data
Backward recovery
Physical recovery
Logical recovery
No recovery
Forward recovery
Input extrapartition data sets
Recovery for extrapartition transient data
Recovery for temporary storage
Using post-initialization PLTPI programs
Output extrapartition data sets
Backward recovery
Configuring CICS to support persistent messages
Recovery for Web services
Forward recovery
GROUPTSRECOV PREFIXDF LOCATIONAUXILIARY RECOVERYYES
Procedure
Defining local queues in a service provider
Results What to do next
Procedure
Persistent message processing
Error processing
For example, your recovery transaction could
140 CICS TS for z/OS 4.1 Recovery and Restart Guide
Designing applications for recovery
Chapter 13. Programming for recovery
Splitting the application into transactions
About this task Procedure
142 CICS TS for z/OS 4.1 Recovery and Restart Guide
Example What to do next Relationships between processing units
Program design
SAA-compatible applications
Dividing transactions into units of work
Procedure
Conversational processing
Processing dialogs with users
Pseudoconversational processing
Mechanisms for passing data between transactions
CICS recoverable resources
Main storage areas
Temporary storage auxiliary
Designing to avoid transaction deadlocks
Transient data queues
User files and DL/I and DB2 databases
Procedure
Implications of interval control START requests
Implications of automatic task initiation TD trigger level
Using transient data queues
Implications of presenting large amounts of data to the user
Terminal paging through BMS
Transaction failures
Managing transaction and system failures
About this task
About this task
EXEC CICS SYNCPOINT ROLLBACK command
HANDLE ABEND commands
Dynamic transaction backout
System failures
Use of the program error program DFHPEP
Handling abends and program level abend exits
Transaction restart after DTB
Command
Processing the IOERR condition
Information provided
PL/I programs and error handling
START TRANSID commands
Locking enqueuing on resources in application programs
Implicit locking for files
Nonrecoverable files
About this task
Recoverable files
v READ for UPDATE v WRITE v DELETE
Implicit enqueuing on recoverable temporary storage queues
Implicit enqueuing on logically recoverable TD destinations
v WRITEQ TD v READQ TD v DELETEQ TD
Explicit enqueuing by the application programmer
Implicit enqueuing on DL/I databases with DBCTL
Direct methods HDAM, HIDAM
Sequential methods HSAM, HISAM, SHISAM
deadlock
Possibility of transaction deadlock
User exits for transaction backout
Where you can add your own code
About this task
XRCINPT exit
XRCINIT exit
XFCBFAIL global user exit
Procedure
XFCBOVER global user exit
XFCLDEL global user exit
XFCBOUT global user exit
Coding transaction backout exits
The CICS-supplied PEP
Chapter 14. Using a program error program PEP
Procedure
About this task
Your own PEP
Chapter 14. Using a program error program PEP
Omitting the PEP
About this task
166 CICS TS for z/OS 4.1 Recovery and Restart Guide
Quiescing RLS data sets
Chapter 15. Resolving retained locks on recoverable resources
About this task
About this task
Illustration of the quiesce flow across two CICS regions
The RLS quiesce and unquiesce functions
Chapter 15. Resolving retained locks on recoverable resources
Other quiesce interface functions
Non-BWO data set backup start
BWO backup start
Non-BWO data set backup end
BWO backup end
Forward recovery complete
Lost locks recovery complete
Switching from RLS to non-RLS access mode
Exception for read-only operations
Quiesce coupling facility cache available
What can prevent a switch to non-RLS access mode?
Investigating which retained locks are held and why
Resolving retained locks before opening data sets in non-RLS mode
Procedure
About this task
INQUIRE UOWDSNFAIL
INQUIRE DSNAME
About this task
Resolving retained locks and preserving data integrity
SHCDS LIST subcommands
About this task Procedure
176 CICS TS for z/OS 4.1 Recovery and Restart Guide
About this task
Choosing data availability over data integrity
The batch-enabling sample programs
CEMT command examples
DFH0BAT1
DFH0BAT2
STATUS RESULTS - OVERTYPE TO MODIFY
SET DSNAME’RLS.ACCOUNTS.ESDS.DBASE1’ RETRY
DsnRLS.ACCOUNTS.ESDS.DBASE1
NORMAL
The DENYNONRLSUPDATE subcommand
The PERMITNONRLSUPDATE subcommand
A special case lost locks
Overriding retained locks
Post-batch processing
Coupling facility data table retained locks
Procedure for moving a data set with retained locks
Chapter 16. Moving recoverable data sets that have retained locks
Using the REPRO method
About this task
SHCDS FRSETRR
SHCDS FRRESETRR
SHCDS FRUNBIND
SHCDS FRBIND
Chapter 16. Moving recoverable data sets that have retained locks
Using the EXPORT and IMPORT functions
About this task
About this task
Rebuilding alternate indexes
Forward recovery of data sets accessed in RLS mode
Chapter 17. Forward recovery procedures
About this task
3. Issue FRSETRR
Recovery of data set with volume still available
4. Issue FRUNBIND
5. Restore the backup
10. Issue the FRBIND subcommand
Recovery of data set with loss of volume
11. Issue the FRRESETRR subcommand
9. Alter the new data set name
Volume recovery procedure using CFVOL QUIESCE
1. VARY SMS,CFVOLvolser,QUIESCE
Example of recovery using data set backup
5. We terminated the SMSVSAM servers using the MVS command
INQUIRE UOWDSNFAIL DSNRLSADSW.VF01D.BANKACCT
194 CICS TS for z/OS 4.1 Recovery and Restart Guide
These commands are issued to each CICS AOR that requires access
196 CICS TS for z/OS 4.1 Recovery and Restart Guide
Example of recovery using volume backup
Catalog recovery
Procedure for failed RLS mode forward recovery operation
Forward recovery of data sets accessed in non-RLS mode
3. Restore the backup
4. Run the forward recovery utility
1. Tidy up any outstanding CICS recovery work, as follows
1 Force shunted indoubt units of work using SET DSNAME
Procedure for failed non-RLS mode forward recovery operation
202 CICS TS for z/OS 4.1 Recovery and Restart Guide
BWO and backups
Chapter 18. Backup-while-open BWO
BWO and concurrent copy
Component name
BWO requirements
Full DFSMS/MVS name
Previous product
Which data sets are eligible for BWO
Hardware requirements
VSAM control interval or control area split
How you request BWO
Specifying BWO using access method services
Results
TYPECICS
About this task
Specifying BWO on CICS file resource definitions
Removing BWO attributes
Systems administration
Batch jobs
Procedure
BWO processing
File opening
First file opened in non-RLS mode against a cluster
Back-level data sets
Subsequent files opened when use count is zero
Subsequent files opened when use count is not zero
Restriction for VSAM upgrade set
File closing non-RLS mode
Shutdown and restart
Data set backup and restore
Controlled shutdown
Immediate or uncontrolled shutdown
Invalid state changes for BWO attributes
VSAM access method services
Data set restore
Forward recovery logging
Data sets
Non-SMS managed storage
Forward recovery
Recovery point non-RLS mode
About this task
Recovering VSAM spheres with AIXs
An assembler program that calls DFSMS callable services
DTTENTHS
DATETIME
RECOVPTP
DATEPACK
PRGCONT
220 CICS TS for z/OS 4.1 Recovery and Restart Guide
BWOFLAGS12,ZEROES
LOAD
END PROG
222 CICS TS for z/OS 4.1 Recovery and Restart Guide
Why have a disaster recovery plan?
Chapter 19. Disaster recovery
Disaster recovery testing
Tier 0 no off-site data
Six tiers of solutions for off-site recovery
Tier 1 - physical removal
The drawbacks are
Tier 3 - electronic vaulting
Tier 2 - physical removal with hot site
Tier
Tier 0-3 solutions
Tier 0
Tier 4 - active secondary site
230 CICS TS for z/OS 4.1 Recovery and Restart Guide
Figure 22. Disaster recovery tier 4 active secondary site
Tier 6 - minimal to zero data loss
Tier 5 - two-site, two-phase commit
Figure 24 summarizes the tier 6 solution
Tier 4
Tier 4-6 solutions
Peer-to-peer remote copy PPRC and extended remote copy XRC
Disaster recovery and high availability
PPRC or XRC?
Use XRC for high volume transactions
Use PPRC for high value transactions
Other benefits of PPRC and XRC
Remote Recovery Data Facility
Forward recovery
236 CICS TS for z/OS 4.1 Recovery and Restart Guide
Disaster recovery personnel considerations
Choosing between RRDF and 3990-6 solutions
About this task
RRDF
MVS system logger recovery support
Disaster recovery facilities
Returning to your primary site
238 CICS TS for z/OS 4.1 Recovery and Restart Guide
Remote Recovery Data Facility support
CICS VSAM Recovery QSAM copy
Remote site recovery for RLS-mode data sets
CICS VR shadowing
Final summary
Copyright IBM Corp. 1982
Part 4. Appendixes
242 CICS TS for z/OS 4.1 Recovery and Restart Guide
Notices
244 CICS TS for z/OS 4.1 Recovery and Restart Guide
Trademarks
Administration
Access to CICS
Bibliography
CICS books for CICS Transaction Server for z/OS
CICSPlex SM books for CICS Transaction Server for z/OS
Administration and Management
Other CICS publications
General
Accessibility
248 CICS TS for z/OS 4.1 Recovery and Restart Guide
Index A
files continued
DL/I continued
locking continued
Page
Recovery and Restart Guide Publication No. SC34-7012-01
Readers’ Comments - Wed Like to Hear from You
CICS Transaction Server for z/OS Version 4 Release
SC34-7012-01
Readers’ Comments - Wed Like to Hear from You
SC34-7012-01
IBM United Kingdom Limited
Page
SC34-7012-01