VSAM specific concepts

VSAM log reading

The VSAM log reader service that runs on z/OS® captures change data from a z/OS System Logger log stream, referred to as the replication log.

The replication log is specified in the VSAM cluster definition (LOGSTREAMID) for each data set mapped in a subscription. Each subscription has a single replication log. A subscription cannot be associated with more than one replication log.

Multiple subscriptions can share the same replication log. However, this is not optimal because each subscription must read all blocks in the replication log.

You must define log streams in data sharing and non-data sharing environments.

The type of log stream that you define depends on whether you access the log stream from one LPAR or from multiple LPARs.

• If you only access a log stream from one LPAR, you can use a DASD-only log stream or you can use a coupling facility log stream.
• If you access a log stream from more than one LPAR, you need a data sharing environment and a coupling facility log stream.

In a data sharing environment, you must define log streams in a coupling facility to enable logging operations to access the same log stream across all participating logical partitions (LPARs) in the sysplex. Multiple LPARs can generate inserts, updates, and deletes to the same VSAM data set. The target CICS® region can write changes from all LPARs in the correct order and maintain transactional consistency.

For example: CICS updates all source files from one LPAR (LPAR1). The log reader runs on a different LPAR (LPAR2). In this case, you need a coupling facility log stream.

A data sharing environment also requires that all the systems that access the source files share an ICF catalog.

For DASD-only log streams, all applications and the log reader must run on the same image. If you have to apply maintenance, you might incur a replication outage.

For additional guidance about setting up log streams in a coupling facility, which type of log stream to define, and sizing the log streams, see the CICS product documentation about defining the logger environment.

Retention periods

When you define a replication log stream, in addition to determining the type of log stream to define you need to consider factors that affect recovery and the volume of data that is generated.

The RETPD and AUTODELETE parameters that you set when you define a log stream determine how long data is retained and when data can be deleted. Manage the values that you set for these parameters to ensure that the required log replication data is available when replication is started.

You must retain log data long enough that you can recover change data if your replication configuration experiences an outage. For example, if your subscription is stopped for four days and the retention period for the replication log is three days, you may be missing data after restart. If your subscription has been stopped longer than the retention period of your replication log, you must refresh the mappings in the subscription before continuing with change replication.

Log volumes

When you activate replication logging, the types and volume of records written to the log is based on the recovery and replication options that you define. The number of updates and the number of files assigned to a given log stream also impact the volume of data that replication logging generates.

For more information about log record types and the effects of recovery and replication options, see the CICS product documentation.

Reference

Table 1 links to additional information about defining replication log streams.

Tieup record (TUR) processing

CICS TS and CICS VR generate tie up records (TURs) in the replication log. It is helpful to understand how these records are processed and why they are important to VSAM Remote Source.

VSAM Remote Source tracks TURs as they are encountered in the replication log for each subscription. The TURs are held in 31-bit storage in the source server and are used by VSAM Remote Source to associate the ApplID + file name with a VSAM base cluster DSN. The engine replicates data based on VSAM base cluster DSNs, but data changes are logged by using the file name. The TUR is required to relate those values (that is, to know which DSN a change impacts). Each change that is received by the log reader is paired to the TUR for the same ApplID + file name so it can be associated with a DSN to determine if that change must be captured and to associate it with a replication mapping for the subscription.

Every file open results in a TUR with FLJB_TUR_OPEN set. CICS TS writes a replacement TUR about every 30 minutes; however, that activity is tied to the AKPFREQ. In environments where nonrecoverable files are exclusively used and no recoverable resources are being updated, the replacement TUR processing might not be triggered. CICS provided the CFCT transaction with APAR PI97207 for those environments to ensure that the TUR replacement can be created at predictable intervals even in nonrecoverable environments. You can use the CFCT transaction to control the duration between replacement TURs for open files.

Having periodic TUR replacement records while the file is open allows VSAM Remote Source to advance its bookmark tracking for the oldest TUR. One timestamp that is tracked in the bookmark is the oldest TUR timestamp for the subscription log stream. This value can be updated with each UOR because it represents the oldest TUR being tracked at the time that tracking started for the UOR in the source server. It is important to recognize that the times are relative to the timestamp of the log data as it is processed in the capture service of the source server and are not the current time. Without a replacement TUR, the oldest TUR time could become quite old if a file is opened and closed rarely (for example, opened at the beginning of the week and left open) and there is a risk the TUR could even be expired from the replication log without replacement TURs.

Each time a replacement TUR is logged it takes the place of the prior version of the TUR for the file and the newer TUR causes the bookmark tracking for the TUR timestamp to move forward in time.

CICS TS and CICS VR also generate file close records in the replication log. The file close record format is described along with the TUR format. VSAM Remote Source uses the file close records to remove TURs from its collection for the change stream. Once a file close is seen, that file's TUR is no longer considered in the oldest TUR value that is contained in the bookmark.

Figure 1 shows the basic flow of TUR and file close records.

Figure 2 shows the files that are tracked by the source server and what is reflected in the bookmark by using a sequence of TUR and file close records for two files, file A and file B.

The TUR timestamp value from the bookmark is used to determine where the log reader starts scanning the log for TUR records when the change stream is restarted. This process is used to rebuild the TURs that are needed to identify and associate changes with replication mappings within the subscription. Between the oldest TUR timestamp and the restart timestamp, which are both included in the bookmark, only TUR and file close records are processed to rebuild the TUR collection for the subscription.

Properly rebuilding the TUR collection is important to change capture. Any changes that are encountered for a file prior to a TUR are discarded because no TUR exists to relate the file name to the VSAM base cluster DSN. It is important to ensure when adding a new subscription that the TUR is found. You can ensure this by having the source VSAM cluster closed in all address spaces when you activate the replication mapping. VSAM Remote Source helps you ensure that you follow this practice by adjusting the log scan start time to 120 minutes earlier for a new subscription. This step results in additional scanning of the replication log for a newly created subscription, but helps avoid missing data.

For existing subscriptions, adding a new replication mapping is simplified by tracking all TURs that exist in the replication log in the bookmark, whether these are replicated by the subscription or not. Effectively, a list of TURs that are currently "of interest" to the subscription controls what changes are replicated. Another list of TURs is maintained that are currently "not of interest" to the subscription. By tracking all of the TURs, VSAM Remote Source makes it easier to add a VSAM cluster replication mapping to the subscription, because the subscription can already locate the TUR for that VSAM cluster by using information from the bookmark, and on restart the TUR moves from the "not of interest" list to the "of interest" list.

CICS TS and CICS VR also generate start of run (SOR) records.

VSAM Remote Source uses the SOR record to help trim TUR tracking for abnormal cases where a file close record was not written (for example, an abend). When an SOR record is encountered, VSAM Remote Source understands the ApplID value (for example, the CICS region) is reconnecting to the replication log, and assumes that any file that is open for that ApplID was closed even if no file close record reached the replication log when the ApplID previously ended. All files generate a file close record during a normal close or ApplID shutdown; finding open files for an ApplID when an SOR record is processed indicates an abnormal outage of the ApplID occurred. Figure 3 shows the SOR record processing:

In the diagram, the two files are separated into different CICS regions (ApplIDs). FileA in the CICS1 region is open when a serious failure occurs in CICS1, preventing its file close record from being logged. In this scenario, the file is not re-opened when CICS1 is restarted. The FileA TUR is tracked after its open and is updated by the replacement TUR prior to the outage of CICS1. Because FileA is not re-opened, the subscription continues tracking the last FileA TUR because no file close record was logged at the point of failure. To assist in avoiding issues where a file close record might not be written, which leads to tracking an old TUR, VSAM Remote Source uses SOR records that are written when an ApplID reconnects to remove FileA from its TUR collection and no longer uses the file in positioning during restart to rebuild the TUR collection. Essentially, VSAM Remote Source uses the SOR record as an implicit file close for any TURs that are tracked for the ApplID.

If CICS1 and the VSAM Remote Source log reader both fail (for example, loss of the LPAR that hosts both address spaces) the scenario is similar. Whether VSAM Remote Source is active or not, VSAM change activity is recorded in the replication log. When VSAM Remote Source restarts, it scans the log from the oldest TUR that is being tracked. At the point of the CICS1 outage, the oldest TUR is the FileA replacement TUR. VSAM Remote Source finds that TUR record on restart and then reads forward through the replication log and processes the SOR record when CICS1 is restarted. At that point, VSAM Remote Source removes FileA from its TUR collection and updates the subscription's bookmark to begin tracking the FileB replacement TUR as the oldest TUR for the subscription.

If a file is opened infrequently, VSAM Remote Source depends on the replacement TUR to ensure that the log scan start time that is recorded in the bookmark does not become too old. On restarts, some log data is read again in order to rebuild the TUR collection, but restarts are relatively infrequent and you can control how much data is reprocessed using the CFCT transaction configuration. However, the tradeoff for having shorter time frames of data to be read again is additional TURs in the replication log, because your CFCT transaction needs to execute more frequently. You would need to run the CFCT transaction in all CICS regions where you have files open (for example, FOR/DORs and possibly AORs when RLS files are opened locally).

All TUR and file close records are written to the replication log whether VSAM Remote Source is currently reading the log or not. When VSAM Remote Source is reading the log, it might be reading latent data from the replication log. The context for the TUR collection and bookmark values is the point where the source server is reading the log.

VSAM units of recovery

Data Replication products for VSAM use the following Unit of Recovery (UOR) Identifiers (Ids). A UOR Id defines a group of changes that are applied together. For recoverable data sources (for example recoverable VSAM under the control of a transaction manager like CICS TS or DFSMSTVS) this is usually a group of changes representing a source commit scope.

The UOR Id is always 32 bytes. The internal format might include binary values to create a unique UOR Id. When necessary, the replication products convert binary data to hex in order to create a printable UOR Id. The maximum printed UOR Id is therefore 64-bytes, but some information in the UOR Id is already printable character data an effort is made to retain existing character values and avoid simply converting each byte to hex to maintain the readability of the UOR Id when that's possible.

Each UOR Id contains information that is related to its source in the first 24 bytes. Some UOR Ids might re-use this information creating multiple UORs with the same information in the first 24 bytes. This occurs often for VSAM. All UOR Ids have a uniqueness STCK added after the first 24 bytes to create unique groups and this ensures when the data in the first 24 bytes is repeated the groups are still unique. Any UOR that was already unique in the first 24 bytes still gets a uniqueness STCK added. The uniqueness STCK value is taken from the first log record that causes tracking of the UOR Id. The log record time that is used might be adjusted by the log reader to ensure uniqueness. Note: ZUORs are an exception to the uniqueness STCK statements since these are already uniquely generated empty groups to impact the subscription bookmark.

1. Recoverable CICS TS workload:
----+----1----+----2----+----3--
ttttttttnnnnaaaaaaaa    ssssssss

Printed example:
0001144CDEMDCI57        DB2B6B1D05150B4A

2. Recoverable CICS TS ESDS inserts:
----+----1----+----2----+----3--
ttttttttnnnnaaaaaaaaESINssssssss

Printed example:
0001144CDEMDCI57    ESINDB2B6B1D05150B4A

3. Non-recoverable CICS TS workload w/o DSNLUORTIMEOUT:
----+----1----+----2----+----3--
ttttttttnnnnaaaaaaaaNRCVssssssss
The first non-recoverable change of the group sets the UOR Id values.  When subsequent non-recoverable changes occur with an in-flight non-recoverable UOR they are included in the existing group until the group is ended and then a new group begins with its first non-recoverable change.  The VSAM replication products determine when the grouping ends for non-recoverable data.

Printed example:
0000958CDEMECI57    NRCVDB27651D51F9972C

4. Non-recoverable CICS TS workload w/  DSNLUORTIMEOUT
----+----1----+----2----+----3--
rrrrrrrrDSNLCICS TS NRCVssssssss

Printed example:
00000088DSNLCICS TS NRCVDB2B88454B2E785A

5. CICS VR workloads
----+----1----+----2----+----3--
jjjjjjjjppppaaaaaaaaNRCVssssssss

Printed example:
@BTCTST1VSMK.RPNEW02NRCVDB2B6ED4B6BCEC2B

6. DFSMSTVS
----+----1----+----2----+----3--
uuuuuuuuuuuuaaaaaaaauuuussssssss

Printed example:
db2b77907ebb9f4000000540.GWTV00201020000db2b779098d16421

7. DFSMSTVS ESDS inserts
----+----1----+----2----+----3--
uuuuuuuuuuuuaaaaaaaauuuussssssss

Printed example:
db2b77907ec313d000000a02eGWTV00101010000db2b77908783f66a

8. Non-recoverable DFSMSRLS workload w/o DSNLUORTIMEOUT
----+----1----+----2----+----3--
jjjjjjjjppppDFSMSRLSNRCVssssssss
The first non-recoverable change of the group sets the UOR Id values.  When subsequent non-recoverable RLS changes occur with an in-flight non-recoverable UOR they are included in the existing group until the group is ended and then a new group begins with its first non-recoverable change.  The VSAM replication products determine when the grouping ends for non-recoverable data.

Printed example:
@RLSESD2RLSEDFSMSRLSNRCVDB27696F24A27A20

9. Non-recoverable DFSMSRLS workload w/  DSNLUORTIMEOUT
----+----1----+----2----+----3--
rrrrrrrrDSNLDFSMSRLSNRCVssssssss

Printed example:
0000010FDSNLDFSMSRLSNRCVDB2B73A4F98FC968

10. VSAM remote source - Refresh UORs
----+----1----+----2----+----3--
                      RFSHssssssss


Printed example:
                      RFSHDB2B211266C90000

11. ZUOR UORs are generated by VSAM file close and tieups.
Temporary, internal UORs that result in ZUOR format (below) UORs replicated to advance the bookmark.  This should only be visible in tracing.
----+----1----+----2----+----3--
                      ZUORssssssss

Printed example:
                      ZUORDB2B2159F3D60000

12. ZUOR UORs to advance restart during inactivity
----+----1----+----2----+----3--
ZUORxxxxxxxxxxxxxxxx

Printed example:
ZUORdb27641a86e58c2a

Legend:
a- Appl Id
   For CICS TS, region where the transaction occurred.
   For CICS VR, unique instance for an LPAR from D SMS,CICSVR,ALL report. The first byte may be unprintable and might be displayed as '.' to ensure a printable character
   For DFSMSTVS, unique instance for an LPAR (TVSNAME) from D SMS,TRANVSAM,ALL report.  The first byte of the TVSNAME is 0x05 in log records, but will be displayed as character '.' except for ESDS inserts that might display as character 'e' (EBCDIC 0x85) or à (ASCII 0x85) in the UOR Id.
j- jobname
n- CICS transaction ID
p- first four bytes of the step name
r- hex, unique, replication object ID (ROID) for the data set
s- uniqueness STCK
t- character conversion of packed decimal task number; last character converted is the sign nibble 'C'
u- RRS URID (hex) - split with first 12 bytes before the Appl Id and last 4 bytes after the Appl Id -- when displayed will be the hex values
x- Converted character display of the hex STCK used to advance the subscription bookmark