Capture cache and flow control overview

Each subscription has its own cache that tracks changes that made in-flight and committed.

Cache creation

The cache is created when you start replication for a subscription and continues long after you stop replication for the subscription. Keeping the cache and continuing to capture changes after replication ends for a subscription enables faster resumption of operations in conditions where replication is restarted before too many changes are captured.

In IMS and VSAM remote source systems, refresh data is stored in the cache before sent to the target. The cache is created when refresh operations for a table start and removed after the last refresh row is sent to the target.

Cache deletion

The contents of an existing cache are deleted when one of the following conditions is met:

Replication is active and a source or target reported an error condition. Cache deletion does not occur if replication ends due to loss of the TCP/IP connection with the target.
Replication is not active for a subscription and the cache reached 80% of capacity.
Replication is not active for a subscription and an error condition is reported.
Replication is not active and a STOP,REPL modify command is issued with the CAPTURE option.
Replication is not active, and a STOP REPLICATION batch command, processor command is processed with the CAPTURE option.
Replication is being restarted and the restart point no longer exists in the cache.
Replication is restarted, and the subscription is changed. This change might include adding or removing a replication mapping, and modifying the log positions of an existing replication mapping.
Replication is being restarted and a SETLOGPOS modify command or SETLOGPOSITION batch command, processor command was issued for the subscription.
When a refresh ends for a table in an IMS or VSAM remote source environment.

Flow control

The capture service uses a flow control model to prevent the subscription capture cache from filling up. When a subscription's cache reaches 80% capacity, the capture service stops caching subsequent updates until it reaches less than 60% capacity. The log reader is also alerted to stop providing data at the 80% level and receives a "resume" notice when the cache reaches 60%.

Subscriptions that reference VSAM datasets have independent change Streams. Ensuring that when flow control is initiated for one subscription, it does not affect the functioning of other subscriptions. Subscriptions that reference IMS databases use a primary/secondary stream model. Which might affect the operation of other subscriptions when flow control is activated for one of the subscribers that are linked to a primary or secondary change stream.

Deleting data from a cache

The capture service uses an optimistic approach to delete data from the subscription capture cache. The assumption is that data that is sent to the target will be successfully applied, thus it can be deleted from the cache quickly after it is sent to the target. Two thresholds are there for determining when to delete data from the subscription capture cache:

After 16 megabytes of application data is sent to the target.
After 2 megabytes of unit-of-recovery tracking data is sent to the target.

Format of data written to the cache

The size and format of application data varies based on the target:

For CDC targets, data is cached based on table mappings that are associated with modified IMS SEGM or VSAM files. Each update includes metadata about the modification, followed by encoding flags and data for each column.
IIDR for IMS targets caches data from a subset of the source IMS data capture log record, together with extra metadata about the change.
For IIDR for VSAM targets, the data that is cached is the replication log data for a change and some additional change metadata.
For a WebSphere MQ target, the data that is cached is the comma-separated data record based on the table mappings with the modified IMS SEGM or VSAM files. Each change includes metadata about the modification and the CSV data for each column.
For an IMS or VSAM remote source refresh operation, a single group refresh message is created with as many rows as possible to fit inside the maximum CDC TCP/IP buffer (64,000 bytes). Each group refresh message is handled as an individual unit of recovery.

Some applications can produce a significant number of single operation units-of-recoveries and ensure that they are regularly deleted. The structure used to track each unit-of-recovery can vary, but it is always greater than 400 bits, resulting in data deletion at around 4800 unit-recovery intervals.

Cache deployment options

A capture cache consists of two separate components:

Message store that caches individual insert, update, delete, and group refresh messages.
Committed transaction store that orders and tracks committed units-of-recovery.

There are different implementation options for both message and committed transaction stores.

The message store supports two models:

File abstraction

Individual changes in a unit-of-recovery are compiled into a 64-kilobyte buffer, which is keyed by a unique unit-of-recovery identifier. There are two different components; data and index. One of these messages can store a maximum of two gigabytes. On z/OS, these can be deployed as files that are mapped to 64-bit shared memory objects or 64-bit data spaces. On Linux, these are processed as files.
Memory map

The changes for a unit-of-recovery are kept in a link list that is accessed by using a hashing algorithm. A single message can span many 4-kilobyte "pages" addressed by the hashing algorithm. The map memory exists outside of the main message pool. Messages are transferred directly to and from the main message pool to external storage.

A single object consists of a control header, a space map, and the actual data pages. On z/OS, this is implemented as a 64-bit shared memory object that is accessible through 64-instructions, with a maximum cache capacity of 35 gigabytes. Therefore, on Linux, this is 32-bit memory that is allocated by using malloc, the maximum capacity for all subscribers is 4 gigabytes, minus memory allocated to the message pool and system working memory.

Also, there are two implementation options for the transaction store:

File abstraction

There is a separate data and index component that tracks each committed unit-of-recovery that is kept in the cache. The data component stores the unit-of-recovery metadata, which is keyed by the commit timestamp.

On z/OS, these can be implemented as 64-bit shared memory objects or 64-bit system data spaces. On Linux, these are processed as files.

However, these have a maximum capacity of 2 gigabytes. The real size is always smaller due to message store limitations.

Binary search list

Unit-of-recovery metadata is added to the end of the linked list of messages stored in the message store. An extra unit-of-recovery tracking structure is allocated from message pool memory and kept in a binary search list that is keyed by the commit timestamp.

Because the contents of the binary search list are volatile, each subscription has its own subpool. The initial and secondary allocation sizes are also 2 MB, providing enough memory to track 32,768 committed units of recovery. To prevent memory fragmentation, the binary search list memory is allocated outside of the message pool and extends in increments that allow for an addition of 32767 units-of-recovery in the list. The initial allocation and each extension make a malloc request for 128 K of memory.

Calculating cache usage

Flow control cache use percentages are computed by using the amount of message store RAM that is in use are determined by the block or message size. When using a message store based on the file abstraction model, each megabyte of cache space can monitor a maximum of 16 units of recovery. Whereas a memory map version can store changes for up to 256 units of recovery per megabyte of cache space.

Default cache sizes

The default cache size for IMS subscriptions is 2 gigabytes, while VSAM subscriptions have a cache size of 256 megabytes. The difference in default cache size is due to the stream models used for reading and capturing changes.

Change stream models

For subscriptions that reference VSAM files, each subscription has its own subscription change stream that only contains updates to the files referred to by active replication objects. Each subscription stream reads the z/OS system log stream for these files and sends the selected data to the capture service. When flow control is enabled for one of these subscribers, the processing of other subscriptions is unaffected. Therefore, there is no need to support large cache sizes because data can always be received from the z/OS system log stream.

IMS referenced subscriptions use bigger cache sizes due to the way IMS changes are captured. IMS changes are written down in a collection of web logs, which are thereafter archived. Many sites use data sharing environments. In the environments, each member generates their own set of logs that must be read and combined before the changes are submitted to the capture service for processing. IMS data captures log records are compressed, and a single logical log record can be converted into several physical log records. This makes filtering (only sending changes relevant to a specific subscription) more difficult and costlier than filing VSAM changes.

To limit the number of times an IMS log is read, the capture service uses a primary/secondary change stream paradigm to record IMS changes. All data captures log records are sent to the capture service (no filtering is done), and if a secondary stream exists, it receives priority in log-read operations (by the IMS log reader) and capture service processing over primary stream processing.

When replication starts for a subscription that refers to IMS databases, the restart point that is retrieved from the target is compared to the last change received on the primary change stream. If the restart point is less than the last change received, a second change must be made. By default, only three secondary change streams can exist concurrently. Once this limit is reached, stream assignment (to new secondary change streams) is delayed until one of the existing secondary change streams is no longer exists. In this case, the subscription waiting for a secondary change stream is reported in a replication continuous state, but no changes are captured. When you initially begin replication for multiple subscriptions or after a failure, you can use the capture service STREAMMGRDELAY configuration parameter to try to get all of the subscriptions assigned to the primary or secondary change streams. The STREAMMGRDELAY sets the time interval that the capture service waits before assigning streams. So, if you start replication for many subscribers soon, they are likely to be allocated to the same change stream. The default STREAMMGRDELAY setting is zero (no delay), but most users set it to 30 seconds or one minute.

Using a larger cache for subscriptions that reference IMS is intended to reduce the number of times flow control is activated. Due to a cache filling to 80% capacity, log reading stops, which might have an impact on capture operations for other subscriptions that are assigned to the same change stream. If subscriptions have different change rates and volumes, you might use a memory map and increase the cache's maximum size. It reduces the effect of flow control on other subscriptions.

Resource usage

The maximum size of the message and transaction store caches is specified by using a single value. On z/OS, when these caches use 64-bit memory, a page frame range is created. But, physical memory is not instantiated until the memory is physically referenced. At that point, the associated page frame is in virtual memory. Once allocated, a page frame is not released until the cache is deleted.

On Linux®, the amount of storage that is used by temporary files are depends on how much data is written to each file.

Defining the size of the cache

The cache size for IIDR for IMS or IIDR for VSAM subscriptions can be configured in Classic Data Architect and set by using CHCCLP. The SET,REPL command, with the CAPTURECACHE parameter, can be used to specify the cache size for any type of subscription. The CAPTURECACHE phrase must be surrounded by single quotation marks and specified in uppercase, according to the command syntax. The value that is provided is in megabytes. For example, SET,REPL,SUBSCR=TEST,'CAPTURECACHE=1024' sets the capture cache for subscription TEST to 1 megabyte.

When a cache is created for a subscription, two level 4 trace messages are generated to the diagnostic log identifying the cache's size. ROCCreate function issues these messages. The first message indicates the size of the message store component, while the second contains information about the transaction store.

Specifying cache deployment options

The Capture Service CACHEINDICATOR configuration parameter specifies the type of message and transaction storage configuration that you want to use. Memory-based caches (memory map and X-list transaction stores) are more efficient (use less CPU) than the other choices. But, they aren't always practical depending on the environment and the number of subscriptions.

The following table describes the various options available in z/OS and recommends that which ones are most suited to various scenarios:

Table 1. z/OS CACHEINDICATOR options
Option	Message Store	Transaction Store	Recommendation
0	64-bit shared memory object that uses a file abstraction	64-bit shared memory object that uses a file abstraction	Not recommended
1	64-bit system data space that uses a file abstraction	64-bit system data space that uses a file abstraction	Not recommended
2	64-bit shared memory object that uses a message map	64-bit shared memory object that uses a file abstraction	Default configuration setting. Used when there are many subscriptions, employing a binary search list transaction storage might be impractical due to message pool requirements.
3	64-bit shared memory object that uses a message map	Binary search list with unit-of-recovery metadata stored in the message store	Used when there are a few subscriptions and sufficient message pool space is available.

The following table describes the options available in Linux® and recommends that which ones are best suited to various scenarios:

Table 2. Linux® CACHEINDICATOR options
Option	Message Store	Transaction Store	Recommendation
0 or 1	File abstraction that uses temporary files	File abstraction that uses temporary files	Default
2	Message map that uses external memory	File abstraction that uses temporary files	Used if there are a few subscriptions and sufficient memory available.
3	Message map that uses external memory	Binary search list with unit-of-recovery metadata stored in the message store	Used when there are a few subscriptions and plenty of external and message pool RAM. A 512 MB message pool, for example, can support up to ten subscriptions each when using a 256 MB memory map.

If you update the CACHEINDICATOR, the new setting takes effect the next time you start replication for a subscription with no existing capture cache. To change the setting for an actively replicating subscription, first stop replication for that subscription, then issue the STOP,REPL command with the CAPTURE option. Finally, restart replication for that subscription.