Handling of invalid data

Invalid data refers to source data that does not agree with the logical table mapping that defines the source data as a relational table for refresh and replication purposes.

An example would be if you defined a source data field as a decimal column in the logical table mapping, but some source records have EBCDIC spaces (0x40 characters) in the positions that are defined by the decimal column. IMS data is not typed and it is common for incorrect table mappings or records that do not follow the normal form to cause errors because of invalid data.

When the logical table references a file that contains multiple record definitions, you can use a view to reduce detection of invalid data. See Mapping tables and views for redefined data for instructions on how to deal with redefined data.

When invalid data is identified during refresh and replication processing, the IMS Remote Source engine repairs and reports on data conflicts to help keep subscriptions replicating.

The following parameters control how invalid data is handled:

CSDATAVALIDATEAC: Controls whether extra data type validation occurs and identifies the action to take if invalid data is found. By default (2 = REPAIR REPORT), the IMS Remote Source engine replaces invalid data with the repaired values. For details, see CSDATAVALIDATEAC.
CSREPORTLOGCOUNT: Prevents excessive logging to the IMS Remote Source log file when many badly formed data records are processed. By default, the engine limits the number of log records that report invalid data to 500. For more information, see CSREPORTLOGCOUNT

Using these defaults allows you to keep most your data replicating while providing ways to identify invalid data both by using the source 0x002f0001 diagnostic messages and at your target system by using the repaired value.

When invalid data is detected and the repair option is enabled, by default the invalid data is replaced with negative 9 characters (-9). For many columns the use of a -9 value for the length of the column can be regarded as a sentinel value because it does not represent an actual value that can be stored in the column by the application.

Some consuming applications have difficulty with this sentinel approach and prefer that invalid numeric data be replaced by zeros or reported as being NULL. Use the following procedure to replace sentinel values with zeros when invalid numeric data is encountered:

Attach to the container by issuing the following command:
```
docker attach ClassicCDCIMS
```

From within the container, issue the following command:

set,config,service=global,COLUMNREPAIREXIT=ceccrep0

The following message is displayed:

[Timestamp] CAC00200I SET,CONFIG,SERVICE=GLOBAL,COLUMNREPAIREXIT=CECCREP0
[Timestamp] CAC00242I GLOBAL CONFIGURATION RECORD UPDATED SUCCESSFULLY.

The new setting takes effect the next time that you start replication for a subscription. The following message is displayed and written to the event log:

[Timestamp] CECX0001I Column repair exit enabled using CECCREP0 for subscription Subscription-Name.

Use the following procedure to treat a column contents as NULL when invalid numeric data is encountered:

Attach to the container by issuing the following command:
```
docker attach ClassicCDCIMS
```

From within the container, issue the following command if you want the column reported as being NULL:

set,config,service=global,COLUMNREPAIREXIT=ceccrepn

The following messages are displayed:

[Timestamp] CAC00200I SET,CONFIG,SERVICE=GLOBAL,COLUMNREPAIREXIT=CECCREPN
[Timestamp] CAC00242I GLOBAL CONFIGURATION RECORD UPDATED SUCCESSFULLY.

The new setting takes effect the next time you start replication for a subscription. The following message is displayed and written to the event log:

[Timestamp] CECX0001I Column repair exit enabled using CECCREPN for subscription Subscription-Name.