Managing data synchronization failures

Changes to objects on the source node are tracked by Migrate While Active. The full details of the tracking lists and the object types that correspond to those lists is found in Object types and migration.

Migrate While Active – Objects that need to be synchronized:

IFS_MIGRATION_LIST view

The IFS_MIGRATION_LIST view returns information from the integrated file system tracking list (ITL).

LIBRARY_MIGRATION_LIST view

The LIBRARY_MIGRATION_LIST view returns information from the library tracking list (LTL) and object tracking list (OTL).

The Migrate While Active data synchronization jobs examine the tracking lists looking for objects that need to be synchronized to the copy node. If an object is successfully synchronized, the entry on the tracking list is deleted, and a row is added (or updated) on the historical list of successfully migrated objects.

If a failure is encountered during the synchronization of an object, the object will remain listed on the tracking list and the FAILURE_MESSAGE column will indicate the type of failure that was encountered. The FAILURE_JOB_NAME column will also be updated, indicating the qualified job name of the synchronization job that encountered the failure.

If the object is not changed again, the migration will be re-attempted as part of final synchronization.

If the object is changed again, an additional row will be added to the tracking list causing another attempt to synchronize the object. If this additional attempt to synchronize is successful, both rows will be removed from the tracking list and a new row will be added to the historical list of successfully synchronized objects. If the object had been previously synchronized, the SUCCESSFUL_MIGRATION_COUNT column of the existing row for the object will be updated.

The following example demonstrates how Migrate While Active automatically recovers from synchronization failures.

Figure 1 shows the creation of libraries and save files within the libraries. It also shows how the LIBRARY_MIGRATION_LIST has tracked that the objects need to be synchronized. The TRACKING_LIST column shows that for this view, some entries are sourced from the Object Tracking List (OTL) and others are sourced from the Library Tracking List (LTL).We can also see that no attempt has been made to migrate the objects. Observe that the tracking list also contains entries for objects in a different library.

The three Migrate While Active data synchronization jobs (Mirrored Objects – QMRDBESYNC, Non-Mirrored Objects – QMRDBNSYNC, and IFS Objects – QMRDBISYNC) execute independently of each other. What this means is that if an object in one tracking list has a dependency on an object that appears in a different tracking list, it is possible that the migration processing could occur in an order that causes a failure condition to be encountered.

In this example, the TOYSTORE1-5 libraries are Db2 Mirror replication eligible, and are therefore processed by the Mirrored Objects – QMRDBESYNC job. The example scenario has a large number of rows being migrated to the copy node for files in a different library. This delays synchronization of the create library operations.

The SAVEIT *FILE (save files) are processed by the Non-Mirrored Objects – QMRDBNSYNC job.

In this example, when the QMRDBNSYNC job attempts to save and restore the save files to the copy node, the restores fail because the libraries do not yet exist on the copy node. This failure is shown in the library migration list in Figure 2.

Figure 3 shows that the resynchronization job log includes the CPF3781 failure message, and the reason for the failure.

Next we are going to cause the save files to try synchronizing a second time. In Figure 4, you can see that the Change Object Description (CHGOBJD) command was used, and the resulting effect to the tracking list. Now every SAVEIT object has two rows on the tracking list, one that was the failure row, and an additional row for the new change which will prompt a future instance of the QMRDBNSYNC job to retry the object synchronization.

Upon retrying the save and restore, the synchronization was successful. This resulted in all of the rows for the now-synchronized objects in the tracking list to be deleted. Figure 5 shows that all the rows for the TOYSTORE libraries are no longer on the tracking list.

In Figure 6, a query of the MIGRATED_LIBRARY view shows the objects that were migrated successfully, how many times the object was migrated, and related detail. From the results, you can see that the libraries and save files had been migrated several times before based on the SUCCESSFUL_MIGRATION_COUNT and the INITIAL_TRACKING_TIME columns.

Migrate While Active will attempt to recover from data synchronization failures if the failed object is changed again. If the object is not changed again, the final data synchronization processing will retry the synchronization.

One reason to cause a retry of objects that failed to migrate is to reduce the amount of data synchronization work that will be needed during final synchronization.

--
-- Prompt failed entries on the LTL to retry synchronization
--
SELECT QSYS2.QCMDEXC('QSYS/CHGOBJD OBJ(' CONCAT LIBRARY_NAME CONCAT '/'
                                         CONCAT OBJECT_NAME CONCAT ') OBJTYPE(' 
                                         CONCAT OBJECT_TYPE CONCAT ') USECOUNT(*RESET)'
    ) AS CHGOBJD_RESULT
    FROM QSYS2.LIBRARY_MIGRATION_LIST
    WHERE FAILURE_MESSAGE IS NOT NULL;

Although the explanations and examples list above were focused on libraries and objects within libraries, the same concepts exist for Migrate While Active and the integrated file system.

--
-- Prompt failed entries on the ITL to retry synchronization
--
SELECT PATH_NAME, QSYS2.QCMDEXC(
           'QSYS/QSH CMD(''touch ' CONCAT PATH_NAME CONCAT ''')')
    FROM QSYS2.IFS_MIGRATION_LIST
    WHERE FAILURE_MESSAGE IS NOT NULL;