Understanding the Federated Data Repository process indexing

The cornerstone of the Federated Data Repository indexing is the queue, which is populated by the Process Engine. The indexing queue is shared among all the nodes of the Business Automation Workflow cluster.

How the Federated Data Repository indexing works
Administering the Federated Data Repository indexing
Monitoring the Federated Data Repository indexing

How the Federated Data Repository indexing works

An entry is inserted in the indexing queue each time the Business Automation Workflow Process Engine creates, updates or deletes a process instance, or creates or updates a process task that is expected to be indexed.

When you start Business Automation Workflow:

An indexer thread is created. You can have more than one indexing thread per node by setting the <fdr-index-threads> property in 100Custom.xml. All the indexer threads that are running in your cluster work without conflicting each other. Consequently, you can use scalability options such as adding new nodes or increasing the number of threads per node to improve the indexing performance.
A maintenance thread is started to handle and coordinate the processing of the various maintenance operations.

The indexer thread starts an indexing cycle by reading up to 1000 entries from the queue. The 1000 batch size is configurable by setting the <fdr-index-batch-size> property in 100Custom.xml.

For all the process instances and tasks referenced in the batch of entries, the indexer thread queries the database to retrieve the data required to build the JSON documents to insert or update in the Federated Data Repository index. A request is sent to the Federated Data Repository REST API to create, update, or delete the documents, as needed.

For the process instances and tasks that could be successfully created, updated, or deleted in the Federated Data Repository:

A row is created, updated or deleted in FED_INDEX_INSTANCE for each process instance and in FED_INDEX_TASK for each task. These rows hold a version number of the corresponding indexed document, which can be later compared with the INDEX_VERSION column of LSW_BPD_INSTANCE and LSW_TASK database tables, respectively, to verify that the indexed data is in sync with the database, or to perform any required reconciliation to bring the index back in sync.
Finally, the queue entries are deleted from the indexing queue.

For each individual documents which failed to be created, updated, or deleted from the federated data repository:

The failure counter for the corresponding queue entry is incremented.
The queue entry is re-attempted on the next indexer cycle, unless the failures count reaches the threshold defined by the <fdr-index-failure-threshold> property in 100Custom.xml. The default value of the threshold is set to 20.
The indexing queue entries that reach the threshold are no longer reprocessed, and remain in the queue until the ‘retry-failures’ maintenance operation is executed. To determine if the queue contain such rows, you can execute the /ops/fdr/bpd/info REST API.

Next, an audit entry is logged in the FED_INDEX_AUDIT table with details about the indexing cycle, such as the start and end time, the number of queue entries being processed, the number of documents indexed, deleted, and failed, and so on. Finally, the current cycle completes and a new cycle starts.

Administering the Federated Data Repository indexing

The Federated Data Repository indexing comes with a set of maintenance operations. Some of them are executed automatically, others are meant to be executed by an administrator by using the REST API for Federated Data Repository indexing.

The REST API enables you to:

Start a maintenance operation: POST /ops/fdr/bpd/maintenanceOperations/{maintenance-operation-type}
Get the latest status of a maintenance operation: GET /ops/fdr/bpd/maintenanceOperations/{maintenance operation type}
Abort, suspend, or resume an on-going maintenance operation: PUT /ops/fdr/bpd/maintenanceOperations/{maintenance operation type}
Get details about the latest maintenance operation runs: GET /ops/fdr/bpd/maintenanceOperations

On each running Business Automation Workflow node, there is a running maintenance daemon that is responsible for handling and coordinating the processing of the various maintenance operations. In a clustered environment, out of all the running nodes, only one maintenance daemon is active. The other maintenance daemons remain idle until the active daemon is stopped. When the active daemon is stopped or becomes unresponsive, another running maintenance daemon becomes active, while all the other daemons remain idle.

Each maintenance operation type is documented in the following subsections.

‘create-index’
‘rebuild’
‘house-keeping’
‘retry-failures’
Failover support
‘reconcile’
‘sync-deletions’

‘create-index’ maintenance operation: The maintenance operation is executed when the Federated Data Repository index is not created yet. It creates the index. If indexed data already exists and the Federated Data Repository index is deleted, the ‘create-index’ recreates the index. Pre-existing rows from FED_INDEX_INSTANCE and FED_INDEX_TASK are deleted when the index is recreated.; The ‘create-index’ maintenance operation cannot be triggered manually, it is only triggered automatically by the maintenance daemon.; To get the status and details about the latest run of the ‘create-index’ maintenance operation, call GET /ops/fdr/bpd/maintenanceOperations/create-index.
To rebuild the index from scratch, delete it using the Elasticsearch or OpenSearch REST API. The index is recreated and repopulated automatically without having to restart your Business Automation Workflow nodes.; Back to top

‘rebuild’ maintenance operation: The ‘rebuild’ maintenance parses the database for process instances, human tasks, and external activities that have documents in the Federated Data Repository. An entry is added to the indexing queue for each process instance, human task, or external activity so that the indexing thread creates or updates the corresponding documents into the Federated Data Repository index.; This maintenance operation is triggered automatically immediately after the index is created by the ‘create-index’ maintenance operation.; To get the status and details about the latest run of the ‘rebuild’ maintenance operation, call GET /ops/fdr/bpd/maintenanceOperations/rebuild.; If problems occur with the Federated Data Repository index, you might need to rebuild it. To rebuild the index, manually trigger the ‘rebuild’ maintenance operation by calling POST /ops/fdr/bpd/maintenanceOperations/rebuild. This will refresh all the existing documents and create the missing documents, if any. During the rebuild, the index data remains searchable.; Back to top

‘house-keeping’ maintenance operation

The indexing records rows for each indexing cycle, and the maintenance daemon also creates records for each maintenance operation run. To prevent the database from growing indefinitely, some housekeeping must be performed on a regular basis. By default, the ‘house-keeping’ maintenance operation is triggered every hour to delete all the audit rows, except:

The indexing cycle audit rows that are by default less than 7 days old. The default value can be customized by using <fdr-index-house-keeping-audit-max-age>, expressed in number of minutes.
The maintenance operation records that are related to the latest 50 executions of each kind. This value can be customized by using <fdr-index-house-keeping-maintenance-log-size>, expressed in number of minutes.

You can change the time interval between two ‘house-keeping’ executions by setting the <fed-index-house-keeping-interval> property in 100Custom.xml. This property is set by default to ‘60’ minutes.

To disable the automatic scheduling of the ‘house-keeping’ maintenance operation, set <fed-index-house-keeping-interval> to a negative value. In such case, ‘house-keeping’ is executed on a regular basis by calling POST /ops/fdr/bpd/maintenanceOperations/house-keeping.

To get the status and details about the latest run of the ‘house-keeping’ maintenance operation, call GET /ops/fdr/bpd/maintenanceOperations/house-keeping.

‘retry-failures’ maintenance operation

Each entry in the indexing queue has a failure counter that is initially set to 0. When an entry is processed, if the indexing in the Federated Data Repository of the corresponding document fails, the counter is incremented, and an attempt is made to reprocess the queue entry in the next indexing cycle. After 20 successive failures, the entry remains in the indexing queue and will be ignored by the indexer.

To find out if such situations might occur, you can check your logs and/or regularly query the GET /ops/fdr/bpd/info API . If this API returns a positive number under indexingQueue.failedCount, this indicates that some entries in the indexing queue have reached the failure threshold. For example, this can occur if the indexing of the corresponding document in the Federated Data Repository index is rejected by Elasticsearch or OpenSearch because it exceeds the configured total fields limit. If you find evidence in the Business Automation Workflow logs that the problem is related to the total fields limit, you can fix the issue by configuring Elasticearch or OpenSearch with a higher Federated Data Repository index fields limit.

If case the root cause of the indexing failure is not clearly identified, contact the IBM® support. After the root cause is fixed (for example, by increasing the total fields limit), you must reset the failure count of the queue entries so that they are processed again by the indexer. To do this, call POST /ops/fdr/bpd/maintenanceOperations/retry-failures.

To get the status and details about the latest run of the ‘retry-failures’ maintenance operation, call GET /ops/fdr/bpd/maintenanceOperations/retry-failures.

Failover support

When you administer your environment to support failover recovery, two distinct topologies are maintained, each on a dedicated data center:

An active topology, where the Federated Data Repository indexing is enabled.
A passive topology, where the Federated Data Repository indexing is disabled.

Automatic database replication must also be in place so that changes to the active site database are instantly replicated on the passive site database. To enable the database replication between the two topologies, refer to your database documentation.

The procedure you must apply to provide a failover recovery consists of:

On the active site, regularly:
- Update the backup state doc by calling PUT /ops/fdr/bpd/backupState, then
- Take snapshots of the Federated Data Repository index on the active site to restore it on the Federated Data Repository of the passive site.
When the active site fails or crashes, you must then:
- Ensure that Federated Data Repository indexing is disabled or not running anymore on the failing topology, which becomes the new “passive” topology, then
- Enable the Federated Data Repository indexing on the passive topology, which becomes the new “active” topology, and
- Once the Federated Data Repository indexing is enabled and the system is running, call the POST /ops/fdr/bpd/maintenanceOperations/restore API on the new “active” site.

Note: After switching to the failover site, the passive topology becomes active and the crashed topology (formerly active) is now considered the passive topology. Consequently, you must also adapt the database replication as needed so that the replication goes from the ‘new’ active topology database to the ‘new’ passive topology database.

‘restore’ maintenance operation: When you execute the ‘restore’ maintenance operation by calling POST /ops/fdr/bpd/maintenanceOperations/restore, theindexing maintenance daemon retrieves the backup state document that contains the latest indexed time of the process instance documents and task documents, respectively, by the time the backup state document was last updated. With this information, the maintenance daemon parses the FED_INDEX_INSTANCE and FED_INDEX_TASK tables to identify all the rows that were updated since these times, and for each of the rows it creates entries in the indexing queue for all the changes that were performed since the time the backup state document was last updated. The indexer daemon will then update the federated data repository index with the latest changes.; With the failover recovery procedure, when you switch to the failover site and then call the ‘restore’ maintenance operation, only the changes that were performed since the latest snapshot of the Federated Data Repository index will be reprocessed before the index gets back in sync. Consequently, the more frequently you update the backup state document, take a snapshot of the active topology, and restore it on the passive topology, the less time it will take for the index to get back in sync upon failover situation.; To get the status and details about the latest run of the ‘restore’ maintenance operation, call GET /ops/fdr/bpd/maintenanceOperations/restore.
‘post-backup’ maintenance operation: As part of this procedure, you regularly call the PUT /ops/fdr/bpd/backupState REST API on the active site to update the document that stores the time of the latest indexed time of the process instance documents and task documents, respectively.; If some tasks or process instances documents are deleted after the backup state document was updated, the Federated Data Repository indexer keeps track of these deletions internally, so that when switching to the failover size and synchronizing the index with the ‘restore’ maintenance operation, the same deletions are also performed on the Federated Data Repository index of the failover site. Tracking these deletions is done by keeping records in the FED_INDEX_INSTANCE and FED_INDEX_TASK tables.; When the backup state document is updated or deleted, the ‘post-backup’ maintenance operation is automatically triggered to remove rows related to previous deletions that are no longer needed. Not deleting these rows would cause the FED_INDEX_INSTANCE and FED_INDEX_TASK tables to grow indefinitely.
The ‘post-backup’ maintenance is purely internal and cannot be executed by calling POST /ops/fdr/bpd/maintenanceOperations/post-backup.; To get the status and details about the latest run of the ‘post-backup' maintenance operation, call GET /ops/fdr/bpd/maintenanceOperations/restore.

‘reconcile’ maintenance operation: When indexing into the Federated Data Repository index, a row is created, updated, or deleted in FED_INDEX_INSTANCE for each process instance and in FED_INDEX_TASK for each task. Such rows have a column that contains the version number of the corresponding indexed document. This version can later be compared with the INDEX_VERSION column of LSW_BPD_INSTANCE and LSW_TASK database tables, respectively, to verify that the indexed data is in sync with the database, or to perform any required reconciliation that brings the index back in sync.; The ‘reconcile’ maintenance operation should not be executed, as the federated data repository index should not become out-of-sync, nevertheless, it might be a useful problem solving tool in unexpected situations.; To execute the ‘reconcile’ maintenance operation, call POST /ops/fdr/bpd/maintenances/reconcile.; Back to top

‘sync-deletions’ maintenance operation

When a process instance is deleted, the Process Engine adds an entry for the corresponding process instance document to be deleted from the Federated Data Repository index to the indexing queue. When the indexer processes this entry, it does the following things:

Identifies all the documents indexed into the Federated Data Repository index that are related to tasks from this process instance, and creates corresponding entries in the indexing queue so that the indexer deletes these documents.
Deletes the process instance document.

On the next indexing cycle, the added entries will be processed and the corresponding task documents will be deleted.

In the current design, you do not need to run the ‘sync-deletions’ maintenance operation, which does the following actions:

Searches in FED_INDEX_INSTANCE for rows related to process instances that do not exist in LSW_BPD_INSTANCE, and creates the corresponding entries in the indexing queue.
Searches in FED_INDEX_TASK for rows related to tasks that do not exist in LSW_TASK, and create the corresponding entries in the indexing queue.

You might use it, for example, when you deleted some tasks without deleting the parent process instance. To have the corresponding task documents also deleted from the Federated Data Repository index, execute the ‘sync-deletions’ maintenance operation by calling

POST
/ops/fdr/bpd/maintenances/sync-deletions

To get the status and details about the latest run of the ‘sync-deletions’ maintenance operation, call GET /ops/fdr/bpd/maintenances/sync-deletions.

Monitoring the Federated Data Repository indexing

You can gather KPIs about the Federated Data Repository indexer by calling the GET /ops/fdr/bpd/info API, which returns the following data:

Table 1. Data returned by the `GET /ops/fdr/bpd/info` API
Field	Description
date	The time at which the information about the Federated Data Repository indexer was retrieved.
indexedContent.instanceCount indexedContent.taskCount	Number of indexed process instance and task documents.
indexedContent.latestInstanceIndexedTime indexedContent.latestTaskIndexedTime	The date of the latest process instance and task documents indexation times.
fdrBpdIndexingQueue.oldestPendingEntry fdrBpdIndexingQueue.youngestPendingEntry	Time range between the oldest entry in the queue and the newest one.
fdrBpdIndexingQueue.pendingUpdateCount fdrBpdIndexingQueue.pendingRebuildCount fdrBpdIndexingQueue.pendingDeletionCount	Number of entries in the indexing queue by kind: pendingUpdateCount refers to the regular indexing of process instances and tasks being actively worked on. pendingRebuildCount refers to the entries in the indexing queue which are related to the rebuild. pendingDeletionCount refers to the entries in the indexing queue which are related to deletions. When the Federated Data Repository indexer retrieves a batch of entries to process from the indexing queue, the priority will always be given to the regular indexing. If there is some space left in the indexing batch, the Federated Data Repository indexer will then process the entries related to the index rebuild, and finally the entries related to the deletions. pendingUpdateCount should not be greater than the indexing batch size, which is by default set to 1000. If pendingUpdateCount is larger than the indexing batch size, then it indicates that the indexing throughput cannot cope with the volume of updates performed by the Process Engine. There may be a bottleneck somewhere. You should first ask your database administrator to check the database overall performance. Also, a possible option would be to scale up the indexing by adding new indexing threads. You can do this by either adding new nodes to your cluster, or by incrementing the <fdr-index-threads> property in 100Custom.xml.
indexedContent.deletedInstanceCount indexedContent.deletedTaskCount	Number of deleted process instances and task documents that are still tracked in the FED_INDEX_INSTANCE table. The deletions are tracked when you enable the failover support. See ‘post-backup’ maintenance operation for details. If you no longer maintain your topology for failover support and that the indexedContent.deletedInstanceCount and indexedContent.deletedTaskCount properties are returning positive numbers, then you should call DELETE /ops/fdr/bpd/backupState to free up some space by stopping tracking the deletions.
fdrBpdIndexingQueue.failedCount	Number of failed queue entries which will not be re-processed until a ‘retry-failures’ maintenance operation is triggered. This indicates that some documents could not be indexed. The most probable root cause is that the indexing of such document would exceed the total fields limit defined on your Federated Data Repository index. See for 'retry-failures' maintenance operation more details.
fdrBpdIndexingQueue.oldestFailedEntry fdrBpdIndexingQueue.youngestFailedEntry	The time range between the oldest and the youngest failed queue entries.