Updating the processing configuration

You can update the default configuration and, if necessary, restore it, declare a new monitoring source, and enable Kafka egress.

About this task

The processing configuration outlines the methodology for event processing, structured as a collection of files. The main configuration file is processing-conf.json. The specification for the processing configuration describes the configurable elements within this file. Additionally, this configuration may reference other files. Below is a list detailing the default files comprising the default processing configuration.

processing-conf.json - The main configuration
transformation.jslt - Defines the transformation of raw events into time series.
opensearch-mapping.json - Defines the OpenSearch mapping that is applied to the default index.
ibm-bai-odm.jslt - The library of functions to transform Operational Decision Manager events.
transformation-data.jslt - Not used by the default configuration. This file is an example of transformation that retains only input and output business data

For Operational Decision Manager applications, you have the following additional files:

ibm-bai-odm.jslt - The library of functions to transform Operational Decision Manager events.
transformation-data.jslt - Not used by the default configuration. This file is an example of transformation that retains only input and output business data

You can export and import the processing configuration to promote changes across environments by using the management service command-line interface (CLI). For instructions about this CLI, see Installing and using the management CLI.

Customizing the processing configuration involves the following steps.

Updating the default configuration
Declaring a new OpenSearch monitoring source
Enabling Kafka egress
Restoring the default configuration

Tip: It is a good practice to test and validate your customization on a non-production environment.

For your Kubernetes deployment of IBM Business Automation Insights to enable creation dashboards for events from custom sources in Business Performance Center, you must configure the monitoring_sources subsection of the custom resource YAML file.

Updating the default configuration

Before you begin

Make sure you work on a local copy of the configuration, and then redeploy the whole configuration. In this way you can keep track of your changes, back them up, and restore them as necessary.

Procedure

Export the current configuration by using the CLI.
For Operational Decision Manager:
```
management-cli processing-conf export --name="dba/bai-odm" --directory=<directory>
```
For Decision Intelligence Client Managed Software (Decision Intelligence):
```
management-cli processing-conf export --name="dba/bai-ads" --directory=<directory>
```
When you install Business Automation Insights with ODM or DICMS processing enabled, the default processing configuration is deployed by the setup job. If you eventually need to restore the default configuration, see Restoring the default configuration.
Update the processing configuration.
You update the configuration in conformance with the processing configuration specification. The provided examples illustrate common use cases. Use them as a good starting point to learn how to customize the processing.
- Example 1: Processing and writing business data or common data: Based on your monitoring requirements, you can choose to process business data or common data or both.
- Example 2: Writing time series to dedicated OpenSearch indices based on the ruleset path: Based on the ruleset path, you can choose to process and write business data or common data to dedicated indexes.
- Example 3: Filtering out data: You can remove data that yo do not want to monitor.
- Example 4: Obfuscating data: You can obfuscate data to comply with regulations.
When you define a new OpenSearch index in the processing configuration, you must declare a monitoring source so that you can later monitor the results in Business Performance Center.
Import the processing configuration by using the CLI.
For Operational Decision Manager:
```
management-cli processing-conf import --name="dba/bai-odm" --directory=<directory>
```
For Decision Intelligence:
```
management-cli processing-conf import --name="dba/bai-ads" --directory=<directory>
```
The updated configuration replaces the existing configuration. The odm or ads processing job loads the configuration when it starts. Next, you restart the job to load your changes to the configuration.

Redeploy the odm or ads processing job.

Retrieve the name of the Kubernetes job of the odm or ads processor.

For Operational Decision Manager:

oc get jobs -o=custom-columns=Name:.metadata.name --no-headers | grep bai-odm

For Decision Intelligence:

oc get jobs -o=custom-columns=Name:.metadata.name --no-headers | grep bai-ads

Retrieve the parallelism of the ODM or DICMS processing application.

For Operational Decision Manager:

kubectl get job <ODM_processor_job_name> -o jsonpath='{.spec.template.spec.containers[?(@.name=="<ODM_processor_job_name>")].env[?(@.name=="JOB_PARALLELISM")].value}'

For Decision Intelligence:

kubectl get job <ADS_processor_job_name> -o jsonpath='{.spec.template.spec.containers[?(@.name=="<ADS_processor_job_name>")].env[?(@.name=="JOB_PARALLELISM")].value}'

Redeploy the ODM or DICMS processing application.

For Operational Decision Manager:

management-cli processing-app deploy --name="dba/bai-odm" --parallelism="<ODM_processing_app_parallelism>"

For Decision Intelligence:

management-cli processing-app deploy --name="dba/bai-ads" --parallelism="<ADS_processing_app_parallelism>"

If you remove a Kafka ingress or egress from the processing configuration, or alter the uid value of either, the Flink job crashes due to a Flink limitation. Specifically, the Flink job is unable to recover from a savepoint. To resolve this issue, you need to pass the --allow-non-restored-state option when deploying the processing application:
For Operational Decision Manager:
```
management-cli processing-app deploy --name="dba/bai-odm" --parallelism="<ODM_processing_app_parallelism>" --allow-non-restored-state
```
For Decision Intelligence:
```
management-cli processing-app deploy --name="dba/bai-ads" --parallelism="<ADS_processing_app_parallelism>" --allow-non-restored-state
```

Declaring a new OpenSearch monitoring source

About this task

When you update the processing configuration, you might decide to send the results to one or more new OpenSearch indices. To learn how to declare a new OpenSearch monitoring source, see Configuring a monitoring source for Operational Decision Manager events.

Example

For Operational Decision Manager:

- monitoringSourceType: 'Decision (ODM) Loan Validation'
    opensearchIndex: icp4ba-bai-odm-loanvalidation-ibm-bai
    monitoringSourceFields:
      - field: rulesetPath

For Decision Intelligence:

- monitoringSourceType: 'Decision (DICMS) Loan Validation'
    opensearchIndex: icp4ba-bai-ads-loanvalidation-ibm-bai
    monitoringSourceFields:
      - field: decision.serviceId
      - field: decision.serviceVersion

Enabling Kafka egress

About this task

By default, Kafka egress is disabled.

Procedure

To enable Kafka egress, follow the procedure to update the processing-conf.json processing configuration file.
Set the value of the Kafka egress enabled property to true.

Restoring the default configuration

Before you begin

Tip:

Make sure that the jq command-line JSON processor is installed. Some of these troubleshooting procedures require this tool. The jq tool is available from this page: https://stedolan.github.io/jq/ External link opens a new window or tab .

About this task

The default processing configuration is deployed when Business Automation Insights is installed. If you have imported changes and you need to restore the default configuration, follow this procedure.

Procedure

Before you restore the default configuration, ensure that the bai-setup job is available.
```
oc get job <custom_resource_name>-bai-setup
```

Delete the dba/bai-odm processing application.

For Operational Decision Manager:

management-cli processing-app delete --name="dba/bai-odm"

For Decision Intelligence:

management-cli processing-app delete --name="dba/bai-ads"

Restart the bai-setup Kubernetes job, which restarts the bai-setup pods.

oc get job <custom_resource_name>-bai-setup -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | oc replace --force -f -

Results

After the bai-setup job completes, the default processing configuration is restored.