Known issues and limitations

If your system is installed in the same namespace as IBM Cloud Pak® for Network Automation, the connection to the inventory service breaks. This is caused by a clash with another service with the same name.

Workaround: Use the config.yaml file to enforce the correct values for INVENTORY_SERVICE_HOST and INVENTORY_SERVICE_PORT.

Example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: noi-topology-sizing
  namespace: namespace
data:
  asm: |
    ui-api:
      containers:
        ui-api:
          env:
            - name: INVENTORY_SERVICE_HOST
              value: noi-topology-inventory.namespace.svc
            - name: INVENTORY_SERVICE_HOST
              value: "9178"

After you install IBM Concert Operate for a production environment deployment, various services might not be available due to connection issues with Cassandra. To resolve this issue if it occurs, restart Cassandra and the schema creation pods.

If your environment has many secrets and ConfigMaps, when the ibm-aiops-orchestrator (lead operator) attempts to build its cache, the operator can exceed its memory allocation and cause a Kubernetes out-of-memory error for the container. This error can prevent the IBM Concert Operate installation from reconciling, blocking the installation from completing.

If you encounter this issue, the operator requires more memory resources to build its cache. Override the subscription resource to increase the memory limits for the pod and avoid the out-of-memory issue.

Kafka topics are created by using the KafkaTopic custom resource or API. Only topics that are created by custom resource are shown when running oc get kafkatopics. To see a full list of all the Kafka topics use the following steps:

1. Install kcat.
2. Get the waiops-mustgather.sh script. For more information, see Installing the Concert Operate MustGather tool.
3. Run waiops-mustgather.sh -V kafka-topics

The ibm-cs-postgres-backup and ibm-zen-objectstore-backup-pvc persistent volume claims (PVCs) are installed by the Zen component of IBM Concert Operate, but are not used. If the storageclass binding mode is WaitForFirstConsumer, then the PVCs remain in Pending state until a consuming pod uses them. As these PVCs are not used by any pod, they remain in Pending state. This has no impact to IBM Concert Operate or its functionality. Do not delete these PVCs as zenservice reconciliation will recreate them.

If you are using or intend to use Instana to monitor IBM Concert Operate, note that Instana AutoTrace is not supported and must not be enabled for the IBM Concert Operate namespace. Enabling AutoTrace increases storage usage, which can exceed the storage limits on IBM Concert Operate pods. When this happens, pods are evicted, preventing both installation and upgrade of IBM Concert Operate.

When enabling the Instana AutoTrace webhook, disable AutoTrace specifically for the IBM Concert Operate namespace by adding the following parameter to the helm install command that you run from the Instana documentation:

--set autotrace.exclude.namespaces[0]=<namespace>

Where <namespace> is the namespace of your IBM Concert Operate deployment.

For more information see Configuring AutoTrace webhook in the Instana documentation.

If Instana monitoring is already configured for your IBM Concert Operate deployment and AutoTrace is enabled, disable it by running the following command:

oc patch mutatingwebhookconfigurations instana-autotrace-webhook --type='json' -p='[{"op":"add","path":"/webhooks/0/namespaceSelector","value":{"matchExpressions":[
{"key":"kubernetes.io/metadata.name","operator":"NotIn","values":["<namespace>"]}]}}]'

Where <namespace> is the namespace of your IBM Concert Operate deployment.

If you have upgraded through every successive release from 4.10.x to 4.11.x to 4.12.x to 4.13.x, a stale reference to the shared PostgreSQL connection secret that was removed in 4.11.x may still exist in the ASM and IR Core custom resources. This reference causes the ASM and IR Core operators to attempt an obsolete migration flow during upgrade, where they look for an IBM Operator for PostgreSQL cluster that does not yet exist, and then get stuck because the original shared secret no longer exists.

Ensure that you follow the upgrade procedures which have a step to remove stale references if they exist. For more information, see the upgrade documentation for your deployment type.

Exact templates cannot be exported and imported, as they are created from exact internal IDs, which change when the environemnt changes. You must manually recreate these templates after a migration.

By default, an Automation Analyst role is displayed within the Concert Operate console Access control page when you are assigning a role to a user. This default role is used within the IBM Automation family of offerings, which includes IBM Concert Operate, however, this role is not used within IBM Concert Operate.

This role does not include or provide any permissions within IBM Concert Operate and should not be assigned to users within IBM Concert Operate.

When you delete a user group, the users that were included in the group remain in your list of users. Any role that is inherited through the deleted user group is removed from the users. If the users were assigned roles individually, they continue to have those roles and can continue to log in to the UI console and complete tasks. If the users that were in the deleted user group need to be removed completely, an administrator needs to manually remove the users. Users can be removed by clicking the Delete icon for the user's entry within the list of users on the Access control Users tab.

When you have users within a user group and view the Manage assignees pane of the Incidents and alerts page, you might not see some users who are listed. This error can occur when the users from the LDAP user group are not individually onboarded. To verify whether a user is onboarded, go to the Access control > Users tab and check whether the user is listed. If the user is not listed, that user must first log in to the console, which validates their roles and permissions. After logging in, the user can display in the list of users and on the Manage assignees pane.

The Manage assignees pane is viewable from the list of all Incidents. Select an incident and then click Manage assignees. After you select an existing user group, you should see the included users who are listed.

When creating an edge via either the File or Rest observers, the POST request returns a Topology service location URL that is not accessible. The URL cannot be used to manage the edge because the relevant API is not exposed. Workaround: None

IBM Concert Operate does not receive any notification from Turbonomic for actions that are closed without being executed. For example, an action related to an erroneous condition that is no longer occurring gets automatically closed in Turbonomic. But its corresponding IBM Concert Operate alert remains open indefinitely and must be cleared manually from the console.

The limitation is that the datalayer imposes an 8 kb size limit on the details field in the alert schema. The details field is populated by the log anomaly event, which provides the relevant information to display in slack when trying to view alerts in the chatops. Whenever the details field size exceeds 8 kb, the returned json object is truncated and therefore when the user clicks view alerts to retrieve the alerts related to an incident, expected results are not seen and an error is recorded.

The current fields under the details objects are:

 end_timestamp: int
 original_group_id: str
 causality: dict
 detected_at: float
 source_application_id: str
 log_anomaly_confidence: float
 log_anomaly_model: List[str]
 prediction_error: dict
 error_templates: List[int]
 count_vector: List[int]
 text_dict: dict
 application_group_id: str
 application_id: str
 model_version: str
 severity_from_model: int
 description: str

If you have many active log integrations, such multiple Kafka, ELK, Splunk, and Falcon LogScale integrations, and you create and enable another Falcon LogScale integration, you might notice that the integration status is stuck in an error or restarting state. This state can even occur after the integration is operating as expected.

This can occur if you exceed the limit for the number of jobs that can run on the underlying service, which results in insufficent resource available to start the integration. To resolve this issue, complete one or more of the following tasks:

• Increase the replica count of your task managers.
• Increase the task manager count per replica.
• Change the parallelism of your integrations.
• Cancel other integrations

If you have an integration that you are restoring, the status for these integrations can be in error after the restore process completes. To resolve this status, you need to edit and save your integrations with the Integrations in the Concert Operate Concert Operate console. Editing the integration regenerates the associated Flink job for the integration, which updates the status.

If you enable historical data flow for a ServiceNow integration, you might notice that the historical data is not collected from ServiceNow. For instance, when you check the grpc-snow pod, you can see ticket data available, but when you check the Flink job or OpenSearch, you can notice that no data was collected. If this issue occurs, turning off the historical data flow and turning it back on can cause the data to begin to be collected.

If you have a Dynatrace integration created and pull historical data with multiple metrics for Metric Anomaly AI Training, you can encounter an issue where the Dynatrace pod restarts, but does not complete retrieving the expected historical data for training. This issue can occur intermittently, depending on the number of metrics that are selected for the integration and the amount of data to be retrieved.

If this potential out-of-memory or out-of-resources issue occurs, consider creating separate integrations to monitor different and smaller sets of metrics. By splitting the integrations, you can reduce the amount of data to be retrieved through the initial integration that can cause this issue.

If there is an active ServiceNow integration with data collection enabled and the ServiceNow credentials change, the ServiceNow user account can get locked out. ServiceNow has an automatic login locking script called "SNC User Lockout Check", which locks users out after more than 5 failed attempts (including any failed API calls).

If you check the Incidents and alert page, you will see also an alert saying "ServiceNow instance authentication failed".

When this problem occurs, unlock the user in ServiceNow. Then change the password in the ServiceNow integration and save. When authentication fails in the ServiceNow integration, there is a 1-minute wait time before you can access it, to prevent a lockout from occuring quickly.

In some cases it is observed that log anomaly training fails on large data due to being Out Of Memory (OOM) or if there is a problem with ES shards. The solution is to scale up the resources to handle large data training.

For more information about shard management, see About indices and shards. For more information about increasing ES Resources, see Log anomaly training pods CPU and Memory resource management.

On creating a integration, the Flink job retrieves data normally and the gRPC pod is running without error. However, the console shows that the integration status is 'not running'.

When a log data integration (Falcon Logscale, ELK, Mezmo, Custom, Splunk) is running in Historical data for initial AI training mode, and a custom regex is added in the field_mapping section, the data processing can take a long time. Although the Data collection status might be shown on the UI as Done, data could still be being processed and written to OpenSearch in the background.

To speed up this process, you can increase the Base parallelism number that is associated with that integration. For more information, see Increasing data streaming capacity.

If you have an IBM Tivoli Netcool/Impact integration, you can encounter an issue where the integration temporarily stops processing during the sending of an event stream to IBM Concert Operate.

This issue can occur when you have an IBM Concert Operate policy that triggers an IBM Tivoli Netcool/Impact policy when certain types of events are received. If this issue occurs and stops the event processing, the Impact integration logs or Impact policylogger logs can include messages that are similar to the following example exceptions:

[6/14/23, 11:38:45:816 UTC] 0000005d ConnectorMana W failed to send status update
...
[6/14/23, 11:38:45:815 UTC] 000023ca StandardConne W configuration stream terminated with an error
...
[6/14/23, 11:38:45:816 UTC] 000023cc GRPCCloudEven W consume stream terminated with an error: channel=cp4waiops-cartridge.lifecycle.output.connector-requests

If you encounter this issue, you might need to restart the impact-connector pod to begin the processing of the event stream again.

If the IBM Tivoli Netcool/Impact cluster uses a non-default cluster name ("NCICLUSTER"), the integration can fail to validate the integration. The IBM Tivoli Netcool/Impact server may report DynamicBindingException errors in the impactgui.log:

com.micromuse.common.nameserver.DynamicBindingException: DynamicBindingException: Service [NCICLUSTER] not in nameserver.

To resolve the issue, wait for the backend IBM Tivoli Netcool/Impact server to finish initializing before starting or restarting the Impact GUI server.

If IBM Tivoli Netcool/Impact is running fix pack 7.1.0.26 or later, you can also resolve the issue by setting the nameserver.defaultcluster property in the GUI server. Add the following line to $IMPACT_HOME/etc/nameserver.props:

impact.nameserver.defaultcluster=CLUSTERNAME

where CLUSTERNAME is the name of the IBM Tivoli Netcool/Impact cluster.

When data flow is enabled for the Splunk connector, the connection fails with a Bad Request for connection error.

Workaround: Check that the models are deployed correctly. Then deploy the models in the AI hub UI.

The following known issues have been obeserved with the GitHub connector:

• In the Incident Overview > Add tickets to this incident panel, GitHub issues are missing the Updated by information. Additionally, searching for GitHub similar past resolution tickets from the Source drop-down menu in this panel will not display any tickets.
• The GitHub connector might be missing from the list of integrations in training modules such as Similar tickets and Change risk.
• The default issue mappings in a GitHub integration does not have assignees. However, if assignees are added to the mappings issues are not created in GitHub.

If the ServiceNow change request, incident, or problem contains a large amount of text, such as work notes with close to 150,000 characters, the ticket is dropped and a warning is logged in the pod log. Dropping the ticket affects change risk for that ticket, as it will not occur or that ticket will not be used for similar ticket detection.

If there is an active ServiceNow integration with data collection enabled and the ServiceNow credentials change, the ServiceNow user account can get locked out. ServiceNow has an automatic login locking script that is called "SNC User Lockout Check", which locks users out after more than five failed attempts (including any failed API calls).

If you check the Incidents and alert page, you see also an alert saying "ServiceNow instance authentication failed".

When this problem occurs, unlock the user in ServiceNow. Then, change the password in the ServiceNow integration and save. When authentication fails in the ServiceNow integration, there is a 1-minute wait time before you can access it, to prevent a lockout from occurring quickly.

You might notice that the New Relic observer job fails due to an authorization error. The issue occurs due to changes in the New Relic API.

The error can resemble the following example:

ERROR  [2025-02-06 16:02:51,209] [pool-12-thread-1] c.i.i.t.o.n.j.NewRelicLoadJob -  Failed to validate connection: javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized

There is no workaround currently available.

When you insert the modified string into ServiceNow, text such as incident h9md-6f1q is returned to Concert Operate.

Workaround: Avoid altering the text in ServiceNow.

You might notice your external observer jobs failing with internal server error and invalid API call messages. This issue is caused by a temporary outage of the NASA FIRMS (Fire Information for Resource Management System) API, which is used by the external risk observer jobs.

For more information about the outage, see the Data Outages page in the NASA FIRMS website.

After upgrading Concert Operate to version 5.1.0, you might notice that you can neither delete the existing Instana listener jobs nor modify their status by using the application UI.

Workaround: Use the following steps to delete the Instana listener jobs:

1. Run the following command to extract the required credentials:

   PROJECT_CP4AIOPS=<cp4aiops_namespace>
   oc -n ${PROJECT_CP4AIOPS} get secret aiops-topology-asm-credentials -o jsonpath='{.data.username}' | base64 -d
   oc -n ${PROJECT_CP4AIOPS} get secret aiops-topology-asm-credentials -o jsonpath='{.data.password}' | base64 -d
   

2. Identify the aiops-topology-topology pod by using the following command:

   oc get pods -n ${PROJECT_CP4AIOPS} -o name | grep aiops-topology-topology
   

   Sample response:

   aiops-topology-topology-d5f4bfcc8-zjrzx                           1/1     Running     0            9d
   

3. Connect to the identified toplogy pod by using the following command:

   oc rsh -n ${PROJECT_CP4AIOPS} <aiops_topology_topology_pod_name>
   

4. Extract the ID of the Instana listener job by using the following API call:

   curl -X 'GET' \
      'https://localhost:8080/1.0/topology/mgmt_artifacts?_filter=observerName=instana-observer&_filter=tags=LISTEN_JOB&_field=name&_type=ASM_OBSERVER_JOB' \
      -H 'accept: application/json' \
      -H 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255' \
      -u <asm_credentials_username>:<asm_credentials_password> \
      -ks | jq
   

   asm_credentials_username and asm_credentials_password correspond to the credentials extracted in step 1.

   Sample response:

   {
      "_executionTime": 10,
      "_offset": 0,
      "_limit": 50,
      "_items": [
         {
            "_id": "GLy7Kp0wQGS1FSRKy8j7rw",
            "name": "d49864e6-4a7f-454c-9f85-e831c350dd77"
         }
      ]
   }
   

   Verify that the name in the response corresponds the listener job you wish to delete as shown in the UI.

5. By using the _id in the response, make an API call to delete the job with that ID:

   curl -X 'DELETE' \
      'https://localhost:8080/1.0/topology/mgmt_artifacts/<listener_job_id>' \
      -H 'accept: application/json' \
      -H 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255' \
      -u <asm_credentials_username>:<asm_credentials_password> \
      -ks
   

   listener_job_id is the _id value that is returned in step 4. asm_credentials_username and asm_credentials_password correspond to the credentials extracted in step 1.

6. Repeat the API call in step 4 to verify that the job is deleted. If the job is successfully deleted, it is also removed from the application UI.

Under certain circumstances search does not return results when special characters (such as &) are used as search terms.

Workaround: Avoid searching special characters.

On rare occasions a composite resource may contain more than one geolocation marker. All of these will be plotted on the Resource map. If one of these locations falls outside the displayed map area, its status is not displayed.

Workaround: None. Be aware of this quirk when viewing composite resource on the Resource map.

For the Azure Observer, a subnet can be intermittently missing the relationship with an IP address in the topology for a resource. While the relationship can be intermittently missing, both the subnet and IP address verticies remain available in the topology.

If you install Infrastructure Automation, you, or an administrator, must add the required Kubernetes permissions to user roles before your users can begin to access and use Managed services or the Service catalog.

As an administrator, add the following permissions to your use roles:

For more information about how to add permissions to a role, see Managing roles for Infrastructure Automation .

Non-LDAP authenticated users cannot be used with Single Sign-On for Infrastructure Management. If you attempt to use Infrastructure Management with a non-LDAP authenticated user, you can encounter the following error:

While logged in to the Infrastructure Automation UI console with a non-LDAP user, attempting to start Infrastructure Management fails with an error. This is a limitation.

The error states:

OpenID Connect Provider error: Error in handling response type.

Installation of Infrastructure Management in IBM Concert Operate does not support Red Hat Advanced Cluster Management and IBM Cloud Pak for Multicloud Management core. You can continue to use the Kubernetes cluster life-cycle templates and services to create a Kubernetes cluster and import the cluster to an existing installation of Red Hat Advanced Cluster Management, if an installation is available. Deploying hybrid applications are also not supported by Infrastructure Automation.

When you are working within Infrastructure Automation and log out and then log back in, you can be redirected to the Administration panel instead of the Infrastructure Automation home page or other page that you were previously using. If this occurs, you can use the Cloud Pak switcher in the upper right of the UI console to switch to the Infrastructure Automation home page and then return to the page that you were previously using.

If you are creating the database for the Infrastructure Management appliance and you encounter an error, such as the database creation failing to complete successfully, you might not be able to continue with your setup without redeploying. For instance, if the creation fails, resetting the database to clean up your database and deployment can also fail. To resolve this issue, you need to redeploy the Infrastructure Management appliance image before reattempting to create the database.

After you install Infrastructure Automation, you can encounter an error where the cam-tenant-api pod displays as running, but not in a ready state. When this error occurs, you can see the following message:

[ERROR] init-platform-security - >>>>>>>>>> Failed to configure Platform Security. Will retry in 60 seconds <<<<<<<<<<<<< OperationalError: [object Object]

If this error occurs, delete the cam-tenant-api pod to cause the pod to restart and attempt to enter a ready state.

Infrastructure Automation Test deploy from a Service Overview page fails to deploy.

When you attempt to deploy an Ansible playbook on an Infrastructure Management appliance through an embedded Ansible deployment, the playbook deployment can fail with the following error:

<35.237.119.31> ESTABLISH SSH CONNECTION FOR USER: ubuntu
fatal: [35.237.119.31]: FAILED! => {
"msg": "Unable to create local directories(/home/manageiq/.ansible/cp): [Errno 13] Permission denied: b'/home/manageiq'"
}

If you encounter this error, log in to the appliance as the root user and then deploy the playbook again:

1. Run the command:

   mkdir -p /home/manageiq
   

2. Run the command:

   chown manageiq:manageiq /home/manageiq
   

3. Deploy the Ansible playbook again.

After you restore Managed services (cam) from a backup, the deployment instance fails with a socket hang up error.

If this error occurs, restart the cam-iaas pod by running the following command:

oc delete pod <cam-iaas-xxxx> -n <namespace>

Where <namespace> is the project (namespace) where Infrastructure Automation is installed, and <cam-iaas-xxxx> is the name of the cam-iaas pod to restart.

With this restart, the service deployment can complete successfully.

The Embedded Terraform feature in Infrastructure Automation is not yet supported in a FIPS enabled OpenShift Container Platform cluster.

After installing RPMs on an Infrastructure Management appliance, the opentofu-runner podman container filesystem permissions are reset to user=manageiq group=manageiq. This causes the opentofu-runner container to fail the next time it is started.

Workaround:

1. Stop the evmserverd process.

   systemctl stop evmserverd
   

2. Run the following command.

   rm -rf /var/www/miq/vmdb/data/containers
   

3. Import the OpenTofu image again.

4. Start the evmserverd process.

   systemctl start evmserverd
   

The AWS Terraform plug-ins for Power and zLinux architectures are not included in the current IA release. These plug-ins contain critical security vulnerabilities that may cause significant risks.

Starting with version 4.13.0, the operator no longer creates the `aimanager-aio-training-priority` `PriorityClass` during installation in order to avoid requiring cluster-level permissions.

Algorithms such as 'Similar Incident', 'Change Risk', and 'Legacy Log Anomaly Detection' dynamically create training or precheck jobs that depend on this `PriorityClass`. If the `PriorityClass` is not present in the cluster, these jobs might fail to start.

If you use catch-all templates for mapping uncategorized messages during AI model training, you can encounter an issue where the log parsing assigns messages for an error to the catch-all templates instead of generating an expected template for that error. If this issue occurs, you might not see expected anomalies.

If you suspect this issue is occurring and you do not see expected anomalies, complete the following steps to manually verify your training templates, and remove any catch-all templates that incorrectly generated.

1. Retrieve the normalized logs from your logtrain indices.

2. Identify the logs that are error logs. Review those logs to determine the template mappings.

3. Retrieve the identified templates from OpenSearch.

4. Use the error log contents and the template ID from the retrieved normalized logs to identify the template string within the retrieved templates.

5. If the template string is comprised entirely of parameters, or a single word and parameters, the template might be a catch-all template. For example, the following string is an example of a catch-all template:

   <>to <><><><><>
   <> <> <> <>-<>-<> <> <> <> <> <> <> <> <> <> <>
   

6. Manually delete any catch-all templates.

When you push a large training file (for example 60 M records, such as logs or events) to Kafka through your configured integration, the number of records that are ingested and displayed on OpenSearch might not match. OpenSearch record count might be lower than Kafka count due to deduplication. If you encounter this issue, split large files into smaller batches and send them individually to Kafka (for example, 5 M records each). When you are pushing a batch, ensure that you wait for an ingest to complete and the associated records display on OpenSearch before you push the next batch of records.

If you set a value for the EXPIRY_SECONDS environment variable and upgrade, the environment variable is not retained after the upgrade.

After the upgrade is completed, set the environment variable again. For more information about setting the variable, see Configuring expiry time for log anomaly detection alerts.

In some cases a model that has been trained successfully is unable to detect certain log anomalies. The quality of the model is independent of whether it trained successfully, and model quality tends to improve as more training data is available. If the model is not detecting anomalies in your logs, consider training the model again but using additional days of training data to improve the model quality.

If you have metric anomaly detection training scheduled to run, such as daily, you can encounter an issue where the training does not run as scheduled. If the training job does not run on schedule, log in to the Concert Operate console and click the Metric anomaly detection algorithm tile and then Train models.

On rare occasions, a Change risk model fails to create, even though Precheck data indicates that the data is good. This failure is caused by an insufficient number of problematic change risk tickets being available to create a good model. This problem resolves itself when enough tickets become available for the model. (For more information, see Closed change ticket count requirements).

To confirm that insufficient problem tickets is causing the failure, view the Change risk logs on the training pods.

To retrieve the pod:

oc get pod | grep training-cr

View the logs for training Change risk models.

oc logs <pod-name> # Ex: training-cr-1b5ef57f-9053-4037-95ca-c1e8b8748fc5

Check whether the log contains the following

size of the problematic (aka labels) tickets is insufficient

If confirmed, ensure that enough problematic change tickets are available before training the model again.

In IBM Concert Operate 5.1.0, alerts might not be generated for the Log Anomaly - Golden Signals algorithm when inference log data contains name and value pairs. These pairs are tokens with the key=value pattern. They might prevent anomalies from being matched to their respective templates.

For example, training generates the following log template:

<> exe="/usr/bin/dbus-daemon" sauid=UNKNOWN_VAR hostname=? addr=? terminal=?'

Then, during inference, incoming log data is matched against the template from training. If the incoming log data contains tokens with the key=value pattern like in the following example, the logs are classified as unmatched.

[3557470.922719]  exe=\"/usr/bin/dbus-daemon\" sauid=103 hostname=? addr=? terminal=?'
[3557971.107893]  exe=\"/usr/bin/dbus-daemon\" sauid=103 hostname=? addr=? terminal=?'

Alerts from this set of logs are not displayed.

When the log anomaly detection - golden signals algorithm generates alerts in IBM Concert Operate 5.1.0, the counts against those template patterns are not updated in the training UI table. Enable historic alert storing in OpenSearch to access alert counts.

oc get installation
oc edit installation <installation-name>

Or access the installation with the Red Hat OpenShift console. Go to Operators > Installed Operators > IBM Concert Operate > IBM Concert Operate and edit the YAML file for aiops-installation.

spec:
  automationFoundation: {}
  license:
    accept: true
  pakModules:
    - config:
      - name: ir-core-operator   # Find the config item with this name, or add this item if it does not exist
        spec:
          issueresolutioncore:
            customSizing:
              deployments:
              - name: datarouting
                replicas: 1   # Use 3 for large deployment
              - name: esarchiving
                replicas: 1
      enabled: true
      name: applicationManager  # Find the pakModules item with this name, or add this item if it does not exist

After the components are enabled, new alerts are stored to OpenSearch indices, and the alert counts for subsequent alerts are updated correctly in the training UI table.

Similar tickets training is not available in IBM Concert Operate on Linux. You can manually add tickets from your ticketing integrations in the incident overview.

If you are viewing a closed incident that has all associated alerts resolved, you can encounter an error when you view the Alerts tab. This "An unknown error occurred" error displays when there are no associated alerts. You can ignore this error message as the incident and alerts are resolved and closed.

Incidents with status of "Closed" are missing topology information on the incident Overview tab. The associated alerts of the closed incidents are also missing from Alerts tab. The following critical error is displayed on the Topology tab for closed incidents: "No resource exists with the specified identifier and time point".

In some cases, when you click the checkbox in the Related alerts, it does not add the related anomaly to the metric anomaly chart.

In scenarios where large amounts of historical event data are ingested into the system, it's possible that problems and resolutions can be processed out of order, sometimes resulting in alerts not clearing as expected. To avoid this issue, try ingesting smaller batches of event data into the system.

In a case where an alert meets the incident-creation conditions of multiple policies, only one incident is created. However, all policies that proposed an incident, and the system incident creation policy has the same Last run times on the Policies hub. Each of these policies also increment their Matched counts by 1 in the Details tab of the side panel.

If you have a Concert Operate policy that is triggered based on incident-updated, in some cases, the policy might run more times than expected. This is because not all updates can always be processed atomically and the incident might then be updated multiple times. In turn, the incident-updated trigger will be activated more than once.

This can occur when you try to compare a NULL value in the policy condition. The policy is triggered because you can have a condition of NULL = NULL matched when the parameters referenced in the policy are not present.

To avoid this problem, you can ensure that the alert property is not equal to NULL in the condition set. For example, see the following policy conditions for alert.details.name:

The preset policy "WebSphere resolution action recommendation policy" might initially show a failed status in the policy table. If you encounter this, the policy status should self correct after a period of time.

Normally, you can select a runbook and configure it to run automatically: when an alert is converted to an incident, the runbook is assigned and runs automatically. However, if the parameter value alert.suppressed is used, the runbook does not run automatically as it reads this as a Boolean value rather than a string value. Therefore, it is necessary to manually run the runbook.