Planning an installation of IBM Concert Operate on Linux
Learn about the system requirements for an installation of IBM Concert Operate on Linux.
Before you begin
- There is not a small starter size option for the installation of IBM Concert Operate on Linux. You can only deploy a production size deployment.
- vCPU is defined as when one x86 CPU splits each of its physical cores into virtual cores (vCPU). It is assumed that one x86 CPU's physical core can be split into two logical vCPUs.
- If Linux is installed on VMware virtual machines, set the value of the
sched.cpu.latencySensitivityparameter to high. - You can deploy a base deployment of IBM Concert Operate that does not have log anomaly or ticket analysis capabilities, or an extended deployment of IBM Concert Operate with these capabilities. Learn more.
- The DB2 Integration is not available for IBM Concert Operate on Linux.
- Ensure that all cluster nodes have sufficient entropy availability (greater than 1000) at
/dev/randomto support OpenSearch. For more information, see Prerequisites. - If you use Instana to monitor IBM Concert Operate, ensure that Instana AutoTrace is disabled for the IBM Concert Operate namespace. Failure to do so can cause operational issues, and installation or upgrade failures. For more information, see Instana AutoTrace causes pod eviction and prevents install and upgrade.
Supported platforms
- Red Hat Enterprise Linux 8.10 only
- Red Hat Enterprise Linux 9.4 only
- Red Hat Enterprise Linux 9.6 only
- Ubuntu 24.04 LTS
Security-enhanced Linux (SELinux) is only supported if it is set to permissive, not enforcing. For more information, see Permanent changes in SELinux states and modes in the Red Hat documentation.
If you want to install IBM Concert Operate on Red Hat OpenShift Container Platform or on a cloud platform, see Deploying IBM Concert Operate on OpenShift.
Hardware requirements
Multiple Linux nodes are needed, and IBM Concert Operate is installed on this cluster of nodes. The cluster must be reserved for the sole use of IBM Concert Operate.
Three of these nodes must be control plane nodes. The control plane nodes coordinate the running of IBM Concert Operate across the other nodes, and are the entry point into the product. Worker or agent nodes provide more compute resources to run IBM Concert Operate services. All the nodes can be assigned as agent nodes, but three of the nodes must also be assigned as control plane nodes.
The size of each of your nodes can vary, but each worker node must meet the minimum requirements in Table 2 to accommodate the placement of larger IBM Concert Operate pods, and the combined resources of your nodes must comply with the resource totals in Table 1. You should try to spread out resources across your nodes to make it easier for workloads to be scheduled.
You must ensure that the clocks on your Linux cluster are synchronized. Each node in the cluster must have access to an NTP server to synchronize their clocks. Discrepancies between the clocks on the nodes can cause IBM Concert Operate to experience operational issues.
Overall cluster requirements
Your cluster must meet or exceed the overall size requirements in the following table.
| Resource | Base deployment requirement | Extended deployment requirement |
|---|---|---|
| Node count | 9 | 9 |
| Total vCPU | 146 | 173 |
| Total memory (GB) | 348 | 408 |
This table shows the hardware requirements for a deployment of IBM Concert Operate on a Linux cluster, for a base deployment and an extended deployment. For more information about the differences between a base deployment and an extended deployment, see Incremental adoption.
- Three of these nodes will be control plane nodes.
- An additional 1 VCPU and 3GB memory is required in the deployment for each integration that you configure. For example, if you configure two Netcool integrations then you will require an additional 2 vCPUs and 6GB memory. An integration can run on any worker node, so long as the worker node can provide the required 1 CPU and 3GB. The total resources needed by all integrations do not have to be available on the same worker node.
- You will also require an additional server for a load balancer. For more information, see Load balancing.
Minimum node size
Your cluster can include nodes of varying sizes, but all worker nodes and control plane nodes must meet at least the minimum specifications listed in the following table.
| Resource | Requirement |
|---|---|
| Minimum vCPU per node | 16 |
| Minimum memory per node (GB) | 20 |
| Minimum disk per node (GB) | 120 |
Integrations
When you are configuring an integration in IBM Concert Operate, it is important to consider the performance and footprint. If you have large amounts of data on the selected target system, you can expect relatively higher resource usage from collecting data from that system when compared to a system with less data. Additionally, the sizing of the IBM Concert Operate installation affects the number of resources that are available for running integrations. Review the following considerations for each integration category and the "Integration installation" section before you configure a new integration.
Integration installation
IBM Concert Operate integrations for metrics, events, and logs are configured in the Integrations UI page, which can be used to create, edit, delete, and track the status of integrations. Integrations run as pods and have minimum and maximum resource allocations in terms of CPU, memory and storage. Some integrations offer flexibility in terms of how to install the integration in the UI, but some integrations only offer one option.
- Local: Install the integration in the same cluster and namespace where IBM Concert Operate is installed. The integration’s status is displayed in the Integrations UI page and is automatically managed by IBM Concert Operate.
- Remote: Install the integration anywhere you choose, for example, a different network region, on SaaS, or remote on-premises (VM, cluster or container). After adding the integration, you can use the script to run the integration pod using podman.
Hardware requirements - Integrations
Before you create an integration for both local and remote installation, make sure that IBM Concert Operate or remote environment contains the hardware resources that are required to run the integrations.
The minimum resource requirements for the integrations are listed in the following table:
| Integration name | Memory limit (Mb) | CPU limit | Ephemeral storage limit (Mb) |
|---|---|---|---|
| AppDynamics, New Relic, Splunk | 2500 | 1 | 500 |
| AWS CloudWatch, Dynatrace (metrics-only), Infrastructure Management, Generic Webhook, Zabbix | 4096 | 1 | 500 |
| Dynatrace (metrics, events, topology) | 10000 | 4 | 8000 |
| DB2, GitHub, ServiceNow | 800 | 1 | 1000 |
| Email Notifications, IMPACT, Jira | 512 | 1 | 1000 |
| Instana | 4096 | 4 | 1000 |
| Netcool ObjectServer | 4096 | 2 | 2000 |
| Custom, Elk, Falcon LogScale, Mezmo | 1536 | 1 | 1000 |
- Dynatrace (metrics, events, topology) supports up to 5 integrations.
- The hardware sizing in the preceding table does not include resource slots for logs integrations or observer jobs. For more information about resource slots for logs integrations, see Performance considerations for logs data collection. For more information about defining observer jobs, see Observer jobs.
Processing abilities
Review the following sections to find out about the processing abilities of a production deployment of IBM Concert Operate. Higher rates are supported by custom sizes. For more information about customizing the size of your IBM Concert Operate deployment according to your processing and footprint requirements, see Custom sizing.
Supported resource number and throughput rates for production deployments
The following table details the number of records, events, Key Performance Indicators (KPIs), and resources that can be processed by a production deployment of IBM Concert Operate. This includes resource and throughput values for the AI algorithms.
| Component | Resource | Production deployment |
|---|---|---|
| Change risk | Incidents and change request records per second | 30 |
| Metric anomaly detection | Maximum throughput - (KPIs) for all metric integrations |
120,000 |
| Log anomaly detection (non-kafka integration) | Maximum throughput (log messages per second) for non-kafka log integrations |
8000 |
| Log anomaly detection (kafka integration) | Maximum throughput (log messages per second) for kafka log integrations |
25,000 |
| Events (through Netcool integration) | Steady state event rate throughput per second Burst rate event throughput per second |
150 250 |
| Automation runbooks | Fully automated runbooks run per second | 2 |
| Topology management | Maximum number of topology resources | 5,000,000 |
| UI users | Active users supported | 20 |
| Standing alert count | Number of stored alerts | 200,000 |
- Event rates in the preceding table assume a deduplication rate of 10 to 1 (10% unique events). For example, a rate of 100 alerts per second sent to IBM Concert Operate can be the end result of an initial 1,000 alerts per second before deduplication and other filtering is applied.
- For metric anomaly detection, the number of key performance indicators (KPIs) that can be processed for each deployment size is shown, for an aggregation period of 5 minutes and a training period of 4 weeks.
- If you are using additional integrations for metric anomaly detection with IBM Concert Operate, you can use default available policies to further refine the volume of data routed for issue resolution lifecycle actions by your users. You can also create custom policies tailored for your environment. For instance, you can use custom suppression policies to help determine which anomalies should be raised as alerts for user action. For more information about custom policies, see Suppress alerts.
- The events (through Netcool integration) throughput rates represents a refined volume of alerts that corresponds to a worst case scenario where the ratio of IBM Tivoli Netcool/OMNIbus events to IBM Concert Operate alerts has no deduplication, and is essentially a 1:1 mapping of events to alerts. However, in most production deployments, the correlation and deduplication on the IBM Tivoli Netcool/OMNIbus server side reduces the volumes of alert data that requires processing within IBM Concert Operate. As part of further optimizing the workload of data presented to IBM Concert Operate, additional IBM Tivoli Netcool/OMNIbus probe rules can filter out events of no interest to IBM Concert Operate. For instance, typical IBM Tivoli Netcool/OMNIbus maintenance events are filtered out as they are not relevant on the IBM Concert Operate side.
- The number of alerts for your system varies based on alerts being cleared or expiring. In addition alerts include a variety of event types so you might not always see the same alerts when you view the Alert Viewer in the UI.
- If you are using the File observer for more than 600,000 resources, then additional resources are required. For more information, see Configuring the File observer.
- For 200,000 stored alerts, it is recommended to set IR_UI_MAX_ALERT_FETCH_LIMIT to a maximum value of 10,000 to avoid performance impacts. For more information, see Restricting the number of alerts returned by the data layer to the Alert Viewer.
Event, alert, and incident rates
IBM Concert Operate include robust capabilities for managing events from your various applications, services, and devices. If you are integrating IBM Concert Operate with IBM Tivoli Netcool/OMNIbus the benefits that you can leverage for event management are significantly increased. This integration can give you end-to-end alert processing with an on-premises IBM Tivoli Netcool/OMNIbus server so that you can complete part of the event and incident management lifecycle on the IBM Tivoli Netcool/OMNIbus server before events are processed and delivered for action in IBM Concert Operate.
By default, IBM Tivoli Netcool/OMNIbus policies and triggers, such as correlation and deduplication activities, can execute to "pre-process" event workloads, thereby reducing the overall volume of active events on the IBM Tivoli Netcool/OMNIbus server. This overall volume presents a refined (event) workload for subsequent processing within the overall incident resolution (IR) lifecycle. On the IBM Concert Operate side, automation policies run on the remaining events that are flowing from the IBM Tivoli Netcool/OMNIbus server. IBM Concert Operate applies additional suppression and grouping filters to minimize effort, and executes runbooks to automatically resolve events where warranted, and promote the remaining events to Alerts and carefully refined Incidents for ITOps to take action on the most critical concerns.
- As a basic example, a small production IBM Tivoli Netcool/OMNIbus environment with an average incoming event rate of 50 events per second, with a correlation and deduplication ratio of 10:1 raw to correlated events (incidents), can result in a refined volume of 5 Alerts per second being sent to IBM Concert Operate for subsequent processing. With a combination of default available issue resolution (IR) policies and analytics, the alerts can be further reduced (by 90% noise reduction) to less than 1 Incident per second over time on the IBM Concert Operate side.
- As a secondary, larger example, a Production IBM Tivoli Netcool/OMNIbus environment with an average event rate of 500 events per second (with the same correlation and deduplication ratio of 10:1), can in turn present a refined volume of 50 Alerts per second being sent to IBM Concert Operate. By using the same combination of default available issue resolution (IR) policies and analytics, the alerts can be further reduced by 90% noise reduction, with a resultant 5 Incidents per second raised in IBM Concert Operate. Additional issue resolution (IR) policies can be authored to further reduce and refine Incident creation. By leveraging other advanced capabilities within IBM Concert Operate, such as fully automated Runbooks, the volume of actionable incidents that are presented for user interaction can be further reduced.
Custom sizing
The default production deployment size enables the full capabilities of IBM Concert Operate to be used with the workload volumes that are stated in the Processing abilities section. If different workload volumes are required or resource constraints are an issue, then specific capabilities such as `Metric Anomaly Detection`, `Log Anomaly Detection`, and `Runbook Automation` can be sized accordingly. IBM Sales representatives and Business Partners have access to a custom sizing tool that can assess your runtime requirements and provide a custom profile that scales IBM Concert Operate components. The custom profile is applied when you install IBM Concert Operate. This custom profile cannot be applied after installation, and attempting to do so can break your IBM Concert Operate deployment. If you require custom sizing, contact IBM Sales representatives and Business Partners with details of your intended workloads.
Custom patches, labels, and manual adjustments to IBM Concert Operate resources (such as increased CPU and memory values) are lost when an event such as upgrade, pod restart, resource editing, or node restart triggers a reconciliation. Reconciliation causes any manually implemented adjustments to be reverted to their original default values. Depending on the parameters that you want to adjust, you might be able to use a custom profile to persist your changes.
- Example 1 (Minimum scale) represents a minimally sized deployment of IBM Concert Operate for the evaluation of event management. It demonstrates event analytics, noise reduction, and the automation of issue resolution on a small topology. Metric and log anomaly detection and change risk assessment are de-emphasized. Example 1 requires 49.36 vCPU and 155GB of memory to run on a 9 node Linux cluster, this includes an overhead of 12GB of memory and 0.36 of a vCPU. If there are more than 9 nodes in the cluster, then extra resources are required as follows: 0.06 of a core and 3GB for each control plane node, and 0.03 of a core and 0.5GB for each worker node..
- Example 2 (Event management focused) represents a production deployment of IBM Concert Operate which is focused on event management capabilities. It supports event analytics, noise reduction, and the automation of issue resolution on a large topology. Metric and log anomaly detection and change risk assessment are de-emphasized. Example 2 requires 176.36 vCPU and 378 GB memory to run on a 9 node Linux cluster, this includes an overhead of 12GB of memory and 0.36 of a vCPU. If there are more than 9 nodes in the cluster, then extra resources are required as follows: 0.06 of a core and 3GB for each control plane node, and 0.03 of a core and 0.5GB for each worker node.
- Example 3 shows an example towards the upper limits that IBM Concert Operate can be scaled to across all of its capabilities, though user interface response times might vary depending on workloads and system specification. If your requirements are towards the upper limit scale, similar to this example, contact your IBM representative to ensure that your sizing is accurate to meet your requirements.
| Component | Resource | Example 1 | Example 2 | Example 3 |
|---|---|---|---|---|
| Change risk | Incidents and change request records per second | 0 | 0 | N/A |
| Metric anomaly detection | Maximum throughput - (KPIs) (for all metric integration) |
0 | 0 | 10,000,000 |
| Log anomaly detection | Maximum throughput (log messages per second) (for all metric integration) |
0 | 0 | 25,000 |
| Events (through Netcool integration) | Steady state event rate throughput per second Burst rate event throughput per second |
10 | 600 | 100 1000 |
| Automation runbooks | Fully automated runbooks run per second | 1 | 2 | 4 |
| Topology management | Maximum number of topology resources | 5,000 | 5,000,000 | 25,000,000 |
| Topology groups | Number of topology groups | 180 | 30,000 | 60,000 |
| Standing alert count | Number of alerts | 20,000 | 200,000 | 800,000 |
| UI users | Active users supported | 5 | 40 | 1000 |
Storage requirements
IBM Concert Operate deployments on Linux have the following persistent local storage requirements:
| Storage type | Total requirement | Minimum per node | Default storage path |
|---|---|---|---|
| Application storage | 3030GB | 250GB | /var/lib/aiops/storage |
| Log storage | 3GB for each node in the cluster | 3GB | /var/log/pods |
| Storage type | Total requirement | Minimum per node | Default storage path |
|---|---|---|---|
| Platform storage | 100GB | 25GB | /var/lib/aiops/platform |
To meet the local storage requirements, you must configure distributed storage across the nodes in your Linux cluster, and dedicate specific disks and logical volumes for application and platform storage. Follow the instructions in Configuring local volumes for an installation of IBM Concert Operate on Linux. The default storage paths can be configured as described in the installation instructions.
Additional requirements for offline deployments
If you are installing in an air-gapped environment (offline), you must also ensure that you have adequate space to download the IBM Concert Operate images to the target registry in your offline environment. The IBM Concert Operate images total 188 GB.
Storage performance requirements
Each node of the storage solution requires a minimum of one disk (SSD or high-performance storage array). The performance of your storage can vary depending on your exact usage, datasets, hardware, storage solution, and more.
The following table specifies the storage performance metrics that must be achieved to support a deployment of IBM Concert Operate. Storage is accessed from multiple nodes due to the distributed nature of IBM Concert Operate workloads, and the total IOPS across all nodes using storage must meet the minimum sequential IOPS requirements. If your deployment is custom-sized to support higher rates than the default production rates listed in Processing abilities, then your storage performance must exceed these metrics.
| Metric | Read | Write |
|---|---|---|
| Minimum sequential IOPS (higher is better, lower is worse) | 5000 | 5000 |
| Minimum sequential bandwidth (higher is better, lower is worse) | 20 Mi/sec | 20 Mi/sec |
| Maximum average sequential latency (lower is better, higher is worse) | 500 usec | 1000 usec |
Using a network storage system typically entails higher performance requirements on the disks due to factors such as replication and network latency. Performance at the application layer can be tested after the cluster is provisioned. A benchmarking tool is supplied that can be used to compare your storage's performance with these metrics before you install IBM Concert Operate. For more information, see Evaluate storage performance.
Network requirements
The control plane nodes require the following access:
| Port number | Direction | Protocol | Description |
|---|---|---|---|
| 80 | Inbound from load balancer | TCP | Application HTTP port |
| 443 | Inbound from load balancer Outgoing to load balancer |
TCP | Application HTTPS port |
| 2379 | Between cluster nodes | TCP | etcd port |
| 2380 | Between cluster nodes | TCP | etcd port |
| 6443 | Between cluster nodes | TCP | Control plane API server and distributed registry |
| 5001 | Between cluster nodes | TCP | Distributed registry |
| 8472 | Between cluster nodes | UDP | Virtual network |
| 10250 | Between cluster nodes | TCP | Kubelet metrics |
Worker nodes require the following access:
| Port number | Direction | Protocol | Description |
|---|---|---|---|
| 443 | Outgoing to load balancer | TCP | Application HTTPS port |
| 6443 | Between cluster nodes | TCP | Control plane API server and distributed registry |
| 5001 | Between cluster nodes | TCP | Distributed registry |
| 8472 | Between cluster nodes | UDP | Virtual network |
| 10250 | Between cluster nodes | TCP | Kubelet metrics |
Proxy configuration
If your environment requires a proxy for external network access, you can configure this at installation time. For more information, see Online installation or Offline installation.
CIDR ranges
10.42.0.0/16for pod IPs (-\-cluster-cidr)10.43.0.0/16for service IPs (-\-service-cidr)
If you have other services running within these ranges in your internal network, then at installation time you must change the default CIDR blocks that IBM Concert Operate uses. Changes to the service CIDR require a corresponding change to the cluster DNS. For more information, see Online installation or Offline installation.
Firewall rules
Some Linux distributions require extra firewall rules to avoid potential conflicts or restrictions.
- Check whether the firewall is enabled.
systemctl status firewalld- If the firewall is enabled, then the command output provides the status of the firewall, including any active rules.
- If the firewall is disabled, then the command outputs the message
No such file or directory.
- If the firewall is enabled, run the following commands:
firewall-cmd -\-permanent -\-add-port=6443/tcp #apiserver firewall-cmd -\-permanent -\-zone=trusted -\-add-source=10.42.0.0/16 #pods firewall-cmd -\-permanent -\-zone=trusted -\-add-source=10.43.0.0/16 #services firewall-cmd -\-reload - If you are permitting the collection of usage data, then ensure that outbound traffic to https://api.segment.io is allowed. For more information, see Updating usage data collection preferences.
nm-cloud-setup service
nm-cloud-setup service must not be enabled. Run the following steps on each of your nodes.- Check if
nm-cloud-setup.serviceis enabled.
If the service is enabled, the command returnssystemctl is-enabled nm-cloud-setup.serviceenabled. - If the service is enabled, then disable it and reboot the node.
systemctl disable nm-cloud-setup.service nm-cloud-setup.timer reboot
High availability considerations
IBM Concert Operate uses a minimum of three control plane nodes and multiple worker nodes to improve resiliency if a node failure occurs.
A procedure is also provided to delete a failed and inaccessible node, for more information, see Removing a failed node.
Geo-redundant multi-cluster support
A multi-cluster geo-redundant disaster recovery solution for IBM Concert Operate is also available. For more information, see Geo-redundant multi-cluster support.
Load balancing
IBM Concert Operate requires a load balancer to distribute incoming traffic among the three control plane nodes. The load balancer is a mandatory component, and if you do not already have a load balancer then you must configure one of your choice.
The load balancer requires the following access:
| Port number | Description |
|---|---|
| 443 | Inbound IBM Concert Operate requests |
| 6443 | Inter-cluster communications: inbound from worker nodes, outbound to control plane nodes |
| (optional) 80 | For inbound IBM Concert Operate requests, redirects to secure port 443. If you disable this port, users must type `https://` as part of the URL when they first navigate to the Concert Operate console. |
The load balancer is the entry point for accessing IBM Concert Operate. It is important to try to ensure the load balancer's high availability, as any downtime or outage will render IBM Concert Operate inaccessible. Consider adding the load balancer to the startup configuration (such as `init.d`) on the server where it is installed. This configuration will enable the load balancer to launch automatically during the boot process, minimizing downtime if an outage occurs.
For more information about load balancers, see the IBM topic What is load balancing?.
The following example shows a configuration for a HAProxy load balancer. This example is provided as a reference only, consider your own internal policies and practices when you create the configuration for your chosen load balancer.
#---- load balancer configuration start ---
frontend aiops-frontend
bind *:443
mode tcp
option tcplog
default_backend aiops-backend
backend aiops-backend
mode tcp
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s
server server-1 10.11.12.105:443 check
server server-2 10.11.12.108:443 check
server server-3 10.11.12.111:443 check
frontend cncf-frontend
bind *:6443
mode tcp
option tcplog
default_backend cncf-backend
backend cncf-backend
mode tcp
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s
server server-1 10.11.12.105:6443 check
server server-2 10.11.12.108:6443 check
server server-3 10.11.12.111:6443 check
frontend aiops-legacy-frontend
bind *:80
mode tcp
option tcplog
default_backend aiops-legacy-backend
backend aiops-legacy-backend
mode tcp
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s
server server-1 10.11.12.105:80 check
server server-2 10.11.12.108:80 check
server server-3 10.11.12.111:80 check
DNS requirements
<load_balancer_ip> aiops-cpd.<your fully-qualified server domain name><load_balancer_ip> cp-console-aiops.<your fully-qualified server domain name>
<load_balancer_ip> is the IP address of your load balancer.The DNS server must also provide name resolution for all of the nodes in the cluster, and for any integration endpoints such as IBM Tivoli Netcool/Impact.
Incoming IBM Concert Operate integrations require additional hostnames, and some IBM Concert Operate integrations use unique or unpredictable hostnames (such as UUIDs) for API access. You can create individual DNS entries for each integration, but this is time-consuming and can cause delays as entries propagate through the DNS system. A better approach is to configure the authoritative DNS server with a wildcard entry that resolves any wildcard cluster hostnames to the cluster's load balancer IP addresses.
whconn-j6iqfugy6sgy27csulfil6vudpea74jp-aiops.apps.mycluster.mycompany.comCreating a single DNS entry for any integration or host in the cluster (for example creating an entry for apps.mycluster.mycompany.com) is more practical. This approach is more efficient and enables new integrations to work without the need for DNS updates.
Security considerations
Ensure that you keep up to date on any security-related news or vulnerabilities with IBM Concert Operate by subscribing to security bulletins. The `aiopsctl` tool must run with root privileges.
For more information about the permissions that are required for IBM Concert Operate, see Permissions.
You can use a custom certificate for IBM Concert Operate instead of the default cluster certificate. For more information, see Using a custom certificate.
Licensing
Review information about the licensing and entitlements for IBM Concert Operate 5.1.5. For more information, see Licensing.