Performance considerations for logs data collection

When you configure an integration in IBM Cloud Pak® for AIOps that collects logs data, such as Custom, Elk, Falcon LogScale, Kafka, Mezmo, and Splunk Enterprise, it is important to consider the resource requirements for creating a new integration.

Consider the following actions to help ensure that your integration works as expected:

• Use filters when possible to reduce the risk of exceeding the limitations of the API or integration. The filters available for each integration are documented in the topic for each integration.
• Each integration needs a unique API token. Each API token has its own requests limit. If integrations share API tokens, then they are at a risk of exceeding the request limits.
• Monitor the of the number of logs integrations that are created and the size of the Cloud Pak for AIOps installation. The amount of logs data collected per interval must not exceed the limitations of log anomaly detection. For more information, see Custom sizing.

Cloud Pak for AIOps applies various analytics and transformations to the stream of log data. For scaling to high-volume workloads, the data stream is parallelized using the concept of slots. Each slots consists of a given unit of CPU and memory. Each integration requires one or more slots dependencing on the base parallelism setting, data collection setting, and the data volume. Additional slots are required to support high availability (failover) scenarios, which are available only on large installations of Cloud Pak for AIOps.

When creating a new integration, make sure that there are enough slots on Cloud Pak for AIOps to support the logs integration.