Creating custom integrations

Adding a Custom integration

1. Log in to IBM Cloud Pak for AIOps console.

2. Expand the navigation menu (four horizontal bars), then click Define > Integrations.

3. On the Integrations page, click Add integration.

4. From the list of available integrations, find and click the Custom tile.

   Note: If you do not immediately see the integration that you want to create, you can filter the tiles by type of integration. Click the type of integration that you want in the Category section.

5. On the side-panel, review the instructions and when ready to continue, click Get started.

   

Specifying integration parameters

1. On the Add integration page enter the following integration information:

   • Name: The display name of your integration.

   • Description: An optional description for the integration.

   • Extract data from an array of log lines embedded in JSON: When enabled, logs are extracted from an array that is embedded within a JSON object that is returned by the REST service. This option works only for an array of flat JSON objects that is embedded within a JSON object.

     

   • Embedded array path: Type the path to the array from which you want to extract log data. The following examples, show how to configure the array path:

     {“results”: [{“message”: “myMessage”}]} would be results[*]
     {“results”: {“events”: [{“message”: “myMessage”}]}} would be results.events[*]
     {“results”: [{“events”: [{“message”: “myMessage”}]}]} would be results[0].events[*]
     

   • REST method: The method by which your data gets transferred, either GET or POST. If you select POST a POST JSON (Optional) field appears also.

     

   • Authentication type: Select one of the following values:

     • User ID/password: The user ID and password that is required for accessing the REST service. You must enter both in the integration configuration. The HTTPS authentication header is sent to the Service URL in the format 'Authorization: Basic MyCredential'. 'MyCredential' is base64 encoded and in the format 'username:password'.

     • API key: The REST service is authenticated with an API key. The HTTPS authentication header is sent to the Service URL in the format 'Authorization: MyAPIKey'.

     • Token: The REST service is authenticated with a temporary token. The HTTPS authentication header is sent to the Service URL in the format 'Authorization: Bearer MyToken'.

     • Query Key: The REST service is authenticated with a query key. The HTTPS authentication header is sent to the Service URL in the format X-Query-Key: MyQueryKey.

     • Custom: Your custom REST service has specific authentication requirements that are not covered by the other authentication methods. You must enter both a Header and Value to configure your authentication. Note, both the Header and Value are plain text properties that are case-sensitive and does not remove spaces. The HTTPS authentication header is sent to the Service URL in the format 'Authorization: MyHeader MyValue'.

     • None: Your custom REST service requires no authentication. No Authorization header is sent to the target system.

   • Certificate (optional): Certificate used to verify the SSL/TLS connection to the REST service.

   • UI URL: The hostname or IP address of the UI of your custom source.

   • Service URL: The URL for the service that you want to integrate. For more information about customizing your Service URL, see Customizing GET and POST integrations

     

   • Base parallelism:It is recommended to use a higher value than 1 so that you can process data in parallel. In a small environment the available flinks slots is 16 and for a large environment, the maximum available slots is 32.

   • Sampling rate: The rate at which data is pulled from live source (in seconds). The default value is 60.

   • JSON processing option: Select a JSON processing option.

     • None: The default option. The JSON is not processed or modified.

     • Flatten: This option flattens the JSON object by removing the opening and closing braces.

     • Filter: This option extracts the JSON object and replaces it with an empty string.

     • For more information about the options, see Managing embedded JSON.

     Note: To improve data throughput, you can increase the base parallelism value incrementally. For more information about maximum base parallelism for starter and production deployments, see Improving data streaming performance for log anomaly detection.

2. Click Test connection.

3. Click Next to move to the next page.

Specifying field mapping

1. On the Field mapping page you can improve search performance by mapping the fields from your implementation fields to standard fields within IBM Cloud Pak for AIOps.

   • For more information about how field mappings are defined, see Mapping data from incoming sources.
   • For more information about using mappings to clean your data for use in IBM Cloud Pak for AIOps, see Cleaning mapped data using regular expressions.

   The following code snippet displays an example of field mapping that uses the supported format. When you are coding your mapping, use this example to help you.

   {
     "codec": "custom",
     "message_field": "message",
     "log_entity_types": "kubernetes.container_name",
     "instance_id_field": "_app",
     "timestamp_settings":{
        "timestamp_field":"_ts", <MANDATORY>
        "multiplier": 1000, <OPTIONAL>
        "pattern": "yyyy-MM-dd hh:mm:ss z" <OPTIONAL, SPECIFIC TO RAW DATA OUTPUT>
     }
   }
   

   If you are configuring an integration to WebSphere, then you must use the following mapping. In this mapping, all fields must be specified exactly as shown, except for the log_entity_types parameter. This can be empty, as shown in the following example.

   {
   "codec": "custom",
   "rolling_time": 10,
   "instance_id_field": "ibm_serverName",
   "log_entity_types": "",
   "message_field": "message",
   "timestamp_settings": {
      "timestamp_field": "ibm_datetime",
      "pattern": "yyyy-MM-dd'T'HH:mm:ss.SSSZZZZZ"
                         }
   }
   

   For more information about specific custom mappings, see Custom mappings.

   Alternatively, if you want to include specific fields from your WebSphere record within the log alert resource field to be generated by the log anomaly detection algorithms, then specify those WebSphere fields in the log_entity_types parameter, as shown in the example below.

   {
   "codec": "custom",
   "rolling_time": 10,
   "instance_id_field": "ibm_serverName",
   "log_entity_types": "host,ibm_userDir,ibm_serverName,module",
   "message_field": "message",
   "timestamp_settings": {
      "timestamp_field": "ibm_datetime",
      "pattern": "yyyy-MM-dd'T'HH:mm:ss.SSSZZZZZ"
                         }
   }
   

   For more information about the log alerts generated by the log anomaly detection algorithms, see the following topics:

   • Natural language examples
   • Statistical baseline examples

2. Click Next to move to the next page.

Specifying how log data is collected for AI training

1. On the AI training and log data page, select how you want to manage collecting data for use in AI training and anomaly detection. Click the Data collection toggle to turn on data collection then select how you want to collect data:

   • Live data for continuous AI training and anomaly detection: A continuous collection of data from your integration is used to both train AI models and analyze your data for anomalous behavior.

     Note: After an initial installation, there is no data at all in the system. If you select this option, then the two different log anomaly detection algorithms behave in the following ways:

     • Natural language log anomaly detection does not initially detect anomalies as no model has been trained. If Live data for continuous AI training and anomaly detection is set to on, the system gathers training data live and after a few days there is enough data to train a model. When this model is deployed, then it detects anomalies as normal.

     • Statistical baseline log anomaly detection does not detect anomalies for the first 30 minutes of data collection. This is because it does not have a baseline yet. After 30 minutes of live data collection the baseline is automatically created. After that it detects anomalies on an ongoing basis, while continuing to gather data and improve its model every 30 minutes.

   • Live data for initial AI training: A single set of training data used to define your AI model. Data collection takes place over a specified time period that starts when you create your integration.

     Note: Selecting this option causes the system to continue to collect data while the option is enabled; however, the data is collected for training only, and not for log anomaly detection. For more information about AI model training, including minimum and ideal data quantities, see Configuring AI training.

     

2. Important: Keep in mind the following considerations when you select your data collection type:

   • Anomaly detection for your integration occurs if you select Live data for continuous AI training and anomaly detection.
   • Custom integrations cannot use historical data for AI training.
   • Different types of AI models have different requirements to properly train a model. Make sure that your settings satisfy minimum data requirements. For more information about how much data you need to train different AI models, see Configuring AI training.

3. Click Next.

4. On the Resource requirements page, you can review the slot usage for your log integrations to see if there are enough slots to fully support the integration for multizone high availability.

   If you set the Data collection toggle to On, you will see the resource management overview.

   • If your current usage and other usage are less than the provisioned slots, but the HA slots exceed the provisioned slots, you will be able to create the integration, but will see a warning that you do not have enough slots. The integration will not have multizone high availability.

   • If your projected usage exceeds the provisioned slots, you will not be able to create the integration because you do not have enough slots on your system for log data integrations.

   • If your total slots, including HA slots, are within the provisioned slots, the integration will have multizone high availability.

     Note: HA operation assumes high availability for three zones.

   If you set the Data collection toggle to Off, you will see a message stating that you need to enable logs data collection to see the resource management overview. When data collection is off, no slots are used by that integration.

5. Click Done.

You have created a custom integration in your instance. After you create your integration, you must enable the data collection to connect your integration with the AI of IBM Cloud Pak for AIOps. For more information about enabling your integration, see Enabling custom integrations.

To create more integrations (such as a ChatOps integration), see Configuring Integrations.

For more information about working with the insights provided by your integrations, see ChatOps insight management.

GET integrations

If you customize the Service URL of your integration, you must include the start and end time in the URL with properly formatted timestamps (SimpleDateFormat). For epoch timestamps, milliseconds and seconds are supported. The following timestamp identifiers are supported in the Service URL field:

• <start-epoch-ms>, <end-epoch-ms>. For example, http://exampleurl.com/v1/export?to=<end-epoch-ms>&from=<start-epoch-ms>.
• <start-epoch-s>, <end-epoch-s>. For example, http://exampleurl.com/v1/export?to=<end-epoch-s>&from=<start-epoch-s>.
• <start-SimpleDateFormat>, <end-SimpleDateFormat>. For example, http://exampleurl.com/v1/export?to=<end-yyyy-MM-dd'T'HH:mm:ss.SSSXXX>&from=<start-yyyy-MM-dd'T'HH:mm:ss.SSSXXX>.

POST integrations

For POST integrations, you do not need to customize the Service URL value. For these integrations, the payload is in the body of the JSON and you can customize the format, but must include the start and end times with properly formatted timestamps (SimpleDateFormat). For epoch timestamps, milliseconds and seconds are supported. The following timestamp identifiers are supported in the Service URL field:

• <start-epoch-ms>, <end-epoch-ms>. For example, {"queryString":"exampleQuery","start":"<start-epoch-ms>", "end":"<end-epoch-ms>","isLive":false}
• <start-epoch-s>, <end-epoch-s>. For example, {"queryString":"exampleQuery","start":"<start-epoch-s>", "end":"<end-epoch-s>","isLive":false}
• <start-SimpleDateFormat>, <end-SimpleDateFormat>. For example, {"query": {"range": {"@timestamp": {"gte": "<start-yyyy-MM-dd'T'HH:mm:ss.SSSXXX>","lt": "<end-yyyy-MM-dd'T'HH:mm:ss.SSSXXX>"}}},"size": 10000}

Custom mapping with String type timestamps

When timestamps are represented as Strings, they must be parsed and converted to 13-digit UNIX timestamps. In this case, the pattern field must contain the parsing pattern. For example, if the timestamp is 2020-12-31 23:59:59 PST, the integration mapping can be defined as follows:

{
    "rolling_time": 10,
    "instance_id_field": "_account",
    "log_entity_types": "kubernetes.container_name",
    "message_field": "_line",
    "timestamp_settings":{
      "timestamp_field":"_ts",
      "pattern": "yyyy-MM-dd hh:mm:ss z"
    }
}

Custom mapping with Double type timestamps

It might occur that timestamps are not represented in milliseconds. In this case, you must multiply or divide the input timestamps to obtain the milliseconds. The following example illustrates how you can use the multiplier field to obtain milliseconds.

Consider the following source timestamp without milliseconds:

{
   "_ts": 1569873609.539,
   "_account": "...",
   "_cluster": "...",
   "_host": "...",
   "_logtype": "...",
   "_line": "...",
   "_id": "...",
   "__key": "...",
}

You can set the multiplier field in the mapping to show the timestamps in milliseconds:

{
    "rolling_time": 10,
    "instance_id_field": "_account",
    "log_entity_types": "kubernetes.container_name",
    "message_field": "_line",
    "timestamp_settings":{
      "timestamp_field":"_ts",
      "multiplier": 1000
    }
}

Custom mapping with static instance ID support

You can input a static value for the instance_id_field in the IBM Cloud Pak for AIOps pipeline. See the following example:

{
   "rolling_time":10,
   "instance_id_field":"%aStaticInstanceId%",
   "log_entity_types":"kubernetes.container_name",
   "message_field":"_line",
   "timestamp_settings":{
      "timestamp_field":"_ts",
      "pattern":"yyyy-MM-dd hh:mm:ss z"
   }
}

The custom mapping places the value that is contained between %...% in the instance_id_field field throughout the IBM Cloud Pak for AIOps data collection. The following example illustrates the resulting data from the preceding custom mapping:

{
   "message":"exampleLogMessage",
   "instance_id":"aStaticInstanceId",
   "entities":{
      "pod":"examplePod",
      "_cluster":"exampleCluster",
      "container":"exampleContainer"
   },
   ...,
   "timestamp":1605195377154
}

Supported custom-mapping fields