Mapping data from incoming sources

With any connection, you can specify your own mappings to the standard IBM Cloud Pak for Watson AIOps values to optimize search performance for any data type.

For each type of connection, you can edit which fields map to the standard values. By default, each connection type (including custom connections) uses a default field mapping based on what the source data typically looks like. When you map source fields, you're replacing or supplementing those values with something else in your logs.

Mapping data types

Cleaning mapped data by using regular expressions

While you can use the Field mapping section of the connection page to customize field mappings to data values, you can also clean incoming data from your connection here. You can use regular expressions in the mapping field to strip formatting if you know that your data comes formatted in a specific way that affects training. For example, if your log files are always prefixed by the date and time, your training includes that prefix. As a result, IBM Cloud Pak for Watson AIOps flags items that are also prefixed with the same timestamp. The following example illustrates how you can use the Field mapping section to clean your message prefixes.

Note: Using regular expressions in the Field mapping section is not limited to the following example. You can affect change on any mapped field in the same manner.

In this example, your LogDNA output produces the following timestamp at the beginning of every log entry:

Mar  9 03:30:01 TRACE [Controller id=2] Leader imbalance ratio for broker 2 is 0.0 (kafka.controller.KafkaController)

You can add custom_regex immediately following the mapping that you want to affect to exclude entries that match your proposed regular expression. In this example, you want to exclude timestamps that look like Mar 9 03:30:01. You can write a regular expression that follows the formatting of "MMM D HH:MM:SS" to ignore the month value to ensure that your training applies to all months. The following regular expression excludes the timestamp in the example log:


Note: You can use AND (&) and OR operators (|) in combination with regular expressions to exclude more patterns. You can implement these operators in the Field mapping section by including the following content:

  "codec": "logdna",
  "message_field": "message",
  "custom_regex": [
  "log_entity_types": "container",
  "instance_id_field": "_app",
  "timestamp_field": "_ts"

Important: If you previously trained your AI models with a different mapping, you must delete and retrain your models with the new mapping in place. For more information about altering models, see Configuring AI models.