Sample filters for modifying analytics data

Review sample filters to see how to code your own filters for refining IBM® API Connect Analytics data by adding fields, removing fields, or modifying field contents.

You can use these sample filters in the ingestion.filter and the external.offload.filter configurations.

Adding a new field

You can add custom data to API event data by adding a field to the logging document. For example, if you want to search or visualize data that is stored in request and response headers, you can add fields to a filter to include that information.

To avoid naming conflicts with future or current analytics fields, include a prefix that ensures your new field name is unique. For example, instead of naming the field employee_num you might name it x_mycompany_employee_num.

The following example copies the contents of the X-Employee-Num field from the request header and adds it to the x_mycompany_employee_num field in the event data. Adding the field to the event data enables you to use the information in visualizations.

if [request_http_headers] {
  ruby {
    code => "event.get('[request_http_headers]').collect {|i| event.set('[x_mycompany_employee_Num]', i['X-Employee-Num']) if i.has_key?('X-Employee-Num')}"
  }
}

Modifying an existing field

Sometimes you don't want to remove a field entirely from your data, you just want to redact sensitive information such as IDs to prevent them from being exposed in visualizations. You can modify the contents of a field and replace information with a symbols or a message.

Remember: The following fields should not be modified if the data is being written to internal analytics storage: org_id, catalog_id, space_id, developer_org_id, datetime, and @timestamp.

The following example replaces sensitive information in the X-Employee-Name and X-Employee-ID request headers with the string: ********sanitized********.

if [request_http_headers] {
  ruby {
    code => "headers=['X-Employee-Name','X-Employee-ID']; newHeaders = event.get('[request_http_headers]').collect {|i| headers.each {|header| i[header] = '********sanitized********' if i.has_key?(header)}; i}; event.set('[request_http_headers]', newHeaders)"
  }
}

Removing an existing field

Use the mutate remove_field operation to delete a field from the Analytics data. To remove multiple fields, delimit the field names with commas.

The following example removes multiple fields (request_http_headers, response_http_headers, request_body, response_body, and query_string) from the Analytics data:

mutate {
  remove_field => ["request_http_headers", "response_http_headers", "request_body", "response_body", "query_string"]
}

Combining filters with offload and ingestion configuration

Offload events from catalog1 only:
   offload:
     enabled: true
     if  [catalog_name] == "offload1" {
          http {
            url => "http://offloadhost.example.com/offload"
            http_method => "post"
            codec => "json"
            content_type => "application/json"
            id => "offload_http"
          }
        }
Offload to different locations based on catalog name:
offload:
  enabled: true
  output: |
    if  [catalog_name] == "offload1" {
      http {
        url => "http://offloadhost.example.com /offload1"
        http_method => "post"
        codec => "json"
        content_type => "application/json"
        id => "offload1_http"
      }
    }
    else if  [catalog_name] == "offload2" {
      http {
        url => "http://offloadhost2.example.com /offload2"
        http_method => "post"
        codec => "json"
        content_type => "application/json"
        id => "offload2_http"
      }
    }
Use the catalog name as a variable in the offload location:
 offload:
   enabled: true
   http {
     url => "http://offloadhost.example.com /%{[catalog_name]}"
     http_method => "post"
     codec => "json"
     content_type => "application/json"
     id => "offload_http"
   }

Do not offload events from apic and sandbox catalogs:
offload:
   enabled: true
   if  [catalog_name] !~ /sandbox|apic/ {
     http {
       url => "http://offloadhost.example.com /%{[catalog_name]}"
       http_method => "post"
       codec => "json"
       content_type => "application/json"
       id => "offload_http"
     }
   }
Do not store events from apic and sandbox catalogs:
  ingestion:
    filter: |
      if  [catalog_name] =~ /sandbox|apic/ {
        drop { }
      }

Filter and output plugin for splunk

When you create a filter for use with splunk, you must also create an output plugin for it.

Filter:
if "apicapievent" in [tags] {
    ruby {
      code => "event.set('[@metadata][newevent]', event.to_json)"
    }
  }
Output plugin:
http {
      url => "http://your-domain/services/collector/event"
      http_method => "post"
      codec => "json"
      content_type => "application/json"
      id => "offload_http"
      format => "message"
      message => '{"event": "%{[@metadata][newevent]}","index": "applications-int-ms","sourcetype": "json:apimlogs"}'
      headers => ["Authorization", "Splunk XXXX]
    }