Sample filters for modifying analytics data

Review sample filters to see how to code your own filters for refining IBM® API Connect Analytics data by adding fields, removing fields, or modifying field contents.

You can use these sample filters in the ingestion.filter and the external.offload.filter configurations.

Adding a new field

You can add custom data to API event data by adding a field to the logging document. For example, if you want to search or visualize data that is stored in request and response headers, you can add fields to a filter to include that information.

To avoid naming conflicts with future or current analytics fields, include a prefix that ensures your new field name is unique. For example, instead of naming the field employee_num you might name it x_mycompany_employee_num.

The following example copies the contents of the X-Employee-Num field from the request header and adds it to the x_mycompany_employee_num field in the event data. Adding the field to the event data enables you to use the information in visualizations.

if [request_http_headers] {
  ruby {
    code => "event.get('[request_http_headers]').collect {|i| event.set('[x_mycompany_employee_Num]', i['X-Employee-Num']) if i.has_key?('X-Employee-Num')}"
  }
}

Modifying an existing field

Sometimes you don't want to remove a field entirely from your data, you just want to redact sensitive information such as IDs to prevent them from being exposed in visualizations. You can modify the contents of a field and replace information with a symbols or a message.

Remember: The following fields should not be modified if the data is being written to internal analytics storage: org_id, catalog_id, space_id, developer_org_id, datetime, and @timestamp.

The following example replaces sensitive information in the X-Employee-Name and X-Employee-ID request headers with the string: ********sanitized********.

if [request_http_headers] {
  ruby {
    code => "headers=['X-Employee-Name','X-Employee-ID']; newHeaders = event.get('[request_http_headers]').collect {|i| headers.each {|header| i[header] = '********sanitized********' if i.has_key?(header)}; i}; event.set('[request_http_headers]', newHeaders)"
  }
}

Removing an existing field

Use the mutate remove_field operation to delete a field from the Analytics data. To remove multiple fields, delimit the field names with commas.

The following example removes multiple fields (request_http_headers, response_http_headers, request_body, response_body, and query_string) from the Analytics data:

mutate {
  remove_field => ["request_http_headers", "response_http_headers", "request_body", "response_body", "query_string"]
}

Filter and output plugin for splunk

When you create a filter for use with splunk, you must also create an output plugin for it.

Filter:

if "apicapievent" in [tags] {
    ruby {
      code => "event.set('[@metadata][newevent]', event.to_json)"
    }
  }

Output plugin:

http {
      url => "http://your-domain/services/collector/event"
      http_method => "post"
      codec => "json"
      content_type => "application/json"
      id => "offload_http"
      format => "message"
      message => '{"event": "%{[@metadata][newevent]}","index": "applications-int-ms","sourcetype": "json:apimlogs"}'
      headers => ["Authorization", "Splunk XXXX]
    }