Sample filters for modifying analytics data
Review sample filters to see how to code your own filters for refining IBM® API Connect Analytics data by adding fields, removing fields, or modifying field contents.
You can use these sample filters in the ingestion.filter
and the
external.offload.filter
configurations.
Adding a new field
You can add custom data to API event data by adding a field to the logging document. For example, if you want to search or visualize data that is stored in request and response headers, you can add fields to a filter to include that information.
To avoid naming conflicts with future or current analytics fields, include a prefix that ensures
your new field name is unique. For example, instead of naming the field
employee_num
you might name it x_mycompany_employee_num
.
The following example copies the contents of the X-Employee-Num
field from the
request header and adds it to the x_mycompany_employee_num
field in the event data.
Adding the field to the event data enables you to use the information in visualizations.
if [request_http_headers] {
ruby {
code => "event.get('[request_http_headers]').collect {|i| event.set('[x_mycompany_employee_Num]', i['X-Employee-Num']) if i.has_key?('X-Employee-Num')}"
}
}
Modifying an existing field
Sometimes you don't want to remove a field entirely from your data, you just want to redact sensitive information such as IDs to prevent them from being exposed in visualizations. You can modify the contents of a field and replace information with a symbols or a message.
org_id
, catalog_id
,
space_id
, developer_org_id
, datetime
, and
@timestamp
.The following example replaces sensitive information in the X-Employee-Name
and
X-Employee-ID
request headers with the string:
********sanitized********
.
if [request_http_headers] {
ruby {
code => "headers=['X-Employee-Name','X-Employee-ID']; newHeaders = event.get('[request_http_headers]').collect {|i| headers.each {|header| i[header] = '********sanitized********' if i.has_key?(header)}; i}; event.set('[request_http_headers]', newHeaders)"
}
}
Removing an existing field
Use the mutate remove_field
operation to delete a field from the Analytics data.
To remove multiple fields, delimit the field names with commas.
The following example removes multiple fields (request_http_headers
,
response_http_headers
, request_body
,
response_body
, and query_string
) from the Analytics data:
mutate {
remove_field => ["request_http_headers", "response_http_headers", "request_body", "response_body", "query_string"]
}
Combining filters with offload and ingestion configuration
catalog1
only: offload:
enabled: true
if [catalog_name] == "offload1" {
http {
url => "http://offloadhost.example.com/offload"
http_method => "post"
codec => "json"
content_type => "application/json"
id => "offload_http"
}
}
offload:
enabled: true
output: |
if [catalog_name] == "offload1" {
http {
url => "http://offloadhost.example.com /offload1"
http_method => "post"
codec => "json"
content_type => "application/json"
id => "offload1_http"
}
}
else if [catalog_name] == "offload2" {
http {
url => "http://offloadhost2.example.com /offload2"
http_method => "post"
codec => "json"
content_type => "application/json"
id => "offload2_http"
}
}
offload:
enabled: true
http {
url => "http://offloadhost.example.com /%{[catalog_name]}"
http_method => "post"
codec => "json"
content_type => "application/json"
id => "offload_http"
}
apic
and sandbox
catalogs:offload:
enabled: true
if [catalog_name] !~ /sandbox|apic/ {
http {
url => "http://offloadhost.example.com /%{[catalog_name]}"
http_method => "post"
codec => "json"
content_type => "application/json"
id => "offload_http"
}
}
apic
and sandbox
catalogs: ingestion:
filter: |
if [catalog_name] =~ /sandbox|apic/ {
drop { }
}
Filter and output plugin for splunk
When you create a filter for use with splunk, you must also create an output plugin for it.
if "apicapievent" in [tags] {
ruby {
code => "event.set('[@metadata][newevent]', event.to_json)"
}
}
http {
url => "http://your-domain/services/collector/event"
http_method => "post"
codec => "json"
content_type => "application/json"
id => "offload_http"
format => "message"
message => '{"event": "%{[@metadata][newevent]}","index": "applications-int-ms","sourcetype": "json:apimlogs"}'
headers => ["Authorization", "Splunk XXXX]
}