Example ingestion filters

Review example ingestion filters to see how to add, modify, and delete fields from API event records.

You can use these example filters in the ingestion.filter and the external.offload.filter configurations. For more information about the filter syntax, see the Logstash documentation: https://www.elastic.co/guide/en/logstash/current/filter-plugins.html.

Add a field

You can add custom data to API event data by adding a field.

To avoid naming conflicts with future or current analytics fields, include a prefix that ensures your new field name is unique. For example, instead of naming the field employee_num you might name it x_mycompany_employee_num.

The following example copies the contents of the X-Employee-Num field from the request header and adds it to the x_mycompany_employee_num field.

if [request_http_headers] {
  ruby {
    code => "event.get('[request_http_headers]').collect {|i| event.set('[x_mycompany_employee_Num]', i['X-Employee-Num']) if i.has_key?('X-Employee-Num')}"
  }
}

Modifying an existing field

Sometimes you don't want to remove a field entirely from your data, but you want to redact sensitive information such as IDs. You can modify the contents of a field and replace information with symbols or a message.

Remember: The following fields must not be modified if the data is being written to internal analytics storage: org_id, catalog_id, space_id, developer_org_id, datetime, and @timestamp.

The following example replaces sensitive information in the X-Employee-Name and X-Employee-ID request headers with the string: ********sanitized********.

if [request_http_headers] {
  ruby {
    code => "headers=['X-Employee-Name','X-Employee-ID']; newHeaders = event.get('[request_http_headers]').collect {|i| headers.each {|header| i[header] = '********sanitized********' if i.has_key?(header)}; i}; event.set('[request_http_headers]', newHeaders)"
  }
}

Removing an existing field

Use the mutate remove_field operation to delete a field. To remove multiple fields, delimit the field names with commas.

The following example removes multiple fields (request_http_headers, response_http_headers, request_body, response_body, and query_string) from the API event record:

mutate {
  remove_field => ["request_http_headers", "response_http_headers", "request_body", "response_body", "query_string"]
}

Drop API event records

If you want certain API events to not be stored at all, use the drop syntax.

Do not store events from the apic and sandbox catalogs:

if  [catalog_name] =~ /sandbox|apic/ {
  drop { }
}