Planning to modify your incoming analytics data

Review how you can optionally customize the analytics data in your IBM® API Connect analytics deployment before storing it.

You can create filters to customize the analytics API event data before it is sent to internal storage or offloaded to a third-party system. Filters allow you to add new fields, modify existing fields, or remove fields from the data. You can do this globally for the entire analytics subsystem, or you can do it conditionally for a specific API. By default, filters are defined at the global level, so unless you include conditionals, changes are applied to every piece of data flowing through the pipeline.

For information on adding filters to the analytics CR to modify your data, see Kubernetes - Modify incoming analytics data, OpenShift - Modify incoming analytics data.

What fields can you modify?

The fields that are available for you to interact with are described in API event record fields. Each API defined in the Management subsystem and published to your Gateway has its own success and error log policies. The log policy settings determine what data is available for customization. Be sure to verify the existence of a field before trying to modify it; otherwise your pipeline might fail.

Modifying fields is a complex operation and can cause problems with your data. It is highly recommended that you remove fields entirely (instead of modifying them) if you need to sanitize data. If you want to change the format of an existing field, you should create a new, unique field.

Important: Do not change the data type of existing fields or modify fields critical to the analytics subsystem's operations and access control. Doing so can cause problems with retrieving or storing data. The following fields are restricted (note that there might be other fields that are not included in this list):
  • org_id
  • catalog_id
  • space_id
  • developer_org_id
  • datetime
  • @timestamp

The service for the data pipeline is based on Elastic Logstash, which provides a set of predefined filter plugins that you can use for modifying data. However, the plugins are third-party software that IBM does not control, so IBM cannot guarantee support for them.