Removing User Data

Types of Data and their stores in webMethods API Gateway

• Core data

  This consists of APIs, policies, applications, aliases, packages, plans, administration configurations, users, and groups. This is stored in external Elasticsearch and Integration Server store.

• Runtime transactions

  This consists of the transaction events, monitoring events, error events, policy violation events, threat protection events, lifecycle events and performance metrics. This information can be stored in Elasticsearch or external destinations.

• Application logs

  This consists of UI, server, Elasticsearch, Kibana, and Platform logs . This is stored in filesystem and Elasticsearch.

• Audit logs

  This is stored in Elasticsearch.

• Tracer logs

  This consists of the runtime transactions that are captured when you trace the API. This information can be stored in Elasticsearch.

Handling Core Data

webMethods API Gateway strongly recommends the use of internal or technical user for policy configurations like Authorize user, Invoke IS Service and Outbound Authentication that has user credentials. This avoids the life cycle of actual or real user object from impacting the webMethods API Gateway configurations.

In case, real users are used in policy configurations, then the API Provider or API Administrators should take the responsibility of changing the configurations once the user is deleted from the system.

Handling Application Logs

In webMethods API Gateway, as you increase the log level to Debug or Trace, there can be messages that include the userid. So when a user has to be deleted from the system, Administrators have to clean up this data. The application logs are stored in file system till webMethods API Gateway version 10.2. From webMethods API Gateway 10.3, API Administrators have an option to persist the logs additionally in Elasticsearch.

The following is a sample query run to mask the user data in the application log stored either in the Elasticsearch.

curl -X POST -H 'Accept: application/json' -H 'Content-Type: application/json'
http://hostname:port/gateway_default_log/doc/_update_by_query -d ' {
"script": { "id": "findAndReplace", "params": {
"find": "user123", "replace": "******"} } } 

• hostname:port refers to the host name and port of the system where the Elasticsearch or the external data store that contains the data resides.
• The Find field contains the data or user information, such as username, id, that is to be masked.
• The field Replace contains the string that is used to mask the data mentioned in the Find field and replaces the data with the string provided in the logs.

Handling Audit Logs

If enabled, audit logs would have information about user actions. This contains user reference and has to be cleaned up after user deletion. You can achieve the cleanup in the following ways:

• As an Administrator, you can clean up the audit logs persisted in Elasticsearch by running the following curl query to replace or mask the desired data.

  The following is a sample query run to mask the user data in the Audit logs stored in the Elasticsearch.

  curl -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' 
  http://hostname:port/gateway_default_audit_auditlogs/_update_by_query -d ' { 
  "script": { "id": "findAndReplace", "params": { 
  "find": "user123", "replace": "******"} } } '

  • hostname:port refers to the host name of the system where the Elasticsearch resides and the corresponding port.
  • The Find field contains the data or user information, such as username, id, that is to be masked.
  • The field Replace contains the string that is used to mask the data mentioned in the Find field and replaces the data with the string provided in the logs.
• As an Administrator, you can use the extended setting saveAuditlogsWithPayload to not store the request payloads in the audit logs. Though this does not completely eliminate the user information in audit logs it certainly minimizes the occurrences. Use this property with caution, as enabling this option might result in less audit data being captured.

Handling Tracer Logs

If you enable the tracer, the data that webMethods API Gateway captures might have user-specific information. You can either mask or remove the user data from the server log trace span, mediator trace span, and request response trace span indices. You can either mask full or partial text in the user data.

The following sample query is used to mask the user data in the server log trace span:

curl -X POST http(s)://elasticsearch_hostname:elasticsearch_port/
gateway_tenant_serverlogtracespans/_doc/_update_by_query?pretty -H 'Content-Type: application/json' -d'
{
"script": {
  "id": "findAndReplaceInTracerData",
    "params": {
      "find": "textToFind",
      "replace": "textToReplace"
    }
  }
}'

The following sample query is used to mask the user data in the mediator trace span:

curl -X POST http(s)://elasticsearch_hostname:elasticsearch_port/
gateway_tenant_mediatortracespan/_doc/_update_by_query?pretty -H 'Content-Type: application/json' -d'
{
"script": {
  "id": "findAndReplaceInTracerData",
    "params": {
      "find": "textToFind",
      "replace": "textToReplace"
    }
  }
}
'

The following sample query is used to mask the user data in the request response trace span:

curl -X POST http(s)://elasticsearch_hostname:elasticsearch_port/
gateway_tenant_requestresponsetracespans/_doc/_update_by_query?pretty -H 'Content-Type: application/json' -d'
{
"script": {
  "id": "findAndReplaceInTracerData",
    "params": {
      "find": "textToFind",
      "replace": "textToReplace"
    }
  }
}
'

• The Find field contains the data or user information, such as username, id, that is to be masked.
• The field Replace contains the string that is used to mask the data mentioned in the Find field and replaces the data with the string provided in the server log trace span, mediator trace span, and request response trace span indices.