Removing User Data
Data protection laws and regulations, such as the General Data Protection Regulation (GDPR) might require specific handling of user data, even after a user profile is removed. Additionally, employees or other clients with user accounts on webMethods API Gateway may request that any user identifying information such as user name, email addresses, or client IP addresses be removed from webMethods API Gateway. To comply with data protection requirements and user requests, in addition to deleting the user account, you may need to complete activities such as deleting or masking the user data.
Types of Data and their stores in webMethods API Gateway
- Core data
This consists of APIs, policies, applications, aliases, packages, plans, administration configurations, users, and groups. This is stored in external Elasticsearch and Integration Server store.
- Runtime
transactions
This consists of the transaction events, monitoring events, error events, policy violation events, threat protection events, lifecycle events and performance metrics. This information can be stored in Elasticsearch or external destinations.
- Application
logs
This consists of UI, server, Elasticsearch, Kibana, and Platform logs . This is stored in filesystem and Elasticsearch.
-
Audit logs
This is stored in Elasticsearch.
-
Tracer logs
This consists of the runtime transactions that are captured when you trace the API. This information can be stored in Elasticsearch.
Handling Core Data
webMethods API Gateway strongly recommends the use of internal or technical user for policy configurations like Authorize user, Invoke IS Service and Outbound Authentication that has user credentials. This avoids the life cycle of actual or real user object from impacting the webMethods API Gateway configurations.
In case, real users are used in policy configurations, then the API Provider or API Administrators should take the responsibility of changing the configurations once the user is deleted from the system.
Handling Application Logs
In webMethods API Gateway, as you increase the log level to Debug or Trace, there can be messages that include the userid. So when a user has to be deleted from the system, Administrators have to clean up this data. The application logs are stored in file system till webMethods API Gateway version 10.2. From webMethods API Gateway 10.3, API Administrators have an option to persist the logs additionally in Elasticsearch.
The following is a sample query run to mask the user data in the application log stored either in the Elasticsearch.
curl -X POST -H 'Accept: application/json' -H 'Content-Type: application/json'
http://hostname:port/gateway_default_log/doc/_update_by_query -d ' {
"script": { "id": "findAndReplace", "params": {
"find": "user123", "replace": "******"} } }
- hostname:port refers to the host name and port of the system where the Elasticsearch or the external data store that contains the data resides.
- The Find field contains the data or user information, such as username, id, that is to be masked.
- The field Replace contains the string that is used to mask the data mentioned in the Find field and replaces the data with the string provided in the logs.
- Install Directory\Integration Server\instances\Instance_Name\logs\server.log. For details to cleanup other Integration Server related logs, see IBM webMethods Integration Server Administrator’s Guide.
- Install Directory\Integration Server\instances\Instance_Name\logs\APIGateway.log.
- Install Directory\logs.
Handling Audit Logs
If enabled, audit logs would have information about user actions. This contains user reference and has to be cleaned up after user deletion. You can achieve the cleanup in the following ways:
- As an Administrator, you can clean up the
audit logs persisted in Elasticsearch by running the following curl query to
replace or mask the desired data.
The following is a sample query run to mask the user data in the Audit logs stored in the Elasticsearch.
curl -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' http://hostname:port/gateway_default_audit_auditlogs/_update_by_query -d ' { "script": { "id": "findAndReplace", "params": { "find": "user123", "replace": "******"} } } '
- hostname:port refers to the host name of the system where the Elasticsearch resides and the corresponding port.
- The Find field contains the data or user information, such as username, id, that is to be masked.
- The field Replace contains the string that is used to mask the data mentioned in the Find field and replaces the data with the string provided in the logs.
- As an Administrator, you can use the extended setting saveAuditlogsWithPayload to not store the request payloads in the audit logs. Though this does not completely eliminate the user information in audit logs it certainly minimizes the occurrences. Use this property with caution, as enabling this option might result in less audit data being captured.
Handling Tracer Logs
If you enable the tracer, the data that webMethods API Gateway captures might have user-specific information. You can either mask or remove the user data from the server log trace span, mediator trace span, and request response trace span indices. You can either mask full or partial text in the user data.
The following sample query is used to mask the user data in the server log trace span:
curl -X POST http(s)://elasticsearch_hostname:elasticsearch_port/
gateway_tenant_serverlogtracespans/_doc/_update_by_query?pretty -H 'Content-Type: application/json' -d'
{
"script": {
"id": "findAndReplaceInTracerData",
"params": {
"find": "textToFind",
"replace": "textToReplace"
}
}
}'
The following sample query is used to mask the user data in the mediator trace span:
curl -X POST http(s)://elasticsearch_hostname:elasticsearch_port/
gateway_tenant_mediatortracespan/_doc/_update_by_query?pretty -H 'Content-Type: application/json' -d'
{
"script": {
"id": "findAndReplaceInTracerData",
"params": {
"find": "textToFind",
"replace": "textToReplace"
}
}
}
'
The following sample query is used to mask the user data in the request response trace span:
curl -X POST http(s)://elasticsearch_hostname:elasticsearch_port/
gateway_tenant_requestresponsetracespans/_doc/_update_by_query?pretty -H 'Content-Type: application/json' -d'
{
"script": {
"id": "findAndReplaceInTracerData",
"params": {
"find": "textToFind",
"replace": "textToReplace"
}
}
}
'
- The Find field contains the data or user information, such as username, id, that is to be masked.
- The field Replace contains the string that is used to mask the data mentioned in the Find field and replaces the data with the string provided in the server log trace span, mediator trace span, and request response trace span indices.