Learn how to set up log forwarding and collect audit logs that are passed through the Kubernetes API server to IBM Log Analysis to check who initiated a request and when they did so.

As a cluster administrator, by following the simple steps in this blog post, you should be able to answer questions about Kubernetes audit logs, like who initiated a request to delete a Kubernetes resource? When did it happen? On what did it happen?

What are audit logs?

Audit logs allow you to better understand the operations that are initiated by users in your cluster, which can help you troubleshoot issues or report compliance to industry and internal standards.

Although the Kubernetes API server for your cluster is enabled for auditing by default, no auditing data is available until you set up log forwarding. You can forward audit logs for the IBM Cloud Kubernetes Service, the Kubernetes API server and the worker nodes to a logging instance on IBM Cloud.

Prerequisites

• Ensure that you completed the steps to Forward the Kubernetes API audit logs to Log Analysis.
• To quickly set up a Kubernetes cluster on VPC, you can use the Terraform scripts provided in the “Multizone Kubernetes and VPC Load Balancer Setup” post.

Query and decode the logs

You can always enable and launch logging from your Kubernetes cluster’s overview page. By now, you should see the audit logs on the IBM Log Analysis view:

1. To test, set the context for your cluster without the --admin flag: 

   ibmcloud ks cluster config --cluster <CLUSTER ID or NAME>

   Scroll to view full table

   Note: Using the --admin flag will show the cluster-admin context and may not reveal the IAM user overriding the RBAC.

2. Create a namespace with the following command: 

   kubectl create namespace test123

   Scroll to view full table
3. In the Log Analysis UI, enter the following query: 

   verb:create objectRef.name:test123 objectRef.resource:namespaces

   Scroll to view full table

   
   

   So, by now you know which IAM user (who) created the namespace (what) and when was it created. Note: The objectRef.resources is optional and can be any Kubernetes resource (e.g., secrets, configmaps, services, etc.).

4. Similarly, you can delete the namespace with the command kubectl delete namespace test123. Then, from the audit logs, you can quickly find out who (IAM user) deleted the namespace and at what time. The query to decode who deleted the namespace will be as follows: 

   verb:delete objectRef.name:test123 objectRef.resource:namespaces

   Scroll to view full table

   
   

5. To create a custom view out of the query:
   • Click Unsaved View > Save as new view.
   • Enter a Name for your view. 
   • Optionally, you can select a Category and Alert value.
   • Click Save View. Your view is listed under your selected category. If you didn’t select a category, it will be listed under UNCATEGORIZED.
6. To add an alert to your custom view, check the add alert to custom view section of IBM Cloud Kubernetes documentation.

Conclusion

Following the steps in the post, you learned what audit logs are, what the audit logs capture and how to forward and collect the audit logs in IBM Log Analysis to query and decode the logs to understand the operations that are initiated by users in your cluster. 

You can always control user access with IBM Cloud IAM and Kubernetes RBAC. To understand more about Kubernetes auditing and the audit policy, refer to the Kubernetes documentation.

If you have any queries, feel free to reach out to me on Twitter or on LinkedIn.