Learn how to set up log forwarding and collect audit logs that are passed through the Kubernetes API server to IBM Log Analysis to check who initiated a request and when they did so.

As a cluster administrator, by following the simple steps in this blog post, you should be able to answer questions about Kubernetes audit logs, like who initiated a request to delete a Kubernetes resource? When did it happen? On what did it happen?

What are audit logs?

Audit logs allow you to better understand the operations that are initiated by users in your cluster, which can help you troubleshoot issues or report compliance to industry and internal standards.

Although the Kubernetes API server for your cluster is enabled for auditing by default, no auditing data is available until you set up log forwarding. You can forward audit logs for the IBM Cloud Kubernetes Service, the Kubernetes API server and the worker nodes to a logging instance on IBM Cloud.


Query and decode the logs

You can always enable and launch logging from your Kubernetes cluster’s overview page. By now, you should see the audit logs on the IBM Log Analysis view:

  1. To test, set the context for your cluster without the --admin flag: 
    ibmcloud ks cluster config --cluster <CLUSTER ID or NAME>
    Scroll to view full table

    Note: Using the --admin flag will show the cluster-admin context and may not reveal the IAM user overriding the RBAC.

  2. Create a namespace with the following command: 
    kubectl create namespace test123
    Scroll to view full table
  3. In the Log Analysis UI, enter the following query: 
    verb:create objectRef.name:test123 objectRef.resource:namespaces
    Scroll to view full table

    So, by now you know which IAM user (who) created the namespace (what) and when was it created. Note: The objectRef.resources is optional and can be any Kubernetes resource (e.g., secrets, configmaps, services, etc.).

  4. Similarly, you can delete the namespace with the command kubectl delete namespace test123. Then, from the audit logs, you can quickly find out who (IAM user) deleted the namespace and at what time. The query to decode who deleted the namespace will be as follows: 
    verb:delete objectRef.name:test123 objectRef.resource:namespaces
    Scroll to view full table

  5. To create a custom view out of the query:
    • Click Unsaved View > Save as new view.
    • Enter a Name for your view. 
    • Optionally, you can select a Category and Alert value.
    • Click Save View. Your view is listed under your selected category. If you didn’t select a category, it will be listed under UNCATEGORIZED.
  6. To add an alert to your custom view, check the add alert to custom view section of IBM Cloud Kubernetes documentation.


Following the steps in the post, you learned what audit logs are, what the audit logs capture and how to forward and collect the audit logs in IBM Log Analysis to query and decode the logs to understand the operations that are initiated by users in your cluster. 

You can always control user access with IBM Cloud IAM and Kubernetes RBAC. To understand more about Kubernetes auditing and the audit policy, refer to the Kubernetes documentation.

If you have any queries, feel free to reach out to me on Twitter or on LinkedIn

More from Cloud

Strengthening cybersecurity in life sciences with IBM and AWS

7 min read - Cloud is transforming the way life sciences organizations are doing business. Cloud computing offers the potential to redefine and personalize customer relationships, transform and optimize operations, improve governance and transparency, and expand business agility and capability. Leading life science companies are leveraging cloud for innovation around operational, revenue and business models. According to a report on mapping the cloud maturity curve from the EIU, 48% of industry executives said cloud has improved data access, analysis and utilization, 45% say cloud…

7 min read

Kubernetes version 1.27 now available in IBM Cloud Kubernetes Service

< 1 min read - We are excited to announce the availability of Kubernetes version 1.27 for your clusters that are running in IBM Cloud Kubernetes Service. This is our 22nd release of Kubernetes. With our Kubernetes service, you can easily upgrade your clusters without the need for deep Kubernetes knowledge. When you deploy new clusters, the default Kubernetes version remains 1.25 (soon to be 1.26); you can also choose to immediately deploy version 1.27. Learn more about deploying clusters here. Kubernetes version 1.27 In…

< 1 min read

Redefining the consumer experience: Diageo partners with SAP and IBM on global digital transformation

3 min read - In an era of evolving consumer preferences and economic uncertainties, the beverage industry stands as a vibrant reflection of changing trends and shifting priorities. Despite the challenges posed by inflation and the cost-of-living crisis, a dichotomy has emerged in consumer behavior, where individuals untouched by the crisis continue to indulge in their favorite beverages, while those directly affected pivot towards more affordable luxuries, such as a bottle of something special. This intriguing juxtaposition highlights the resilient nature of consumers and…

3 min read

IBM Cloud releases 2023 IBM Cloud for Financial Services Agreed-Upon Procedures (AUP) Report

2 min read - IBM Cloud completed its 2023 independent review of IBM Cloud services and processes. The review report demonstrates to its clients, partners and other interested parties that IBM Cloud services have implemented and adhere to the technical, administrative and physical control requirements of IBM Cloud Framework for Financial Services. What is the IBM Cloud Framework for Financial Services? IBM Cloud for Financial Services┬« is designed to build trust and enable a transparent public cloud ecosystem with features for security, compliance and…

2 min read