Configuring audit logging

K3s is a Kubernetes distribution that provides audit logging, which is disabled by default. As an administrator, you can enable audit logging on K3s-based IBM® Security Edge Gateway cluster.

Before you begin

Your Edge Gateway must be version 1.8.1 or later to enable K3s auditing.

About this task

The configuration that is required for audit logging includes:

  • An audit policy definition file, to which events are logged.
  • Audit backend configuration, for collecting and storing the logs.
All of the additional configurations are created under /var/lib/rancher/k3s/server/manifests/ and the logs are output to /var/log/kubernetes/audit/ as follows:
sudo mkdir -p /var/lib/rancher/k3s/server/manifests /var/log/kubernetes/audit

Procedure

  1. Create your audit policy file and define the policy rules.

    The following example records only the deployment creations, without the request/response body:

    cat >/var/lib/rancher/k3s/server/manifests/policy.yaml <<EOF
# Log only deployment creations at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    verbs: ["create"]
    resources:
    - group: "apps"
      resources: ["deployments"]
EOF
    For more information about how to create the audit policy, see the Kubernetes documentation.
  2. Choose the audit backend that you want from the two available:
    • Log backend: which writes events into the file system.
    • Webhook backend: which sends events to an external HTTP API.
    The backend is configured in K3s’ systemd unit configuration.
  3. Depending on which audit backend you are using, complete one of the following procedures.
    To enable audit logging with the local file system log backend, you must add the related configurations into K3s systemd starting arguments, reload the configuration, and restart K3s service, as follows:
    1. In the k3s.service configuration file, under the Service section, append options in the ExecStart:
      vi /etc/systemd/system/k3s.service

      The following two options are required and both must be passed by --kube-apiserver-arg=option-name=option-value:

      • audit-policy-file is the path of audit policy, as described in the first step.

      • audit-log-path is the path of output audit log.

      You can use the following optional arguments to control the log rotation:

      • audit-log-maxsize is the maximum size in MB before log rotation.
      • audit-log-maxbackup is the maximum number of rotated logs retained.
      • audit-log-maxage is the maximum days to retain old rotated files.
      The following graphic shows an example:
      example of the command line arguments
    2. After you complete the systemd unit configuration, reload and restart the service using the following commands:
      systemctl daemon-reload &&
systemctl restart k3s
      The output audit events are stored in JSONline files in the location specified in service startup argument. The following example shows output after pairing a new Edge Gateway with the audit policy that is described in this procedure:
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"b20ac0e0-bd02-4b93-90fd-dd24b73c9330","stage":"RequestReceived","requestURI":"/apis/apps/v1/namespaces/cfd3995c-c011-4e1f-aa05-8c6787d1861b/deployments","verb":"create","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"okhttp/3.12.12","objectRef":{"resource":"deployments","namespace":"cfd3995c-c011-4e1f-aa05-8c6787d1861b","apiGroup":"apps","apiVersion":"v1"},"requestReceivedTimestamp":"2022-11-30T07:52:23.440012Z","stageTimestamp":"2022-11-30T07:52:23.440012Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"b20ac0e0-bd02-4b93-90fd-dd24b73c9330","stage":"ResponseComplete","requestURI":"/apis/apps/v1/namespaces/cfd3995c-c011-4e1f-aa05-8c6787d1861b/deployments","verb":"create","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"okhttp/3.12.12","objectRef":{"resource":"deployments","namespace":"cfd3995c-c011-4e1f-aa05-8c6787d1861b","name":"deployment-synchronizer","apiGroup":"apps","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2022-11-30T07:52:23.440012Z","stageTimestamp":"2022-11-30T07:52:23.446914Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"c10353df-095a-4454-96c5-ca028988911a","stage":"RequestReceived","requestURI":"/apis/apps/v1/namespaces/cfd3995c-c011-4e1f-aa05-8c6787d1861b/deployments","verb":"create","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"okhttp/3.12.12","objectRef":{"resource":"deployments","namespace":"cfd3995c-c011-4e1f-aa05-8c6787d1861b","apiGroup":"apps","apiVersion":"v1"},"requestReceivedTimestamp":"2022-11-30T07:52:23.521285Z","stageTimestamp":"2022-11-30T07:52:23.521285Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"c10353df-095a-4454-96c5-ca028988911a","stage":"ResponseComplete","requestURI":"/apis/apps/v1/namespaces/cfd3995c-c011-4e1f-aa05-8c6787d1861b/deployments","verb":"create","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"okhttp/3.12.12","objectRef":{"resource":"deployments","namespace":"cfd3995c-c011-4e1f-aa05-8c6787d1861b","name":"deployment-operator","apiGroup":"apps","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2022-11-30T07:52:23.521285Z","stageTimestamp":"2022-11-30T07:52:23.531923Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}

      For more information, see the Kubernetes documentation.

    If you are using the remote webhook log backend, you must create the webhook configuration, add the related configurations into K3s systemd starting arguments, reload the configuration, and restart K3s service, as follows:
    1. Create a webhook configuration file. This file contains the address and the credentials for HTTP connections, and is similar to the kubectl configuration in $HOME/.kube/config. The following example defines a loopback backend with basic authentication:
      cat >/var/lib/rancher/k3s/server/manifests/webhook.yaml <<EOF
apiVersion: v1
kind: Config
preferences: {}

clusters:
# address and port of remote server
- name: example-audit-backend
  cluster:
    server: http://localhost:8888

users:
# client credential
- name: example-user
  user:
    username: example-username
    password: example-password

contexts:
# link the remote server address and credential
- name: example-context
  context:
    cluster: example-audit-backend
    user: example-user
current-context: example-context
EOF

      For more information, see the Kubernetes documentation.

    2. In k3s.service configuration file, under Service section, append options in the ExecStart:
      vi /etc/systemd/system/k3s.service

      The following two options are required and both must be passed by --kube-apiserver-arg=option-name=option-value:

      • audit-policy-file is the path of audit policy, as described in the first step.

      • audit-webhook-config-file is the path of the webhook configuration, as described in the previous step.
        exampe of the command line options described in the text
    3. After editing the systemd unit configuration, reload and restart the service:
      systemctl daemon-reload &&
systemctl restart k3s

      The output audit events will be sent in batch as lists of JSON in HTTP POST body. The following is an example of output after pairing a new Edge Gateway with the audit policy described in this procedure and the webhook request body:

      {
  "kind":"EventList",
  "apiVersion":"audit.k8s.io/v1",
  "metadata":{},
  "items":
  [
    {"level":"Metadata","auditID":"86fcb3ac-9ae2-4f8f-a102-9efa0747f513","stage":"RequestReceived","requestURI":"/apis/apps/v1/namespaces/cfd3995c-c011-4e1f-aa05-8c6787d1861b/deployments","verb":"create","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"okhttp/3.12.12","objectRef":{"resource":"deployments","namespace":"cfd3995c-c011-4e1f-aa05-8c6787d1861b","apiGroup":"apps","apiVersion":"v1"},"requestReceivedTimestamp":"2022-12-02T09:02:10.502140Z","stageTimestamp":"2022-12-02T09:02:10.502140Z"},
    {"level":"Metadata","auditID":"86fcb3ac-9ae2-4f8f-a102-9efa0747f513","stage":"ResponseComplete","requestURI":"/apis/apps/v1/namespaces/cfd3995c-c011-4e1f-aa05-8c6787d1861b/deployments","verb":"create","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"okhttp/3.12.12","objectRef":{"resource":"deployments","namespace":"cfd3995c-c011-4e1f-aa05-8c6787d1861b","name":"deployment-synchronizer","apiGroup":"apps","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2022-12-02T09:02:10.502140Z","stageTimestamp":"2022-12-02T09:02:10.511688Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}},
    {"level":"Metadata","auditID":"4366e4f2-f437-44f7-bc8d-97dec21b54ef","stage":"RequestReceived","requestURI":"/apis/apps/v1/namespaces/cfd3995c-c011-4e1f-aa05-8c6787d1861b/deployments","verb":"create","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"okhttp/3.12.12","objectRef":{"resource":"deployments","namespace":"cfd3995c-c011-4e1f-aa05-8c6787d1861b","apiGroup":"apps","apiVersion":"v1"},"requestReceivedTimestamp":"2022-12-02T09:02:10.583000Z","stageTimestamp":"2022-12-02T09:02:10.583000Z"},
    {"level":"Metadata","auditID":"4366e4f2-f437-44f7-bc8d-97dec21b54ef","stage":"ResponseComplete","requestURI":"/apis/apps/v1/namespaces/cfd3995c-c011-4e1f-aa05-8c6787d1861b/deployments","verb":"create","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"okhttp/3.12.12","objectRef":{"resource":"deployments","namespace":"cfd3995c-c011-4e1f-aa05-8c6787d1861b","name":"deployment-operator","apiGroup":"apps","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2022-12-02T09:02:10.583000Z","stageTimestamp":"2022-12-02T09:02:10.612983Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
  ]
}

      For more information, see the Kubernetes documentation.