IBM Cloud Pak foundational services Audit logging

Configure your product cluster to generate audit logs and route the logs to your Security information and event management (SIEM).

Your product audit logging architecture

Audit logging components

AA - Audit Agent

Audit container

Following are the key components of the audit logging architecture:

Audit container

Audit container is a sidecar container. It tails the audit.log file and pipes it by using the system-cat command. It sends the audit logs to the systemd journal.

Every service that generates audit logs writes the logs to /var/log/audit/<service_name>-audit.log file. A service container must share /var/log/audit with the audit container.

An emptyDir volume is used for sharing the directory /var/log/audit among the service containers to audit the sidecar container in a pod.

As the audit container must write to the systemd journal, it also needs to mount the host file system where the system journal exists.

The logrotate tool is used to monitor the logs in the /var/log/audit directory for size, rotate period, and other parameters, and to recycle the audit logs as specified in the configuration.

Journald (systemd journal)

systemd is a service that runs on a node. Audit logs that are generated by your product services that run in pods that have audit sidecar container, are sent to systemd journal. The systemd journal stores the data in binary format. Data can be only appended. After systemd journal receives the audit data, it is picked up by fluentd and then sent to Elasticsearch or SIEM.

Fluentd

Fluentd is a log collector that uses input and output plug-ins to collect data from multiple sources and to distribute or send data to multiple destinations.

Fluentd collects audit logs from systemd journal by using the fluent-plugin-systemd input plug-in. This plug-in has data-filtering ability, and it collects only audit logs from the systemd journal.

Fluentd containers mount a host file system where the journal log data is stored. The default location is /run/log/journal.

Fluentd can be configured to send logs to an enterprise SIEM tool such as IBM QRadar.

Audit log events

Following events can be logged as audit events:

Authentication events (successful or failed login attempt)
Authorization events (successful or failed attempt to access resources or data)
System configuration modification
Create, read, delete, or update resources or data
Failure of session management
Changes to user privileges
Database access (success or failure)
Network or application firewall, Intrusion Detection System (IDS), or Intrusion Prevention System (IPS) events
Amount of system use
System start, shutdown, or reboot events
Application or system failure events

Following data must not be included in the audit logs:

Sensitive information
User credentials
Passwords
Bank account details
Access token
Authentication token
File system path or information
Database query or string
Encryption or decryption keys

Audit logging format

Your product follows the Cloud Auditing Data Federation (CADF) standards. The standard defines an event model to collect the required data for auditing. Your product adds some custom fields to generate comprehensive logs.

For more information about CADF, see Cloud Auditing Data Federation Opens in a new tab .

The following fields are important:

{
    "typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event",
    "eventType": "activity",
    "id": "e97c7b00-e215-11e8-abf8-79cb75b57820",
    "action": "create",
    "requestPath": "/identity/api/v1/teams",
    "initiator": {
        "typeURI": "service/security/account/user",
        "name": "admin",
        "credential": {
            "type": "token"
        },
        "host": {
            "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15",
            "address": "icp-management-ingress:8443"
        }
    },
    "target": {
        "id": "c4e8170e90a7c01a228fbef74c22245d2665cefffac1662907a6a75e82319a74",
        "name": "icp-testing-audit-logs",
        "actions": {
            "name": "icp-testing-audit-logs",
            "teamId": "icp-testing-audit-logs",
            "users": [],
            "usergroups": [],
            "directoryList": []
        },
        "typeURI": "service/security/group"
    },
    "observer": {
        "id": "target"
    },
    "severity": "normal",
    "outcome": "success",
    "reason": {
        "reasonType": "HTTP",
        "reasonCode": 200
    },
    "eventTime": "2018-11-06T22:47:17.424Z",
    "kubernetes.container_id": "c4e8170e90a7c01a228fbef74c22245d2665cefffac1662907a6a75e82319a74",
    "kubernetes.container_name": "platform-identity-management",
    "kubernetes.pod": "auth-idp-mw2x9",
    "kubernetes.namespace": "kube-system",
    "origination": "ui",
    "version": "v1.0"
}

Go language CADF structure

type CADF struct {
    TypeURI   string `json:"typeURI"`
    Action    string `json:"action"`
    ID        string `json:"id"`
    Initiator struct {
        Name       string `json:"name"`
        TypeURI    string `json:"typeURI"`
        Credential struct {
            Type string `json:"type"`
        } `json:"credential"`
    } `json:"initiator"`
    Target struct {
        ID      string `json:"id"`
        Name    string `json:"name"`
        TypeURI string `json:"typeURI"`
    } `json:"target"`
    RequestPath             string `json:"requestPath"`
    EventType               string `json:"eventType"`
    Severity                string `json:"severity"`
    Outcome                 string `json:"outcome"`
    EventTime               string `json:"eventTime"`
    KubernetesContainerID   string `json:"kubernetes.container_id"`
    KubernetesContainerName string `json:"kubernetes.container_name"`
    KubernetesPod           string `json:"kubernetes.pod"`
    KubernetesNamespace     string `json:"kubernetes.namespace"`
    Observer                struct {
        ID string `json:"id"`
    } `json:"observer"`
    Origination string `json:"origination"`
    Version     string `json:"version"`
}

Node.js CADF structure

 let cadf = {
      "typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event",
      "eventType": "activity",
      "id": +uuid1,
      "action": action,
      "requestPath": path,
      "initiator": {
    "typeURI": (user ? "service/security/account/user": ''),
        "name": user,
    "credential": {
      "type":"token"
        },
    "host": {
          "user-agent": req.headers['user-agent'],
          "address": req.headers['host']
    }
      },
      "target": {
    "id": cont_id,
    "name": res,
    "actions": actions,
        "typeURI": (map ? map: parseUrl(path)) // pretend this app is a service
      },
      "observer": {
        "id": "target"
      },
      "severity" : severity,
      "outcome": outcome,
      "reason": {
    "reasonType":"HTTP",
        "reasonCode": status, // like 200 or 400
      },
      "eventTime": expT,
      "kubernetes.container_id": cont_id,
      "kubernetes.container_name": process.env.SERVICE_NAME,
      "kubernetes.pod": process.env.POD_NAME || process.env.HOSTNAME,
      "kubernetes.namespace": process.env.POD_NAMESPACE,
      "origination": identifyOrig(req.headers['referer'] || req.headers['user-agent']),
      "version": "v1.0"
    };

Deployment file modification

Add an audit logging sidecar container at the first position under container section in the deployment file:

      - name: icp-audit-service
        image: mycluster.icp:8500/ibmcom/icp-audit-service:3.1.1
        imagePullPolicy: IfNotPresent
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /run/systemd/journal
          name: journal
        - mountPath: /var/log/audit
          name: shared

Note: The image path depends on your product version and installation type. Following example is of an offline installation on your product:

Mount the /var/log/audit/ shared volume on all containers in a pod:

        volumeMounts:
        - mountPath: /var/log/audit
          name: shared

Some of the custom fields in the CADF format can be provided in a container by using the environment variables kubernetes.container_name,kubernetes.pod, and kubernetes.namespace.

      - env:
        - name: SERVICE_NAME
          value: key-management-lifecycle
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CONFIG_PATH
          value: /opt/keyprotect/config/
        - name: ICP_NAMESPACE
          value: kube-system
        - name: CLUSTER_NAME
          valueFrom:
            configMapKeyRef:
              key: CLUSTER_NAME
              name: platform-auth-idp

In a service container, the application that is running creates a <service_name>-audit.log file in the directory that is mounted on /var/log/audit/.
Write or append all audit logs that are generated by the application or service to the <service_name>-audit.log file. Audit sidecar container forwards the logs to journald, and journald forwards to fluentd.

The audit logs must be in JSON format to be correctly parse by Fluentd and a SIEM tool. See the following sample:

 {"typeURI":"http://schemas.dmtf.org/cloud/audit/1.0/event","eventType":"activity","id":"16d5af60-4cc2-11e9-9451-57b95a7e8968","action":"update","requestPath":"/identity/api/v1/teams/test-team/resources","initiator":{"typeURI":"service/security/account/user","name":"admin","credential":{"type":"token"},"host":{"user-agent":"curl/7.47.0","address":"<cluster-ip>:8443"}},"target":{"id":"4b1871f2f163856e3e3f56723fa16c543af3b1386588d311c2b4a07436122671","name":"test-team","actions":"crn:v1:icp:private:k8:mycluster.icp:n/default:::","typeURI":"service/security/group"},"observer":{"id":"target"},"severity":"normal","outcome":"success","reason":{"reasonType":"HTTP","reasonCode":200},"eventTime":"2019-03-22T16:46:50.198Z","kubernetes.container_id":"4b1871f2f163856e3e3f56723fa16c543af3b1386588d311c2b4a07436122671","kubernetes.container_name":"platform-identity-management","kubernetes.pod":"auth-idp-8jdlx","kubernetes.namespace":"kube-system","origination":"cli","version":"v1.0"}

If there are multiple service containers in a single pod, there is a separate audit log file for each service container in the /var/log/audit/ directory and only one file for audit sidecar container. The volume of the /var/log/audit/ directory is shared by all containers in a pod.
You can enable or disable audit logs for a service. Add the AUDIT_ENABLED flag in the ConfigMap. The flag can be added as an environment variable so that the application code has access to enable or disable generation of audit records.

Following is a deployment file example:

# Please edit the following object. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.17.0 ()
  creationTimestamp: null
  labels:
    io.kompose.service: think-blue-demo-app
  name: think-blue-demo-app
  namespace: jkstore
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: think-blue-demo-app
    spec:
      containers:
      - name: icp-audit-service
        image: mycluster.icp:8500/ibmcom/icp-audit-service:3.1.2
        imagePullPolicy: IfNotPresent
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /run/systemd/journal
          name: journal
        - mountPath: /var/log/audit
          name: shared
      - name: think-blue-demo-app
        image: mycluster.icp:8500/kube-system/think-blue-demo_app:0.1
        imagePullPolicy: IfNotPresent
        args:
        - npm
        - start
        ports:
        - containerPort: 30003
        env:
        - name: CLIENT_ID
          value: "9b570f23952a45099966ffa2cf7b6355"
        - name: CLIENT_SECRET
          value: "3rQwon8BrfpOEqzX2RttUXAt4CVkf8S2WuBQKiE5wPHKudMGX0FIlARejVf9"
        - name: AUTH_URL
          value: "https://<CLUSTER_IP>:8443/idprovider/v1/auth/authorize"
        - name: TOKEN_URL
          value: "https://<CLUSTER_IP>:8443/v1/auth/token"
        - name: ISSUER_ID
          value: "https://mycluster.icp:8443/oidc/endpoint/OP"
        - name: CALLBACK_URL
          value: "https://bobcat.rtp.raleigh.ibm.com/auth/liberty/callback"
        - name: LOGOUT_URL
          value: "https://<CLUSTER_IP>:8443/v1/auth/logout"
        - name: AUDIT_ENABLED
          value: "true"
        - name: CONTAINER_ID
          value: "think-blue-demo-app"
        - name: SERVICE_NAME
          value: "think-blue-demo-app"
        - name: AUTHZ_URL
          value: "https://<CLUSTER_IP>:8443/iam-pdp/v1/authz"
        - name: PAYMENT_URL
          value: "https://<CLUSTER_IP>:8443/payments/payment/"
        - name: AUDIT_ENABLED
          valueFrom:
            configMapKeyRef:
              name: "think-blue-demo-ConfigMap"
              key: AUDIT_ENABLED
        resources: {}
        volumeMounts:
        - mountPath: /var/log/audit
          name: shared
      restartPolicy: Always
      volumes:
      - hostPath:
         path: /run/systemd/journal
         type: ""
        name: journal
      - emptyDir: {}
        name: shared
status: {}

Note: icp-audit-service is at first position. The /var/log/audit volume is mounted and shared between application container and audit sidecar container. AUDIT_ENABLED flag is imported from the ConfigMap file.

Configuring log rotation

The default logrotate configuration resembles the following example:

/var/log/audit/*.log {
      copytruncate
      rotate 24
      hourly
      missingok
      notifempty
    }

You can also use a ConfigMap to customize your logrotate configuration. For example,:

 kind: ConfigMap
 apiVersion: v1
 metadata:
   name: logrotate
   labels:
     app: "logrotate"
 data:
   LOGROTATE: |
     /var/log/audit/*.log {
       copytruncate
       rotate 5
       weekly
       missingok
       notifempty
     }
   LOGROTATE_CONF: |-
     include /etc/logrotate.d

Note: Do not use su root root if you define your logrotate configuration in a ConfigMap.

Add the following volumes to your deployment file and the appropriate volumemounts to the audit service's pod.

 volumeMounts:
         - mountPath: /etc/logrotate.d/audit
           name: logrotate
           subPath: audit
         - mountPath: /etc/logrotate.conf
           name: logrotate-conf
           subPath: logrotate.conf
 volumes:
         - configMap:
             defaultMode: 420
             items:
             - key: LOGROTATE
               path: audit
             name: logrotate
           name: logrotate
         - configMap:
             defaultMode: 420
             items:
             - key: LOGROTATE_CONF
               path: logrotate.conf
             name: logrotate
           name: logrotate-conf