Architecture for Audit logging version 3.7.0

Following are the key components of the audit logging architecture:

• Rsyslog Sidecar for forwarding audit records over TLS syslog connections
  • Steps for adoption
• CommonAudit Custom Resource (CR)
  • Configuring CommonAudit
    • CommonAudit settings
    • Sample CommonAudit CR
    • Sample AuditLogging CR (deprecated)
• Fluentd
• Integrating your cluster with IBM QRadar
• Integrating your cluster with Splunk Enterprise
• Audit log events
  
• Audit logging format
  
• Authentication and authorization audit logs
  
• Audit logging in your cluster
  
• Audit logging data statistics
  

Rsyslog Sidecar for forwarding audit records over TLS syslog connections

Audit logging version 3.7.0 introduces support for sending audit logs over Rsyslog. Rsyslog is a dynamic log processing tool that is built on top of the standard Linux® syslogd. Similar to the deprecated audit logging sidecar, a Rsyslog sidecar offers easy adoption of audit logging and shifts the burden of transmitting the messages to the sidecar. The audit logging Rsyslog sidecar can be adopted by IBM Cloud Pak® foundational services and IBM Cloud Pak® applications to forward generated audit record JSON files to the audit logging service over a TLS syslog connection.

The Rsyslog sidecar can be deployed in parallel with a service container on the same pod. The service application container and the Rsyslog sidecar container are configured to share the emptyDir volume /var/log/audit.

Every service that generates audit logs writes the logs to the /var/log/audit/<service_name>-audit.log file. The audit sidecar monitors the file for audit records to forward onto the audit logging service over a TLS syslog connection.

The Rsyslog sidecar uses the Logrotate tool to monitor the logs in the /var/log/audit directory. It monitors size, rotate period, and other parameters, and also recycles the audit logs as specified in the configuration.

The audit container sidecar is used by audit logging adopters in their service offerings. The audit container sidecar is not included in the audit logging service.

Note: Adopters need to indicate to their customers that data in the sidecar container might include Sensitive Private Information (SPI) in the audit logs.

Steps for adoption

1. Add the audit logging Rsyslog sidecar container at the first position under container section in the deployment file. As an example, image, quay.io/opencloudio/audit-sidecar-syslog:1.0.0 includes the image for the 3.7.0 release.

2. Audit logging Rsyslog sidecar containers use a UID. Both the Rsyslog sidecar and the service container must run as the same user. Only the following UIDs are supported:

   1000
   21000
   55555
   65534
   65535
   

3. Mount the /var/log/audit shared volume on all containers in your pod.

4. Mount the audit-server-certs secret to the sidecar container in your pod with the mountPath etc/audit-tls. Set this volume to optional: true. If your service runs outside of the namespace where audit logging fluentd is running, you must copy the secret to your namespace. For more information about how to request that your secret is copied to your namespace, see Bindings for the ibm-auditlogging-operator.
5. Add the environment variable AUDIT_URL by using the value from the key AuditLoggingSyslogIngestURL in the configmap, audit-logging-fluentd-ds-http-ingesturl to the sidecar container in your pod. Link OperandBindInfo for audit logging.

6. Configure Logrotate. The following example resembles the default configuration.

   /var/log/audit/*.log {
         copytruncate
         rotate 24
         hourly
         missingok
         notifempty
       }
   

   If you prefer to use a custom Logrotate configuration, you can use a ConfigMap and mount it to the Rsyslog sidecar. For example,

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: logrotate
     labels:
       app: "logrotate"
   data:
     LOGROTATE: |
       /var/log/audit/*.log {
         copytruncate
         rotate 5
         weekly
         missingok
         notifempty
       }
     LOGROTATE_CONF: |-
       include /etc/logrotate.d
   

   volumeMounts:
           - mountPath: /var/log/audit
             name: shared
           - mountPath: /etc/audit-ingest/
             name: audit-ingest
           - mountPath: /etc/audit-tls
             name: audit-server-certs
           - mountPath: /etc/logrotate.d/audit
             name: logrotate
             subPath: audit
           - mountPath: /etc/logrotate.conf
             name: logrotate-conf
             subPath: logrotate.conf
   volumes:
           - name: audit-ingest
             configMap:
               name: audit-logging-fluentd-ds-http-ingesturl
               optional: true
               items:
               - key: AuditLoggingSyslogIngestURL
                 path: auditurl
           - name: audit-server-certs
             secret:
               secretName: audit-server-certs
               optional: true
           - configMap:
               defaultMode: 420
               items:
               - key: LOGROTATE
                 path: audit
               name: logrotate
             name: logrotate
           - configMap:
               defaultMode: 420
               items:
               - key: LOGROTATE_CONF
                 path: logrotate.conf
               name: logrotate
             name: logrotate-conf
   

7. You can enable or disable audit logs for a service. Add the AUDIT_ENABLED flag to the ConfigMap of one of your services. You can add the flag as an environment variable so that the application code has access to enable or disable generation of audit records. For more information about the AUDIT_ENABLED flag, see Audit logging policy controller.

   Note: You can enable the audit logs for the Identity and Access management (IAM) service. For more information, see Auditing IAM service.

1. Your final deployment resembles the following example.

   Note: Your service and the rsyslog sidecar must run in the same namespace as your fluentd instance. Complete sample deployment follows:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       run: audit-log-test
     name: audit-log-test
     namespace: common-test
   spec:
     replicas: 1
     selector:
       matchLabels:
         run: audit-log-test
     strategy:
       rollingUpdate:
         maxSurge: 25%
         maxUnavailable: 25%
       type: RollingUpdate
     template:
       metadata:
         labels:
           run: audit-log-test
       spec:
         containers:
         - name: audit-service-syslog
           env:
           - name: DEBUG_SYSLOG
             value: "true"
           - name: POD_NAMESPACE
             valueFrom:
               fieldRef:
                 fieldPath: metadata.namespace
           - name: POD_NAME
             valueFrom:
               fieldRef:
                 fieldPath: metadata.name
           - name: SERVICE_NAME
             value: audit-log-test-service
           image: quay.io/opencloudio/audit-syslog-service:1.0.3
           imagePullPolicy: Always
           securityContext:
             runAsUser: 1000
           name: audit-service-syslog
           resources:
             limits:
               cpu: 100m
               memory: 128Mi
             requests:
               cpu: 50m
               memory: 128Mi
           terminationMessagePath: /dev/termination-log
           terminationMessagePolicy: File
           volumeMounts:
           - mountPath: /var/log/audit
             name: shared
           - mountPath: /etc/audit-ingest/
             name: audit-ingest
           - mountPath: /etc/audit-tls
             name: audit-server-certs
           - mountPath: /etc/logrotate.d/audit
             name: logrotate
             subPath: audit
           - mountPath: /etc/logrotate.conf
             name: logrotate-conf
             subPath: logrotate.conf
         - name: audit-log-test
           image: myrepository/audit-log-test:syslog
           imagePullPolicy: Always
           securityContext:
             runAsUser: 1000
           volumeMounts:
           - mountPath: /etc/audit-ingest/
             name: audit-ingest
           - mountPath: /var/log/audit
             name: shared
           env:
           - name: SERVICE_NAME
             value: audit-log-test
           - name: EVENT_COUNT
             value: "10"
           - name: CONTAINER_ID
             value: audit-log-test
           - name: SLEEP_TIME
             value: "900"
           - name: POD_NAME
             valueFrom:
               fieldRef:
                 fieldPath: metadata.name
           - name: POD_NAMESPACE
             valueFrom:
               fieldRef:
                 fieldPath: metadata.namespace
         volumes:
         - name: audit-ingest
           configMap:
             name: audit-logging-fluentd-ds-http-ingesturl
             optional: true
             items:
             - key: AuditLoggingSyslogIngestURL
               path: auditurl
         - name: audit-server-certs
           secret:
             secretName: audit-server-certs
             optional: true
         - name: shared
           emptyDir: {}
         - configMap:
             defaultMode: 420
             items:
             - key: LOGROTATE
               path: audit
             name: logrotate
           name: logrotate
         - configMap:
             defaultMode: 420
             items:
             - key: LOGROTATE_CONF
               path: logrotate.conf
             name: logrotate
           name: logrotate-conf
         serviceAccount: test-sa
         serviceAccountName: test-sa
   

In a service container, the application that is running creates a <service_name>-audit.log file in the directory that is mounted on /var/log/audit.

Write or append all audit logs that are generated by the application or service to the <service_name>-audit.log file. The Rsyslog sidecar container forwards the logs to fluentd over a TLS syslog connection.

The audit logs must be in JSON format to be correctly parsed by Fluentd and a SIEM tool.

CommonAudit Custom Resource (CR)

Audit logging service version 3.7.0 introduces the CommonAudit CR. The CommonAudit CR configures a namespace scoped fluentd. In a multi-tenant configuration, CommonAudit configures the audit logging service and runs independently from audit logging services in other namespaces. Audit records are collected and forwarded only to SIEM tools that are configured within a namespace where a CommonAudit CR is deployed.

AA = Audit Agent. For more information about audit agents, see Rsyslog Sidecar for forwarding audit records over TLS syslog connections.

In the case where a cluster scoped instance of fluentd is needed, you can deploy a single CommonAudit CR to the ibm-common-services namespace. You can then request an OperandBindInfo for services across namespaces to forward logs to the audit logging service.

Note: A CommonAudit CR and AuditLogging CR (deprecated) cannot run together in the same namespace. By default, the Operand Deployment Lifecycle Manager deploys an AuditLogging CR to the ibm-common-services namespace. To deploy a CommonAudit CR in the ibm-common-services namespace, you must first remove the AuditLogging CR.

The CommonAudit fluentd deployment supports multiple replicas. By default, the fluentd deployment is configured as a single replica with minimal memory and CPU load. However, fluentd can be scaled up and out as needed. For more information, see Hardware requirements and recommendations for IBM Cloud Pak foundational services.

The audit logging service supports HTTPS ingest endpoints and Rsyslog forwarding collection and ingestion mechanisms. IBM Cloud Pak foundational services and IBM Cloud Pak® can route selected service audit records to the audit logging services.

Configuring CommonAudit

1. Log in to your cluster.
2. From the Red Hat® OpenShift® Container Platform console, click Operators > Installed Operators.
3. From the Project drop-down list, select the ibm-common-services namespace.
4. Select IBM Audit Logging Operator.
5. Select the CommonAudit tab.
6. Create or edit a CommonAudit instance. Note: CommonAudit instances that are deployed outside of the ibm-common-services namespace do not appear under the CommonAudit tab. You can use the oc get commonaudit --all-namespaces command to view them from your infrastructure node.

CommonAudit settings

Sample CommonAudit CR

apiVersion: operator.ibm.com/v1
kind: CommonAudit
metadata:
  name: example-commonaudit
  labels:
    app.kubernetes.io/instance: common-audit-logging
    app.kubernetes.io/managed-by: ibm-auditlogging-operator
    app.kubernetes.io/name: common-audit-logging
spec:
  enabled: true
  issuer: cs-ca-issuer
  replicas: 1
  fluentd:
    imageRegistry: quay.io/opencloudio/
    pullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 100m
        memory: 200Mi
      limits:
        cpu: 200m
        memory: 300Mi
  outputs:
    splunk:
      enableSIEM: true
      host: splunk.fyre.ibm.com
      port: 8088
      token: abc-123
      enableTLS: true
    syslog:
      enableSIEM: false
      host: qradar.fyre.ibm.com
      port: 514
      hostname: qradar_source_id
      enableTLS: true
    hostAliases:
      - ip: 9.12.34.56
        hostnames:
          - qradar.fyre.ibm.com
      - ip: 9.65.43.21
        hostnames:
          - splunk.fyre.ibm.com

Sample AuditLogging CR (deprecated)

apiVersion: operator.ibm.com/v1alpha1
kind: AuditLogging
metadata:
  creationTimestamp: "2021-09-30T04:11:13Z"
  generation: 2
  labels:
    app.kubernetes.io/instance: common-audit-logging
    app.kubernetes.io/managed-by: ibm-auditlogging-operator
    app.kubernetes.io/name: common-audit-logging
    operator.ibm.com/opreq-control: "true"
  name: example-auditlogging
  resourceVersion: "18197844"
  uid: 39c37997-f3d8-4042-ba07-074718922527
spec:
  fluentd:
    enabled: true
    imageRegistry: quay.io/opencloudio/
    journalPath: /run/log/journal
    pullPolicy: IfNotPresent
    resources:
      limits:
        cpu: 50m
        memory: 150Mi
      requests:
        cpu: 20m
        memory: 20Mi
  policyController:
    enabled: "true"
status:
  nodes:
  - audit-logging-fluentd-ds-8k72m
  - audit-logging-fluentd-ds-qcdjm
  - audit-logging-fluentd-ds-rxprv
  - audit-policy-controller-678b44d788-4kgtl
  versions:
    reconciled: 3.20.0

Fluentd

Fluentd is a log collector that uses input and output plug-ins to collect data from multiple sources and to distribute or send data to multiple destinations.

With Audit logging, fluentd collects audit logs by using the following sources:

• Journald by using the fluent-plugin-systemd input plug-in (deprecated).
• HTTP requests by using the in_http plug-in.
• Syslog protocol with TLS by using the in_syslog input plug-in.

Fluentd containers mount a host file system where the journal log data is stored. The default location is /run/log/journal.

Fluentd can be configured to send logs to an enterprise SIEM tool such as IBM QRadar.

Integrating your cluster with IBM QRadar

• IBM QRadar log source extension to parse your cluster audit logs
  
• Configuring IBM QRadar to receive your cluster audit logs over TLS
  
• Configuring CommonAudit to send audit logs over TLS to IBM QRadar
  
• Configuring IBM QRadar rules
  

Integrating your cluster with Splunk Enterprise

• Configuring Splunk to receive your audit logs
  
• Configuring CommonAudit to send audit logs to Splunk