Integrating your cluster with Splunk

If you use Splunk Enterprise, you can integrate your product audit logs with Splunk.

Configuring your product cluster (client)

Add the Splunk CA certificate in the audit-certs secret for your product

1. Convert Splunk CA certificate into base64 wrap 0 format by using the following command:

    cat splunkCA.pem | base64 -w 0
   

2. From the navigation menu, click Configuration > Secrets.

3. Select audit-certs. Click Edit.
4. The audit-certs secret is in JSON format. Add "splunkCA.pem" as a key and the base64 encoded version of the Splunk CA certificate as the value.
5. Click Submit.

Update audit-logging-fluentd-ds-config and audit-logging-fluentd-ds-splunk-hec-config ConfigMap files for your product

The audit-logging-fluentd-ds-splunk-hec-config ConfigMap file contains an output plugin that is used to forward audit logs to Splunk. The output plugin is included in the main ConfigMap file, audit-logging-fluentd-ds-config.

Update the audit-logging-fluentd-ds-splunk-hec-config ConfigMap file

1. From the navigation menu, click Configuration > ConfigMap.
2. Search for the audit-logging-fluentd-ds-splunk-hec-config file.
3. Click Edit.
4. Add the Splunk server hostname, port number, and SPLUNK_HEC_TOKEN.
5. Click Submit.

The following example is a default audit-logging-fluentd-ds-splunk-hec-config ConfigMap file.

{
  "apiVersion": "v1",
  "kind": "ConfigMap",
  "metadata": {
    "name": "audit-logging-fluentd-ds-splunk-hec-config",
    "namespace": "kube-system",
    "resourceVersion": "11134128",
    "labels": {
      "app": "audit-logging-fluentd",
      "chart": "audit-logging-3.2.0",
      "component": "fluentd",
      "heritage": "Tiller",
      "release": "audit-logging"
    }
  },
  "data": {
    "splunkHEC.conf": "<match icp-audit kube-audit>\n  @type splunk_hec\n  hec_host SPLUNK_SERVER_HOSTNAME\n  hec_port SPLUNK_PORT\n  hec_token SPLUNK_HEC_TOKEN\n  ca_file /fluentd/etc/tls/splunkCA.pem\n\n  source ${tag}\n</match>"
  }
}

Note: The value of the splunkHEC.conf key is in stringify yaml format. \n represents a line change. Spaces between \n and the next word should be retained as is. Changes in spacing can result in errors.

For example, <match icp-audit kube-audit>\n @type splunk_hec\n hec_host SPLUNK_SERVER_HOSTNAME\n. The number of spaces between first \n and word @type is the same as the second \n and the hec_host. Be sure to maintain the same number of spaces if you add a new field.

Update the audit-logging-fluentd-ds-config ConfigMap file

1. From the navigation menu, click Configuration > ConfigMap.
2. Search for the audit-logging-fluentd-ds-config file.
3. Click Edit.
4. Set the ENABLE_AUDIT_LOGGING_FORWARDING key to true to enable forwarding.
5. Uncomment line, @include /fluentd/etc/splunkHEC.conf. Comment other output plugin lines.
6. Click Submit.

The following example is the default audit-logging-fluentd-ds-config ConfigMap file.

{
  "apiVersion": "v1",
  "kind": "ConfigMap",
  "metadata": {
    "name": "audit-logging-fluentd-ds-config",
    "namespace": "kube-system",
    "resourceVersion": "11920745",
    "labels": {
      "app": "audit-logging-fluentd",
      "chart": "audit-logging-3.2.0",
      "component": "fluentd",
      "heritage": "Tiller",
      "release": "audit-logging"
    }
  },
  "data": {
    "ENABLE_AUDIT_LOGGING_FORWARDING": "false",
    "fluent.conf": "# Input plugins\n@include /fluentd/etc/source.conf\n\n# Output plugins\n# Only use one output plugin conf file at a time. Comment or remove other files \n\n# To forward audit logs to QRadar, uncommnet following line, add QRadar server information in the 'audit-logging-fluentd-ds-remote-syslog-config' ConfigMap and restart the 'audit-logging-fluentd-ds-*' pods\n#@include /fluentd/etc/remoteSyslog.conf\n\n#To forward audit logs to Splunk over HTTPS, uncomment following line, add Splunk server information in the 'audit-logging-fluentd-ds-splunk-hec-config' ConfigMap and restart the 'audit-logging-fluentd-ds-*' pods\n#@include /fluentd/etc/splunkHEC.conf"
  }
}

Note: The value of the fluent.conf key is in stringify yaml format. Changes in spacing can result in errors. To comment any configuration file, add # in front of @include.

Remove fluentd audit-logging-fluentd-ds* pods

Use one of the following methods to remove all pods in the audit-logging-fluentd-ds daemonset. The pods will be re-created with appropriate audit log routing.

• Use the console to remove all pods from the daemonset:

  1. Log in to the console.
  2. Navigate to Workload > DaemonSets
  3. Locate and click audit-logging-fluentd-ds daemonset.
  4. Delete all the pods.

• Use the Kubernetes CLI to remove all pods from the daemonset:

  1. Install kubectl. For more information, see Installing the Kubernetes CLI (kubectl).
  2. Run the following command:

     kubectl get pod -n kube-system -o wide | grep audit-logging-fluentd-ds- | awk '{print $1}' | xargs kubectl delete pod -n kube-system
     

Add hostAliases to the audit-logging-fluentd-ds daemonset deployment spec

1. Use the following command to edit the daemonset deployment.

   kubectl -n kube-system edit daemonset audit-logging-fluentd-ds
   

2. Add the Splunk server hostname and IP address mapping to the spec on the pod level. You can add multiple hostnames and IP address mappings.

   hostAliases:
    - hostnames:
      - <Splunk-Server-Host-Name>
      ip: <Splunk-Server-IP-Address>
    - hostnames:
      - <QRadar-Server-Host-Name>
      ip: <QRadar-Server-IP-Address>
   

3. Save the edited file.

Fluentd configuration

Using the fluent-plugin-splunk-hec output plugin

    <match icp-audit kube-audit>
      @type splunk_hec
      hec_host SPLUNK_SERVER_HOSTNAME
      hec_port SPLUNK_PORT
      hec_token SPLUNK_HEC_TOKEN
      ca_file /fluentd/etc/tls/splunkCA.pem

      # Following parameters overwrite HEC default parameters. Optional  parameters are:
      index awesome
      source ${tag}
      sourcetype _json
    </match>

For more information, see the Splunk documentation.

Splunk

You must configure Splunk to get from fluentd over the HTTP event collector.

HTTP Event Collector (HEC)

1. Enable HEC.
2. Create an HEC token and customize it.
3. Enable the token.

For more information, see the Splunk documentation.

Splunk configurations and certificates

The following are default directories for configurations and certificates.

• Default configuration directory - /opt/splunk/etc/system/local
• Default certificates directory - /opt/splunk/etc/auth

/opt/splunk/etc/system/local includes input.conf and server.conf files that you must modify.

Add following http configuration in the input.conf file

# HEC plugin configuration
[http]
port = 8088
disabled = 0
enableSSL = 1
dedicatedIoThreads = 4
maxSockets = 50
maxThreads = 20
serverCert = <server certificates path > # For example - /opt/splunk/etc/auth/myNewServerCertificate.pem
sslPassword = <certificates password>

Add the CA certificate path to [sslConfig] in the server.conf file

[sslConfig]
sslRootCAPath = <ca certificate path> # For example - /opt/splunk/etc/auth/myCACertificate.pem

For custom configuration information, see the Splunk documentation.

For information about certificates, see the Splunk documentation.

Restart the splunkd service

After modifying the configuration, restart the service by using the following command:

$SPLUNK_HOME/bin/splunk restart splunkd