Exporting IBM Software Hub audit records to Splunk

Before you begin

About this task

IBM Software Hub uses the Splunk HTTP Event Collector Output Plugin to integrate with Splunk.

To export your audit records to Splunk, complete the following tasks:

1. Add an HTTP event collector for IBM Software Hub to Splunk.
2. (FIPS-enabled clusters only) Set up a reverse proxy server on the Splunk server
3. Update the zen-audit-secret secret

Add an HTTP event collector for IBM Software Hub to Splunk

From the Splunk dashboard:

1. Click Settings > Data inputs.
2. In the HTTP Event Collector section, click Add new.
3. Give the IBM Software Hub instance a unique name.
4. In the Source name override enter a name for a source to be assigned to events that this endpoint generates.
5. In the Description enter a description for the input.
6. If you want to enable indexer acknowledgment for this token, select Enable indexer acknowledgment.
7. Configure the source type by creating a specific IBM Software Hub source, or by using the automatic detecting option, or by selecting generic JSON source type.
8. Configure App Context and Indexes for the specific use case.
9. Click Review and then click Submit.
10. Save the Generated Token Value so that it can be used in a later step.

Set up a reverse proxy server on the Splunk server

If your Splunk server does not support the preceding TLS protocols, you must set up a reverse proxy server that supports one of these protocols on the Splunk server. For example, if your Splunk server is running on Linux®, you can install NGINX on the Splunk server and use NGINX as a reverse proxy server to forward audit logs to the Splunk server.

To use NGINX as a reverse proxy:

1. Install NGINX, if it is not already installed on the Splunk server:

   sudo yum install nginx

2. Save the self-signed SSL certificate for NGINX:

   openssl req -x509 -nodes -newkey rsa:2048 \
   -keyout /etc/nginx/ssl/server.key \
   -out /etc/nginx/ssl/server.crt

   • The self-signed certificate is saved to /etc/nginx/ssl/server.crt
   • The private key is saved to /etc/nginx/ssl/server.key
3. Open the NGINX configuration file in vi or nano
4. Append the following code to the end of the http context:

   server {
       listen 80;
       server_name <splunk_server_domain>;
   
       location / {
           proxy_pass https://<splunk_server_domain>:8088;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
       }
   }
   server {
       listen 443 ssl;
       server_name <splunk_server_domain>;
   
       ssl_certificate /etc/nginx/ssl/server.crt;
       ssl_certificate_key /etc/nginx/ssl/server.key;
   
       ssl_protocols TLSv1.2 TLSv1.3;
   
       location / {
           proxy_pass https://<splunk_server_domain>:8088;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
       }
   }

5. Replace each instance of <splunk_server_domain> with the domain name or IP address of the Splunk server.
6. Save your changes and exit the file. For example, if you are using vi, press Esc and enter :wq! to save and exit.
7. Restart NGINX to apply the changes:

   sudo systemctl restart nginx

Update the zen-audit-secret secret

Update the zen-audit-secret secret to connect to your Splunk environment.

On the cluster from which you want to export audit records:

1. Set the following environment variables:
   1. If you want to use TLS, set CA_CERT environment variable to the CA certificate:

      export CA_CERT=<certificate>

   2. Set SPLUNK_HOST environment variable to the host name or IP address of the Splunk server:

      export SPLUNK_HOST=<hostname>

      Note: If you are running IBM Software Hub on a FIPS-enabled cluster and you set up a reverse proxy server on a server other than the Splunk server, set the SPLUNK_HOST environment variable to the host name or IP address of the reverse proxy server.
   3. Set SPLUNK_PORT environment variable to the port on the Splunk server. The default port is 8088.

      export SPLUNK_PORT=<port>

      Note: If you are running IBM Software Hub on a FIPS-enabled cluster and you set up a reverse proxy server on a server other than the Splunk server, set the SPLUNK_PORT environment variable to the port number that the reverse proxy server listens on.
   4. Set SPLUNK_TOKEN environment variable to the token that you generated on the Splunk server:

      export SPLUNK_TOKEN=<token>

2. If you want to use TLS, create a file called splunk-ca.crt:

   cat << EOF > ./splunk-ca.crt
   -----BEGIN CERTIFICATE-----
   ${CA_CERT}
   -----END CERTIFICATE-----
   EOF

3. Create a file called splunk-fluent.conf:

   ---

   Configuration for connecting to Splunk using TLS

   Important: The following configuration assumes that you want to verify the CA certificate. If you don't want to verify the CA certificate, uncomment the ssl_verify entry.

   cat << EOF > ./splunk-fluent.conf
   <match records records.** syslog syslog.**> 
       @type copy 
       <store>    
           @type splunk_hec
           host ${SPLUNK_HOST}
           port ${SPLUNK_PORT}
           token ${SPLUNK_TOKEN}
           flush_interval 10s                    # Recommended value
           use_ssl true
           # ssl_verify false
           ca_file /fluentd/config/ca.pem
       </store>
   </match>
   EOF

   ---

   Configuration for connecting to Splunk without TLS

   cat << EOF > ./splunk-fluent.conf
   <match records records.** syslog syslog.**> 
       @type copy 
       <store>    
           @type splunk_hec
           host ${SPLUNK_HOST}
           port ${SPLUNK_PORT}
           token ${SPLUNK_TOKEN}
           flush_interval 10s                    # Recommended value
       </store>
   </match>
   EOF

   ---

4. Update the zen-audit-secret secret:

   Primary IBM Software Hub cluster

   ---

   Connect to Splunk using TLS

   oc patch secret zen-audit-secret \
   --namespace=${PROJECT_CPD_INST_OPERANDS} \
   --type=merge \
   --patch='{"data":{"ca.pem":"'$(cat splunk-ca.crt | base64 -w0)'", "fluent.conf":"'$(cat splunk-fluent.conf | base64 -w0)'"}}'

   ---

   Connect to Splunk without TLS

   oc patch secret zen-audit-secret \
   --namespace=${PROJECT_CPD_INST_OPERANDS} \
   --type=merge \
   --patch='{"data": {"fluent.conf": "'$(cat splunk-fluent.conf | base64 -w0)'"}}'

   ---

   Remote physical location

   ---

   Connect to Splunk using TLS

   oc patch secret zen-audit-secret \
   --namespace=${REMOTE_PROJECT_MANAGEMENT} \
   --type=merge \
   --patch='{"data":{"ca.pem":"'$(cat splunk-ca.crt | base64 -w0)'", "fluent.conf":"'$(cat splunk-fluent.conf | base64 -w0)'"}}'

   ---

   Connect to Splunk without TLS

   oc patch secret zen-audit-secret \
   --namespace=${REMOTE_PROJECT_MANAGEMENT} \
   --type=merge \
   --patch='{"data": {"fluent.conf": "'$(cat splunk-fluent.conf | base64 -w0)'"}}'

   ---

5. Delete all zen-audit pods to force a restart to pick up changes:
   1. Get the list of zen-audit pods:

      oc get pods | grep zen-audit

   2. Delete the zen-audit pods.
      Tip: If there are multiple zen-audit pods, delete each pod individually to avoid a forced restart.

      oc delete pods zen-audit-xxxxxx-xxx

      Replace zen-audit-xxxxxx-xxx with the ID returned by the oc get pods command.

Results

After the changes are applied, new audit events from IBM Software Hub are sent to Splunk.

What to do next

If you use the Identity Management Service, complete Enabling auditing for the Identity Management Service.