Exporting IBM Software Hub audit records to Apache Kafka

You can export your IBM Software Hub audit records to Apache Kafka.

Who needs to complete this task?

To complete this task on the primary IBM Software Hub cluster, you must have one of the following roles:

  • Cluster administrator
  • Instance administrator
To complete this task on a remote physical location, you must have one of the following roles:
  • Cluster administrator
  • An administrator of the management namespace on the remote physical location.
When do you need to complete this task?
Complete this task if you want to export IBM Software Hub audit records to Apache Kafka.
  • If you have multiple instances of IBM Software Hub, you must complete this task for each instance of IBM Software Hub from which you want to export your audit records.
  • If you have multiple remote physical locations, you must complete this task for each remote physical location from which you want to export audit records.
    Best practice: Use the same configuration for exporting audit records from the primary instance of IBM Software Hub and from any remote physical locations associated with the primary instance of IBM Software Hub.

Before you begin

Best practice: You can run many of the commands in this task exactly as written if you set up environment variables for your installation. For instructions, see Setting up installation environment variables.

Ensure that you source the environment variables before you run the commands in this task.

About this task

IBM Software Hub uses the Fluentd to Kafka plug-in to integrate with Apache Kafka.

Update the zen-audit-secret secret

Update the zen-audit-secret secret to connect to your Kafka environment.

On the cluster from which you want to export audit records:

  1. If you want to use TLS:
    1. Set the CA_CERT environment variable to the CA certificate for the Kafka server:
      export CA_CERT=<CA-certificate>
    2. Create a file called kafka-ca-cert.pem:
      cat << EOF > ./kafka-ca-cert.pem
-----BEGIN CERTIFICATE-----
${CA_CERT}
-----END CERTIFICATE-----
EOF
    3. Set the CLIENT_CERT environment variable to the client certificate:
      export CLIENT_CERT=<client-certificate>
    4. Create a file called kafka-client-cert.pem:
      cat << EOF > ./kafka-client-cert.pem
-----BEGIN CERTIFICATE-----
${CLIENT_CERT}
-----END CERTIFICATE-----
EOF
    5. Set the PRIVATE_KEY environment variable to the key associated with the client certificate:
      export PRIVATE_KEY=<client-key>
    6. Create a file called kafka-client-key.pem:
      cat << EOF > ./kafka-client-key.pem
-----BEGIN CERTIFICATE-----
${PRIVATE_KEY}
-----END CERTIFICATE-----
EOF
  2. Set the following environment variables:
    1. Set the KAFKA_HOST environment variable to the address of the Kafka host:
      export KAFKA_HOST=<hostname>
    2. Set the KAFKA_PORT environment variable to the port on the Kafka server. The default port is 9092.
      export KAFKA_PORT=<port-number>
    3. Set the KAFKA_TOPIC environment variable to the topic where you want to store the audit logs:
      export KAFKA_TOPIC=<topic-name>
  3. Create a file called kafka-fluent.conf:
    Configuration for connecting to Kafka using TLS 
    cat << EOF > ./kafka-fluent.conf
<match export export.** records records.** syslog syslog.**> 
    @type copy 
    <match **>
      @type kafka2
      brokers ${KAFKA_HOST}:${KAFKA_POST} 
      use_event_time true
      default_topic ${KAFKA_TOPIC}
      <format>
        @type "json"
      </format>
      ssl_ca_cert /fluentd/config/ca_cert.pem
      ssl_client_cert /fluentd/config/client_cert.pem
      ssl_client_cert_key /fluentd/config/client_key.pem
    </match>
</match>
EOF
    Tip: For additional SSL parameters, see SSL authentication in the fluent-plugin-kafka Readme on GitHub.
    Configuration for connecting to Kafka without TLS 
    cat << EOF > ./kafka-fluent.conf
<match export export.** records records.** syslog syslog.**> 
    @type copy 
    <match **>
      @type kafka2
      brokers ${KAFKA_HOST}:${KAFKA_POST} 
      use_event_time true
      default_topic ${KAFKA_TOPIC}
      <format>
        @type "json"
      </format>
    </match>
</match>
EOF
  4. Update the zen-audit-secret secret:
    Primary IBM Software Hub cluster
    Connect to Kafka using TLS 
    oc patch secret zen-audit-secret \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge \
--patch='{"data":{"CA_cert.pem":"'$(cat kafka-ca-cert.pem | base64 -w0)'", "client_cert.pem":"'$(cat kafka-client-cert.pem | base64 -w0)'", "client_key.pem":"'$(cat kafka-client-key.pem | base64 -w0)'", "fluent.conf":"'$(cat kafka-fluent.conf | base64 -w0)'"}}'
    Connect to Kafka without TLS 
    oc patch secret zen-audit-secret \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge \
--patch='{"data": {"fluent.conf": "'$(cat kafka-fluent.conf | base64 -w0)'"}}'
    Remote physical location
    Connect to Kafka using TLS 
    oc patch secret zen-audit-secret \
--namespace=${REMOTE_PROJECT_MANAGEMENT} \
--type=merge \
--patch='{"data":{"CA_cert.pem":"'$(cat kafka-ca-cert.pem | base64 -w0)'", "client_cert.pem":"'$(cat kafka-client-cert.pem | base64 -w0)'", "client_key.pem":"'$(cat kafka-client-key.pem | base64 -w0)'", "fluent.conf":"'$(cat kafka-fluent.conf | base64 -w0)'"}}'
    Connect to Kafka without TLS 
    oc patch secret zen-audit-secret \
--namespace=${REMOTE_PROJECT_MANAGEMENT} \
--type=merge \
--patch='{"data": {"fluent.conf": "'$(cat kafka-fluent.conf | base64 -w0)'"}}'
  5. Delete all zen-audit pods to force a restart to pick up changes:
    1. Get the list of zen-audit pods:
      oc get pods | grep zen-audit
    2. Delete the zen-audit pods.
      Tip: If there are multiple zen-audit pods, delete each pod individually to avoid a forced restart.
      oc delete pods zen-audit-xxxxxx-xxx

      Replace zen-audit-xxxxxx-xxx with the ID returned by the oc get pods command.

Results

After the changes are applied, new audit events from IBM Software Hub are sent to Kafka.

For more information about the audit events, see:

What to do next

If you use the Identity Management Service, complete Enabling auditing for the Identity Management Service.