Installing and integrating Humio Log Management with IBM Cloud Pak® for Multicloud Management

Humio is an IBM Cloud Pak® for Multicloud Management certified container software technology partner. IBM Cloud Pak® for Multicloud Management integration with Humio's purpose-built log management system delivers an instant visibility and observability solution of all data. Humio minimizes the challenge of sifting through rapidly growing volumes of event and data logs by enabling users to log all data (both structured and unstructured) in real time and at scale. This feature is essential for modern-day applications that are running on both on premises and in hybrid cloud or multicloud environments.

With Humio's logging functions, you can ingest logs, metrics, and traces. You are not restricted by indexing. You can ask queries that you did not plan to ask initially when you started collecting logs. This feature is known as "ask anything".

The benefits of deploying Humio with IBM Cloud Pak® for Multicloud Management include:

Access to all log data enables users to instantly identify availability concerns and threats.
For activities that are being completed, you can aggregate, interact, and gain visibility by using live log data to search, create shareable dashboards, and set up alerts.
Accelerated time to resolution, with real time alerts on key services.
Can handle large multi-TBs daily data loads easily.

Important notes before you install

The following instructions are for an offline (airgap) installation only. You must download the Humio Log Management for IBM Cloud Paks installation package from IBM Passport Advantage® . For more information, see the following installation steps. An online installation is not supported.
In the uninstall section in this information, a link to Humio Log Management documentation is included, complete the uninstall instructions that you are guided to only when you access this documentation.
The version of Humio Log Management currently supported is V1.18.2.

Prerequisites

Ensure IBM Cloud Pak® for Multicloud Management is installed on the hub cluster. For instructions and to choose your IBM Cloud Pak® for Multicloud Management installation route: online or offline, see IBM Cloud Pak® for Multicloud Management installation.
Review Humio's Hardware Requirements to ensure that your environment has sufficient resources to install Humio.
If you are not defining your own storage class, then you must ensure that you have a default storage class set. For more information, see Change the default Storage.

Installing Humio Log Management for IBM Cloud Paks

Download the Humio Log Management for IBM Cloud Paks installation package from IBM Passport Advantage® . Search for the installation package using its part number (G013XEN).

Note: An online installation is not supported. The only supported and tested installation route is to download the installation package in this step from IBM Passport Advantage® and complete an offline (airgap) installation.

For more information about Humio packages, part numbers, and IBM Passport Advantage®, see Partner product Passport Advantage part numbers.
Extract the installation package.
Open the README file, which will point you to the next steps to complete the Humio installation.

Important: Ensure that you complete the following items during the installation.

a. Before you complete the instructions in the README, create the namespace or project for Humio. For example, for OpenShift, run oc new-project logging.

b. If you have a cluster image policy, add the required registry entries. For example:
```
 - name: default-route-openshift-image-registry.silver-1-c0ec0aa088d92fd9ce009fc425f649bd-0000.us-south.containers.appdomain.cloud/*
 - name: docker.io/fluent/*
 - name: docker.io/humio/*
 - name: docker.io/solsson/*
 - name: docker.io/confluentinc
```
c. Ensure that you enter the domain/namespace for the private registry when you are prompted. For example:
```
 default-route-openshift-image-registry.silver-1-c0ec0aa088d92fd9ce009fc425f649bd-0000.us-south.containers.appdomain.cloud/logging
```
d. At the point in the installation when you are updating the values.yaml file, you must add the OpenID connect client configuration parameters and the OpenShift host for Humio to the values.yaml file. Complete step 1.a in Single sign-on configuration.

If you are using a secured registry, you must create a docker pull secret. For example:
```
 kubectl create secret docker-registry humio-pull-secret \
 --docker-server=default-route-openshift-image-registry.silver-1-c0ec0aa088d92fd9ce009fc425f649bd-0000.us-south.containers.appdomain.cloud \
 --docker-username=admin \
 --docker-password=XXX \
 --docker-email=user@us.ibm.com
```
Now, add the pull secret to the values.yaml.

e. If you are using Helm version 2, and the tiller tls is enabled, add the --tls flag to the helm install command. Also, specify --name <release_name> (for example: see --name humio) in the helm install command.

Finally, replace <number> with the number in the.tgz installation package file. It is similar or the same as 0.8.18.

Helm version 2 and Helm version 3 commands are included for your reference.

Helm v2:
```
 helm install --name humio --tls -f values.yaml humio-helm-charts-<number>.tgz --namespace logging
```
Helm v3+:
```
 helm install humio -f values.yaml humio-helm-charts-<number>.tgz --namespace logging
```
f. To retrieve the web URL to access Humio, run oc get routes.

g. If authentication is required for your docker registry, you must set imagePullSecrets for each statefulSet. For more information, see Pull an Image from a Private Registry.
After Humio is installed, complete the remaining steps in Single sign-on configuration (start at steps 1.b onwards). After Single sign-on is successfully configured, complete the steps in Launching the Humio console.

Single sign-on configuration

Configure the OpenID connect client configuration parameters in Humio. Then, create a new client registration in IBM Cloud Pak® for Multicloud Management, which is the OpenID Connect identity provider in the OpenID Connect protocol. You can then log in to the Humio platform with the same credentials that you are using in IBM Cloud Pak® for Multicloud Management. If you are already logged in to IBM Cloud Pak® for Multicloud Management, the same session is used to start the Humio platform, you do not need to log in separately to Humio.

Configure the OpenID connect client configuration parameters in Humio:

a. Copy the parameters in the following code snippet into the values.yaml file. Replace the variables with values. An example of:
- <base_domain> is blue-13.dev.multicloudops.io
- <public URL for Humio> is http://humio-humio-core-http-logging.apps.blue-13.dev.multicloudops.io
- <IBM Cloud Pak® for Multicloud Management_hostname> is https://cp-console.apps.blue-13.dev.multicloudops.io/idauth/oidc/endpoint/OP
Return to step 3.e in Installing Humio after you add these values.
```
---
humio-core:
 enabled: true
 openshift:
   host: <base_domain>
 authenticationMethod: oauth
 oauthConfig:
   publicUrl: <public URL for Humio>
   autoCreateUserOnSuccessfulLogin: true
 env:
 - name: OIDC_PROVIDER
 value: https://<IBM Cloud Pak for Multicloud Management_hostname>/idauth/oidc/endpoint/OP
 - name: OIDC_USERNAME_CLAIM
 value: email
 - name: OIDC_OAUTH_CLIENT_ID
 value: humio
 - name: OIDC_OAUTH_CLIENT_SECRET
 valueFrom:
   secretKeyRef:
     name: humio-client-secret
     key: password
```
b. Create the client secret by running the following command. Replace <Client-Secret-Data> with the password of your choice. It creates a Kubernetes secret containing the data.
```
kubectl create secret generic humio-client-secret --from-literal=password=<Client-Secret-Data>
```
Ensure the IBM Cloud Pak® for Multicloud Management oidcIssuerURL parameter is pointing to the IBM Cloud Pak® for Multicloud Management hostname. To check and complete the steps to change it, go to the Customizing the authentication URL section in Authentication onboarding and single sign-on. Then, complete the steps in Configuring IBM Cloud Platform Common Services to update the oidcIssuerURL parameter. The oidcIssuerURL parameter is listed in the Authentication settings table. For example, on OpenShift, to edit the authentication settings, run the following command:
```
oc edit cm platform-auth-idp -n ibm-common-services
```
The BASE_OIDC_URL and OIDC_ISSUER_URL variables look similar to the following assignments:
```
BASE_OIDC_URL: https://cp-console.silver-1-c0ec0aa088d92fd9ce009fc425f649bd-0000.us-south.containers.appdomain.cloud/oidc/endpoint/OP

OIDC_ISSUER_URL: https://cp-console.silver-1-c0ec0aa088d92fd9ce009fc425f649bd-0000.us-south.containers.appdomain.cloud/idauth/oidc/endpoint/OP
```
If your IBM Cloud Pak® for Multicloud Management installation is configured with valid CA signed certificates, you can ignore these sub steps.

a. Retrieve the certificate.
```
oc get secret -n ibm-common-services cs-ca-certificate-secret -o jsonpath='{.data.ca\.crt}' | base64 -d
```
b. Save the output from the command in the previous step into a certificate file. For example, you can name the file: cert1.pem.

c. Copy the certificate files into the /tmp directory in the Humio API pod. If you have two certificate files, you must run this command for each certificate file. Replace <api_pod_name> with the Humio API pod name and lt;cert_file> with the name of the certificate file. This step must be completed for each API pod. For example, humio-humio-core-0, humio-humio-core-1, and humio-humio-core-2.
```
oc -n humio cp <cert_file> <api_pod_name>:/tmp/
```
For example:
```
oc -n humio cp cert1.pem humio-humio-core-0:/tmp/
```
d. Access the Humio API container. You must complete this step for each API pod.
```
oc exec -it humio-humio-core-0 bash
```
e. Copy the Humio certificate files into a location where they can be modified. You must complete this step for each API pod. For example:
```
cp  /usr/lib/jvm/zulu-13-amd64/lib/security/cacerts /data/humio-data/
```
Note: Ensure you have write permissions for cacerts before you run the keytool command in the next step. If do not have write access, run the chmod +w /data/humio-data/cacerts command.

f. If Humio uses Java, you can import one or more certificates into Java's truststore by running the following command. If you have two certificate files, you must run this command for each certificate file and you must use a different alias each time. Replace <name> with the name of your choice, replace <cert_file> with the certificate file name, and replace <password> with the truststore password of your choice. You must complete this step for each API pod.
```
keytool -import -alias <name> -file /tmp/<cert_file> -keystore /data/humio-data/cacerts -storepass <password>
```
For example:
```
keytool -import -alias cert1 -file /tmp/cert1.pem -keystore /data/humio-data/cacerts -storepass changeit
```
g. Verify that the certificate is imported into Java's truststore. Run this command for each certificate. You must complete this step for each API pod.
```
keytool -list -v -keystore /data/humio-data/cacerts -storepass changeit -alias cert1
```
h. Restart the Humio API pod. Deleting the pod restarts it automatically.
```
oc delete pods humio-humio-core-0
```
i. Update the humio-core statefulset with JVM arguments pointing to new Java keystore. For example, on OpenShift, you can edit the humio-core statefulset by running oc edit statefulset humio-humio-core.
- Search for name: HUMIO_JVM_ARGS.
- Add the following to the existing Humio JVM arguments.
```
-Djavax.net.ssl.trustStore=/data/humio-data/cacerts 
-Djavax.net.ssl.trustStoreType=jks 
-Djavax.net.ssl.trustStorePassword=changeit
```
  The final Humio JVM arguments after you update them should be similar to the following:
```
- name: HUMIO_JVM_ARGS
       value: -Xss2m -Xms256m -Xmx1536m -server -XX:+UseParallelOldGC -XX:+ScavengeBeforeFullGC
         -XX:+DisableExplicitGC -Djavax.net.ssl.trustStore=/data/humio-data/cacerts
         -Djavax.net.ssl.trustStoreType=jks -Djavax.net.ssl.trustStorePassword=changeit
```

Register Humio as an OIDC client with IBM Cloud Pak® for Multicloud Management. There are different ways to complete this registration. The following substeps are using the Onboard by using cloudctl information that is documented in Authentication onboarding and single sign-on.

a. Create a registration.json file using the following template. For "client_secret", enter the same secret that was created in step 1.b. For "redirect_uris", enter the public URL for Humio.

 {
     "allow_regexp_redirects": false,
     "appPasswordAllowed": false,
     "appTokenAllowed": false,
     "application_type": "web",
     "client_id": "humio",
     "client_name": "humio",
     "client_secret": "<enter the client secret>",
     "functional_user_groupIds": [],
     "grant_types": [
         "authorization_code",
         "client_credentials",
         "password",
         "implicit",
         "refresh_token",
         "urn:ietf:params:oauth:grant-type:jwt-bearer"
     ],
     "hash_itr": 0,
     "hash_len": 0,
     "introspect_tokens": true,
     "post_logout_redirect_uris": [],
     "preauthorized_scope": "openid profile email general",
     "proofKeyForCodeExchange": false,
     "publicClient": false,
     "redirect_uris": [
         "https://<enter the the public URL for Humio>/auth/oidc"
     ],
     "resource_ids": [],
     "response_types": [
         "code",
         "token",
         "id_token token"
     ],
     "scope": "openid profile email",
     "subject_type": "public",
     "token_endpoint_auth_method": "client_secret_basic",
     "trusted_uri_prefixes": []
 }

b. To complete the registration, ensure you are logged in to cloudctl and run the following command as the cluster administrator:

   cloudctl iam oauth-client-register -f registration.json

If you run into this error:

   Error from server (NotFound): secrets "platform-oidc-credentials" not found.

Then, switch to the kube-system namespace by running this command:

   oc project kube-system

Then, re-run the previous cloudctl commands again to complete the registration.

c. If you want to verify that the registration is a success, run the following command. Replace <client_name> with the Humio client name.

   cloudctl iam oauth-client <client_name>

Launching the Humio console

You can now access the Humio navigation menu directly from IBM Cloud Pak® for Multicloud Management console, select this menu. You are re-directed to the Humio console.

Note: If the Humio menu is not displayed, restart the common-web-ui pods in the ibm-common-services namespace to fix this issue.

Uninstalling Humio

If you want to uninstall Humio later, see Uninstalling (scroll to the end of the page to access the uninstall steps).