Security Bulletin: Log4j vulnerability CVE-2021-44228 affects IBM Cloud Pak for Data System 1.0

Workarounds and Mitigations

Customers on ICPDS  v1.0.0.0- 1.0.7.5 should apply the mitigation below

Mitigation For OpenShift Container Platform  3.11

Note: Below mitigation is needed and applicable if openshift-logging is enabled on system.

Follow below steps to identify if the system has openshift-logging enabled:

1) Login to control vm e1n1-1-control :  ssh e1n1-1-control

2) Run below command:

oc get dc -n openshift-logging

Example:

When openshift-logging is enabled:

$ oc get dc -n openshift-logging
NAME                              REVISION   DESIRED   CURRENT   TRIGGERED BY
logging-es-data-master-76ovaz98   2          1         1
logging-kibana                    1          1         1         config

When openshift-logging is NOT enabled:

$ oc get dc -n openshift-logging
No resources found.

Follow the below steps to mitigate the reported CVE-2021-44228 for Openshift Container Platform 3.11

1) Login to control vm e1n1-1-control : ssh e1n1-1-control

Run below commands as apadmin user:

2) Change to project where Logging stack deployed (by default "openshift-logging" project)

$ oc project openshift-logging

3) Find the 'elasticsearch' deploymentConfigs deployed for passing later to oc set env command

$ oc get dc -l component=es

NAME                              REVISION   DESIRED   CURRENT   TRIGGERED BY
logging-es-data-master-kfity61t   9          1         1         
logging-es-data-master-o68rc18y   4          1         1         
logging-es-data-master-u6hh29n4   3          1         1         

4) Set environment variable 'ES_JAVA_OPTS' in 'elasticsearch' for system property log4j2.formatMsgNoLookups to true

$ oc set env -c elasticsearch dc/<elasticsearch_deploymentConfig_name> ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"

Note:
Please check if there are already some custom environment variables set for ES_JAVA_OPTS and append them if needed.

5) Confirm before rolling out the variable is present:

$ oc set env -c elasticsearch dc -l component=es --list | grep ES_JAVA_OPTS

6) Rollout new replicationControllers for 'pods' to start with the new values:. Do this for all deploymentConfigs:

$ oc rollout latest dc/<deploymentConfig_name>

7) Check new ES pod has been spawned automatically after the rollout:

$ oc get pods -l component=es

NAME                                            READY   STATUS    RESTARTS   AGE
elasticsearch-cdm-ba9c6evk-1-796f6cfdbc-4dqc6   2/2     Running   0          27m
elasticsearch-cdm-ba9c6evk-2-7959d4d857-z5km9   2/2     Running   0          2d9h
elasticsearch-cdm-ba9c6evk-3-5f9c5d668c-cr8lj   2/2     Running   0          2d9h

8) Open a shell into the newly-spawned 'ES pods' to check Java command-line arguments passed correctly including "-Dlog4j2.formatMsgNoLookups=true"

$ for es_pod in $(oc get pods -l component=es --no-headers -o jsonpath='{range .items[?(@.status.phase=="Running")]}{.metadata.name}{"\n"}{end}'); \
   do echo "Confirm changes on $es_pod" ;  sleep 1 ; \
   oc rsh -Tc elasticsearch $es_pod ps auxwww | grep log4j2.formatMsgNoLookups ; sleep 3; \
   done

-Dlog4j2.formatMsgNoLookups=true should be visible in above output

9) The pods should also have this variable set:

$ for es_pod in $(oc get pods -l component=es --no-headers -o jsonpath='{range .items[?(@.status.phase=="Running")]}{.metadata.name}{"\n"}{end}'); \
   do echo "Confirm changes on $es_pod" ;  sleep 1 ; \
   oc rsh -Tc elasticsearch $es_pod printenv | grep ES_JAVA_OPTS ; sleep 3; \
   done

You should see something like:

ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"
ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"
ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"