Unable to connect to Db2 database instance through the Db2Rest add-on pod

When you try to connect to your Db2 database instance by using the Db2Rest add-on pod, you get an error message from the Db2 CLI driver and are unable to connect.

Symptoms

You notice the following behavior:

You cannot connect to your Db2 database instance (Db2 s11.5.9.0-cn1)
You get an error message from the Db2 CLI driver
The SSL keystore and keystash files weren't created

The GSKit does not create the keystore and keystash files correctly, which causes GSKit commands to fail. This problem happens because the Db2Rest entrypoint script references a file that does not exist in the rest pod (db2_encryption_functions.sh).

Resolving the problem

The workaround involves patching the problematic entrypoint script with a new one.

Save the following script as db2u_rest_entrypoint.sh:

db2u_rest_entrypoint.sh

#!/bin/bash

# *******************************************************************************
#
# COPYRIGHT: P#2 P#1
# (C) COPYRIGHT IBM CORPORATION 2019
#
# The source code for this program is not published or otherwise divested of
# its trade secrets, irrespective of what has been deposited with the U.S.
# Copyright Office.
#
# *******************************************************************************
source ${DB2REST_INSTALL_PATH}/.bashrc
STARTTIME=$(date '+%Y%m%d%H%M%S')
echo "STARTTIME=$STARTTIME"

# Get the PID 1 environ into the container namespace
xargs -L1 -0 echo < /proc/1/environ > /tmp/env
export IBM_PRODUCT="IBM Db2 REST"
source ${DB2U_SCRIPTS}/include/db2u_logger.sh

DB2REST_AUTH_PATH=${DB2REST_INSTALL_PATH}/auth
DB2REST_TLS_KEY=${DB2REST_AUTH_PATH}/dbrest.key
DB2REST_TLS_CRT=${DB2REST_AUTH_PATH}/dbrest.pem

[[ -d ${DB2REST_AUTH_PATH} || "$(ls -A ${DB2REST_AUTH_PATH})" ]] || mkdir -p ${DB2REST_AUTH_PATH}
if [[ ! -f ${DB2REST_TLS_KEY} ]]; then
  #Add SAN to REST self signed cert
  REST_SANS="DNS:$(hostname -f),DNS:$(hostname -s)"

  openssl req -newkey rsa:2048 -nodes -keyout ${DB2REST_TLS_KEY} -x509 -days 365 -out ${DB2REST_TLS_CRT} -subj "/C=CA/ST=ON/L=TO/O=IBM=OU=Db2Rest/CN=ibm.com" -extensions SAN -config <(cat /etc/pki/tls/openssl.cnf <(printf "\n[SAN]\nsubjectAltName=$REST_SANS"))
  chmod 600 ${DB2REST_AUTH_PATH}/dbrest.*
fi

#
### Write DB2REST properties file ###
#
cat <<EOF >> ${DB2REST_CONFIG_PATH}/server.env
DB2REST_TLS_PORT=50050
DB2REST_TLS_KEY=${DB2REST_TLS_KEY}
DB2REST_TLS_CRT=${DB2REST_TLS_CRT}
EOF

#Create shared storage
SHARED_DB2REST_DIR=/mnt/blumeta0/db2rest
TEMP_CONFIG_LOG=${SHARED_DB2REST_DIR}/temp_config.log
CONFIG_LOG=${DB2REST_INSTALL_PATH}/logs/config.log
if [[ ! -d ${SHARED_DB2REST_DIR} || ! "$(ls -A ${SHARED_DB2REST_DIR})" ]]; then
    sudo mkdir -p ${SHARED_DB2REST_DIR}
    sudo chown -R ${REST_USER} ${SHARED_DB2REST_DIR}
    echo "Created shared directory" > ${TEMP_CONFIG_LOG} 2>&1
else
    echo "Shared directory already exists" > ${TEMP_CONFIG_LOG} 2>&1
fi

#Create shared logs directory
SHARED_DB2REST_LOG_DIR=${SHARED_DB2REST_DIR}/logs
if [[ ! -d ${SHARED_DB2REST_LOG_DIR} || ! "$(ls -A ${SHARED_DB2REST_LOG_DIR})" ]]; then
    sudo mkdir -p ${SHARED_DB2REST_LOG_DIR} >> ${TEMP_CONFIG_LOG} 2>&1
    sudo chown -R ${REST_USER} ${SHARED_DB2REST_LOG_DIR} >> ${TEMP_CONFIG_LOG} 2>&1
    echo "Created shared logs directory" >> ${TEMP_CONFIG_LOG} 2>&1
else
    echo "Shared logs directory already exists" >> ${TEMP_CONFIG_LOG} 2>&1
fi
if [[ ! -d ${DB2REST_INSTALL_PATH}/logs ]]; then
    sudo ln -s ${SHARED_DB2REST_LOG_DIR} ${DB2REST_INSTALL_PATH}/logs >> ${TEMP_CONFIG_LOG} 2>&1
    echo "Created shared logs symlink" >> ${TEMP_CONFIG_LOG} 2>&1
else
    echo "Shared logs symlink already exists" >> ${TEMP_CONFIG_LOG} 2>&1
fi
if [[ -f ${CONFIG_LOG} ]]; then
    rm ${CONFIG_LOG} >> ${TEMP_CONFIG_LOG} 2>&1
fi
mv ${TEMP_CONFIG_LOG} ${CONFIG_LOG}

#Create and symlink the app.yaml file 
DB2REST_CFG_FILE=app.yaml
DEFAULT_DB2REST_CFG_FILE=default.app.yaml
SHARED_DB2REST_CFG_FILE=${SHARED_DB2REST_DIR}/${DB2REST_CFG_FILE}
if [[ ! -f ${SHARED_DB2REST_CFG_FILE} ]]; then
    cp ${DB2REST_CONFIG_PATH}/${DEFAULT_DB2REST_CFG_FILE} ${SHARED_DB2REST_CFG_FILE} >> ${CONFIG_LOG} 2>&1
else
    echo "Shared Db2 REST configuration file already exists" >> ${CONFIG_LOG} 2>&1
fi
if [[ ! -f ${DB2REST_CONFIG_PATH}/${DB2REST_CFG_FILE} ]]; then
    sudo ln -s ${SHARED_DB2REST_CFG_FILE} ${DB2REST_CONFIG_PATH}/${DB2REST_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    echo "Create Db2 REST configuration file symlink" >> ${CONFIG_LOG} 2>&1
else
    echo "Shared Db2 REST configuration symlink already exists" >> ${CONFIG_LOG} 2>&1
fi
 
#Create Db2 REST keystore/keystash
SSL_KEY_PREFIX=bludb_ssl
SSL_KEYSTORE_FILE=${SHARED_DB2REST_DIR}/${SSL_KEY_PREFIX}.kdb
SSL_KEYSTASH_FILE=${SHARED_DB2REST_DIR}/${SSL_KEY_PREFIX}.sth
DB2_ROOT_CERTIFICATE=/secrets/db2ssl/ca.crt
if [[ -f ${DB2_ROOT_CERTIFICATE} ]]; then
    echo "Root certificate found" >> ${CONFIG_LOG} 2>&1
    if [[ -f ${SSL_KEYSTORE_FILE} ]]; then
        echo "Remove existing key artifacts" >> ${CONFIG_LOG} 2>&1
        rm -rf ${SHARED_DB2REST_DIR}/${SSL_KEY_PREFIX}.*
    fi
    ssl_keystore_pw=`cat /dev/urandom | tr -dc '[:alnum:][=@=][=_=]' | fold -w ${1:-29} | head -n 1`
    gsk8capicmd_64 -keydb -create -pqc false -db ${SSL_KEYSTORE_FILE} -pw ${ssl_keystore_pw} -type cms -stash >> ${CONFIG_LOG} 2>&1
    gsk8capicmd_64 -cert -add -db ${SSL_KEYSTORE_FILE} -pw ${ssl_keystore_pw} -file ${DB2_ROOT_CERTIFICATE} -format ascii >> ${CONFIG_LOG} 2>&1
else
    echo "Root certificate not found" >> ${CONFIG_LOG} 2>&1
fi

#Create and symlink the db2dsdriver.cfg file
DRIVER_CFG_FILE=db2dsdriver.cfg
DRIVER_HADR_CFG_FILE=db2dsdriver.sample.hadr.cfg
SHARED_DRIVER_CFG_FILE=${SHARED_DB2REST_DIR}/${DRIVER_CFG_FILE}
SHARED_DRIVER_HADR_CFG_FILE=${SHARED_DB2REST_DIR}/${DRIVER_HADR_CFG_FILE}
if [[ ! -f ${SHARED_DRIVER_CFG_FILE} ]]; then
    cp ${DB2REST_CONFIG_PATH}/db2dsdriver.sample.cfg ${SHARED_DRIVER_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    cp ${DB2REST_CONFIG_PATH}/db2dsdriver.sample.hadr.cfg ${SHARED_DB2REST_DIR}/db2dsdriver.sample.hadr.cfg >> ${CONFIG_LOG} 2>&1
    sed -i "s/DB2_HOSTNAME/${DB2REST_DBHOSTNAME}/" ${SHARED_DRIVER_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    sed -i "s/DB2_PORT/${DB2REST_DBPORT}/" ${SHARED_DRIVER_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    sed -i "s/DBNAME/${DB2REST_DBNAME}/" ${SHARED_DRIVER_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    sed -i "s,PATH_TO_KEYSTORE,${SSL_KEYSTORE_FILE}," ${SHARED_DRIVER_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    sed -i "s,PATH_TO_KEYSTASH,${SSL_KEYSTASH_FILE}," ${SHARED_DRIVER_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    sed -i "s/DB2_HOSTNAME/${DB2REST_DBHOSTNAME}/" ${SHARED_DRIVER_HADR_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    sed -i "s/DB2_PORT/${DB2REST_DBPORT}/" ${SHARED_DRIVER_HADR_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    sed -i "s/DBNAME/${DB2REST_DBNAME}/" ${SHARED_DRIVER_HADR_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    sed -i "s,PATH_TO_KEYSTORE,${SSL_KEYSTORE_FILE}," ${SHARED_DRIVER_HADR_CFG_FILE} >> ${CONFIG_LOG} 2>&1
    sed -i "s,PATH_TO_KEYSTASH,${SSL_KEYSTASH_FILE}," ${SHARED_DRIVER_HADR_CFG_FILE} >> ${CONFIG_LOG} 2>&1
else
    echo "Shared driver configuration already exists" >> ${CONFIG_LOG} 2>&1
fi
if [[ ! -f ${DB2REST_DB2HOME}/cfg/${DRIVER_CFG_FILE} ]]; then
    sudo ln -s ${SHARED_DRIVER_CFG_FILE} ${DB2REST_DB2HOME}/cfg/${DRIVER_CFG_FILE} >> ${CONFIG_LOG} 2>&1
else
    echo "Driver configuration already exists" >> ${CONFIG_LOG} 2>&1
fi

logger_info "Starting up ${IBM_PRODUCT} container..."
${DB2REST_INSTALL_PATH}/scripts/db2rest-start.sh

# Setup an init banner for the rest container
cat <<EOF
#############################################################${HASHES}
###  ${IBM_PRODUCT} container was started successfully   ###
#############################################################${HASHES}
* If you used docker/oc/kubectl logs to monitor progress, detach from the
console by pressing Ctrl+C.
* To get a command line prompt, issue the following command from the host:
    [ docker | oc | kubectl -n <NS> ] exec -it <rest container name> bash
#############################################################${HASHES}
EOF

# End-marker to enable readiness probe when deployed via charts
DB2U_MARKER_FILE=${DB2U_DIR}/.db2u_rest_initialized
touch ${DB2U_MARKER_FILE}

set +x

# Keep the container up in detached(-d) mode, unless --exit arg is passed
#/usr/bin/bash
[[ "X${1}" != "X--exit" ]] && \
    tail -f /dev/null

Save the following script as patch_script_cm.sh:

patch_script_cm.sh

#!/bin/bash

# Example usage: ./patch_script_cm.sh db2oltp-crd-hadr-primary standalone-rc manage_hadr.sh /db2u/scripts 755 false
DB2U_ID=$1
NAMESPACE=$2
FILE=$3
MOUNT_PATH=$4
PERM=$5
READ_ONLY=$6

FILENAME=$(basename -- "${FILE}")
EXTENSION="${FILENAME##*.}"
FILENAME="${FILENAME%.*}"
CM=$(echo "${FILENAME}-${EXTENSION}" | tr '_' '-')

echo "-------------------------------------------------"
echo -e "(Re)creating configmap ${CM}...\n"
kubectl create configmap ${CM} -n ${NAMESPACE} --from-file=${FILE}=${FILE} -o yaml --dry-run=client | kubectl apply -f -

echo -e "\n-------------------------------------------------"
echo -e "Mounting ${CM} CM to ${DB2U_ID}-rest deployment..."
DB2U_STS=c-${DB2U_ID}-rest
oc set volume deployment/${DB2U_STS} -n ${NAMESPACE} --add --name=${CM} --type=configmap --containers=rest --mount-path="${MOUNT_PATH}/${FILE}" --sub-path="${FILE}" --configmap-name=${CM} --default-mode="${PERM}" --read-only=${READ_ONLY} --overwrite

In a separate terminal window, log into your cluster's infrastructure node and run the following command:
```
mkdir /root/rest_patch
```
From your local machine (not the infrastructure node terminal window), run the following command:
```
scp db2u_rest_entrypoint.sh root@<CLUSTER
          HOSTNAME>:/root/rest_patch/
```

And then run the following command:

scp patch_script_cm.sh root@<CLUSTER HOSTNAME>:/root/rest_patch/

Switch back to your infrastructure node terminal window and run the following command:

cd /root/rest_patch && chmod 777 patch_script_cm.sh && ./patch_script_cm.sh <NAME OF YOUR
          DB2U DEPLOYMENT> <NAMESPACE OF DB2U DEPLOYMENT> db2u_rest_entrypoint.sh /db2u 755
          false

The rest pod re-creates and fixes the problem.