Data Virtualization fails to start even though provisioning succeeds

After you provision Data Virtualization successfully, your Data Virtualization instance fails to start.

Symptoms

On an OpenShift® cluster with slow I/O performance, that is, much slower than the minimal requirements that are described in Hardware requirements, the Data Virtualization instance might not start after you provision it.

The error message in the Kubernetes pod log from the c-db2u-dv-db2u-0 log looks similar to the following example.

2021-09-29T12:44:45.342695864Z + local 'output=2021-09-29_12.44.45,341_UTC  INFO Not all workers have /mnt/blumeta0/home/db2inst1/hosts/.joined_to_cluster created. Wait for 30 seconds then recheck'
2021-09-29T12:44:45.342772836Z + echo -e 2021-09-29_12.44.45,341_UTC INFO Not all workers have /mnt/blumeta0/home/db2inst1/hosts/.joined_to_cluster created. Wait for 30 seconds then recheck
2021-09-29T12:44:45.342781156Z 2021-09-29_12.44.45,341_UTC INFO Not all workers have /mnt/blumeta0/home/db2inst1/hosts/.joined_to_cluster created. Wait for 30 seconds then recheck
2021-09-29T12:44:45.342846458Z + echo -e 2021-09-29_12.44.45,341_UTC INFO Not all workers have /mnt/blumeta0/home/db2inst1/hosts/.joined_to_cluster created. Wait for 30 seconds then recheck
2021-09-29T12:44:45.342989571Z + sleep 30

Resolving the problem

To work around this issue, run the following manual steps.

  1. Manually create an empty marker file in each worker pod. Run the following command to get the list of worker pods.
    oc -n ${PROJECT_CPD_INST_OPERANDS} get pods | grep -i 'c-db2u-dv-db2u' | grep -v db2u-0 | cut -d' ' -f 1
  2. Create a file that is named /mnt/blumeta0/home/db2inst1/hosts/.registeredHeadCID in each worker pod and make sure it has the same content as the /mnt/blumeta0/home/db2inst1/hosts/.registeredHeadCID file in the head pod.
  3. Run bigsql stop; bigsql start in the c-db2u-dv-db2u-0 head pod by using the following command.
    oc -n ${PROJECT_CPD_INST_OPERANDS} exec -it c-db2u-dv-db2u-0 -- su - db2inst1 -c "bigsql stop; bigsql start"