Provisioning a Data Virtualization instance does not complete

After you provision a Data Virtualization instance, instance shows up in the UI and it shows that it is provisioning, but the provisioning does not complete.

Symptoms

The instance provisioning does not complete. After you wait for the instance provisioning to complete, the pods appear to be running. However, the SETUP_COMPLETE marker is not created and the head and worker pods do not start. In the provisioning UI, an error message eventually indicates that the provisioning process has timed out.

Causes

The Db2U restore-morph script exits early and does not complete all of the steps. The head pod waits for the SETUP_COMPLETE marker that db2u_restore_morph.sh script fails to create.

Diagnosing the problem

After you wait for the instance to be provisioned, determine whether the head pod is waiting for the SETUP_COMPLETE marker. Run the following command to see the markers list. If the SETUP_COMPLETE marker is missing from the marker list, the pod is still waiting for the marker.
```
oc exec -i c-db2u-dv-db2u-0 -n dv1 -c db2u -- bash -c 'db2uctl markers list'
```
Run the following command to see if the db2u_restore_morph.sh script has completed:
```
oc get job c-db2u-dv-restore-morph
```
If the COMPLETIONS column shows 1/1, as is shown in the following example, and the SETUP_COMPLETE marker is missing, then it is confirmed that you have encountered this issue.
```
NAME                     COMPLETIONS  DURATION  AGE
c-db2u-dv-restore-morph  1/1          3m42s     41h
```

If the head pod is waiting for the marker, determine whether the db2u_restore_morph.sh script exited early. In the /var/log/db2u.log file in the c-db2u-dv-db2u-0 pod, look at the last entry for a message similar to the following:

2023-06-12T01:04:21.073626482Z RESTORE DATABASE BLUDB CONTINUE
2023-06-12T01:04:21.073626482Z + read IN
2023-06-12T01:04:32.892167124Z + exit 0

Normally the next message would be similar to the following:

2023-05-26T00:08:18.455178792Z RESTORE DATABASE BLUDB CONTINUE
2023-05-26T00:08:18.455178792Z + read IN
2023-05-26T00:08:44.455302328Z + logger 'INFO: DB20000I  The RESTORE DATABASE command completed successfully.'
2023-05-26T00:08:44.455302328Z + [[ -z /var/log/db2u.log ]]

Resolving the problem

Delete the instance and provision a new Data Virtualization instance.