Operations against an object store are hanging or timing out (Db2 Big SQL)

Operations against an on-premises object store, such as CREATE TABLE, hang, or time out.

Symptoms

You observe that operations such as CREATE TABLE or general interactions with an on-premises object store such as Red Hat® OpenShift® Container Storage are hanging or timing out.

Causes

This problem might be related to how the S3 API converts the s3a locations that are used in Db2® Big SQL to the underlying HTTP requests that are sent to the object store endpoint.

The AWS Java™ SDK for S3 default is to use bucket prefix (also known as virtual hosting) notation when requests are sent. If your object store endpoint is s3.example.com, then requests are sent to bucket.s3.example.com. However, some setup is needed on the storage infrastructure to support this behavior, as described in the article regarding Bucket prefix vs Path access Anatomy of the S3A filesystem client.

Resolving the problem

If the object store is not configured to support Bucket prefix URLs, then you can enable path style access by setting the configuration fs.s3a.path.style.access to true. In this configuration, requests are sent to s3.example.com/bucket instead of bucket.s3.example.com.

  1. Log in to your OpenShift cluster as a project administrator:
    oc login OpenShift_URL:port
  2. Change to the project where the Cloud Pak for Data control plane is installed:
    oc project Project
  3. Create a cpd-cli profile.

    To manage Db2 Big SQL instances (by running the cpd-cli service-instance command), you must create a cpd-cli profile. The profile must be set up with the identity of a user that was granted access to the Db2 Big SQL instance. For more information about creating a cpd-cli profile, see Creating a profile to use the cpd-cli management commands.

  4. Identify the Db2 Big SQL instance ID.
    ./cpd-cli service-instance list --service-type bigsql --profile <profile-name>
  5. Get the name of the Db2 Big SQL head pod:
    head_pod=$(oc get pod -l instance=<instanceid>,app=db2-bigsql,bigsql-node-role=head --no-headers=true -o=custom-columns=NAME:.metadata.name)
  6. Set the path style access configuration:
    oc exec $head_pod -- bash -c '/usr/ibmpacks/IBM-Big_SQL/current/bigsql-cli/python/bigsql_config.py set-property --xml-path /etc/hadoop/conf/core-site.xml --name "fs.s3a.path.style.access" --value "true"'
  7. Restart Db2 Big SQL:
    oc exec -i $head_pod /usr/ibmpacks/IBM-Big_SQL/current/standalone/scripts/bigsql-restart.sh
  8. To persist the modified configuration over head pod restarts, run the following command:
    oc exec -i $head_pod -- bash -c 'mkdir -p /mnt/PV/shared; echo /usr/ibmpacks/IBM-Big_SQL/current/bigsql-cli/python/bigsql_config.py set-property --xml-path /etc/hadoop/conf/core-site.xml --name fs.s3a.path.style.access --value true >> /mnt/PV/shared/hook.sh'
  9. Retry the operation that was previously hanging and confirm that is now completing successfully.