Known issues and limitations for Common core services

Assets cannot be imported when using the Presto C++ engine in the IBM watsonx.data Presto connection

Applies to: 5.3.0

Assets cannot be imported when using the Presto C++ engine in the IBM watsonx.data Presto connection.

Workaround: To fix this issue, use the Presto Java engine in the connection instead of the watsonx.data Presto engine. In the connection, specify the following connection details and point them to the Presto Java engine:

• Engine host
• Engine port
• Engine ID

The watsonx.data Presto connection may fail when using internal ports

Applies to: 5.3.0

The watsonx.data Presto connection may fail when you use the internal port from the Presto engine to create a watsonx.data Presto connection in a project.

Workaround: You will need to provide the SSL certificate in the watsonx.data Presto connection when connecting to the Presto engine in watsonx.data within the same OpenShift cluster.

The watsonx.data Presto connector may return with an error when accessing data

Applies to: 5.3.0

The watsonx.data Presto connector may return the following error when using the Presto C++ engine to query data:

sortedCandidates is null or empty for ModularHashingNodeProvider

Workaround: Restart the Presto C++ engine worker pods.

Delta Lake catalog can have columns with incorrect decimal precision

Applies to: 5.3.0

When creating a Delta Lake table, you can set the decimal precision for the table too high and the write job will accept the precision value. However, when you try to read this table later, it will not be able to read it.

It might show the following error:

ERROR BrowsingService.getAssets: Unable to retrieve data: The assets request failed: CDICO9999E: Internal error occurred: SCAPI error: Columns could not be obtained: Executing query failed with error: presto: query failed (200 OK): "Illegal argument error: Invalid precision and scale combo (100, 0). They should be in the range [0, 38] and scale can not be more than the precision."

Workaround: Set the decimal precision value to be in the range of 0 to 38.

Error for Oracle connections when running queries that contain user-defined data types (UDTs)

Applies to: 5.3.0

If you run an SQL query on a table or a view with columns that contain user-defined data types (UDTs), you might receive an error that contains this message:

[IBM][Oracle JDBC Driver]User defined type not found

The driver requires access to the metadata associated with the user-defined type (UDT) during the query execution. If you only have the SELECT privilege for the table, but you do not have the corresponding privilege for the UDT, the driver cannot access the UDT's metadata.

Workaround: Grant the necessary privilege to the UDT. For example:
GRANT EXEC ON TYPE::[schema].[typename] TO [User];

CouchDB unexpectedly restarts in a non-FIPS environment

Applies to: 5.3.0

When running a cluster with CouchDB in a non-FIPS environment, the CouchDB container may unexpectedly restart. For example:

$ oc get pod -l app=couchdb
NAME            READY   STATUS    RESTARTS         AGE
wdp-couchdb-0   2/2     Running   12 (4m34s ago)   16m
wdp-couchdb-1   2/2     Running   10 (1m15s ago)   16m
wdp-couchdb-2   2/2     Running   2 (4m29s ago)    17m

Workaround: To fix this issue, run the following steps:

1. Access the namespace where IBM Knowledge Catalog is installed:

   $ oc project ${PROJECT_CPD_INST_OPERANDS}
   

2. Update the wdp-couchdb configmap to disable FIPS_mode, which is enabled by default:

   $ oc get configmap wdp-couchdb -o yaml | sed 's/fips_mode true/fips_mode false/' | oc apply -f -
   

3. Once the configmap is updated, conduct a rolling restart of the CouchDB pods:

   $ oc rollout restart sts wdp-couchdb
   

4. After all the CouchDB pods have restarted, place the common core services into maintenance mode to avoid the restarting issue being brought back by CCS reconciling:

   $ oc patch ccs ccs-cr --type merge --patch '{"spec": {"ignoreForMaintenance": true}}'
   

   You may see warning messages like the following when the change is applied:

   $ oc get configmap wdp-couchdb -o yaml | sed 's/fips_mode false/fips_mode true/' | oc apply -f -
   Warning: resource configmaps/wdp-couchdb is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
   configmap/wdp-couchdb configured
   

   This is to be expected and can be ignored, as long as the configmap was successfully configured and you can see a message like the following at the end of the warning message:

   configmap/wdp-couchdb configured
   

5. You can confirm that the changes have been made by running the following command, to confirm that fips_mode has been set to false:

   $ oc get configmap wdp-couchdb -o yaml |grep " -crypto fips_mode false"
    -crypto fips_mode false
   

Ingesting data to default or externally added buckets by using watsonx.data Presto connector might fail

Applies to: 5.3.0

Ingesting data into default buckets or externally added buckets by using the watsonx.data Presto connector with the data policies defined in watsonx.data might fail.

Workaround: Use File connectors to ingest data into storage associated in watsonx.data.

For example, you can use Generic S3 connector to ingest data into any S3 compliant storage in watsonx.data, like Amazon S3 storage or IBM Cloud Object Storage.

Appended data by the IBM watsonx.data connector may not be immediately visible when running queries using the Presto engine

Applies to: 5.3.0

When appending data to a table created using the Presto engine, the appended rows may not be immediately visible when running read queries. This is because the Presto engine caches the state of the tables every two minutes by default.

Workaround
Run the following to fix this issue:

1. Modify the Presto config map on the OpenShift pod by running:

   oc edit cm <presto-config-map>
   

   For example:

   oc edit cm ibm-lh-lakehouse-presto-01-presto-config-cm-blue
   

2. Delete the following properties from the Presto config map:

   hive.metastore-cache-scope=ALL
   hive.metastore-cache-ttl=10m
   hive.metastore-refresh-interval=2m
   

   For example, the config map may show:

   LH_METASTORE_CACHE_ENABLED: "true"
   LH_METASTORE_CACHE_SCOPE: ALL
   LH_METASTORE_CACHE_TTL: 10m
   LH_METASTORE_REFRESH_INTERVAL: 2m
   

The ccs-cr custom resource might get stuck in In Progress status when Portworx storage is full

Applies to: 5.3.0 and later

If Portworx storage is full on all worker nodes for two hours or more, the ccs-cr custom resource can get stuck in In progress status. This is caused by the rabbitmq pods continuously restarting. However, no obvious error is logged in the rabbitmq pod logs.

Workaround: Manually trigger recovery of the rabbitmq pods:

1. Scale down rabbitmq-ha statefulset to 0 replicas:

   oc scale sts/rabbitmq-ha --replicas=0
   

2. Delete the rabbitmq-ha PVCs:

   oc delete pvc -l icpdsupport/module=rabbitmq-ha
   

3. Scale up rabbitmq-ha statefulset to 3 replicas

   oc scale sts/rabbitmq-ha --replicas=3
   

After the rabbitmq statefulset is up, the rabbitmq pods start successfully and the ccs-cr moves to the Completed status.

List of compatible tools is incorrect for some connectors

Applies to: 5.3.0
Fixed in: 5.3.1

The tiles for the Amazon Aurora for MySQL and Amazon Aurora for PostgreSQL connectors show metadata import, data quality rules, and metadata enrichment as compatible tools. However, you can currently not use the Amazon Aurora for MySQL and Amazon Aurora for PostgreSQL connectors with these tools.

The ccs-cr does not reconcile after updating the common core services custom resource (ccs-cr)

Applies to: 5.3.1

After updating the ccs-cr with the oc patch command, the CCS custom resource does not reconcile. The CCS custom resource shows the updates, but it does not automatically trigger reconciliation.

Workaround: Run the following command to patch the ccs operator to increase the /tmp disk size.

   oc patch deployment ibm-cpd-ccs-operator --type='json' -p='[
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/resources/limits/ephemeral-storage",
    "value": "5Gi"
  },
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/resources/limits/memory",
    "value": "5Gi"
  },
  {
    "op": "replace",
    "path": "/spec/template/spec/volumes/0/emptyDir/sizeLimit",
    "value": "5Gi"
  },
  {
    "op": "replace",
    "path": "/spec/template/spec/volumes/1/emptyDir/sizeLimit",
    "value": "5Gi"
  },
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/resources/requests/ephemeral-storage",
    "value": "5Gi"
  }
]'

After upgrading from version 5.3.0 to version 5.3.1 Patch 5 in a FIPS-enabled cluster, pods might run out of scratch space

Applies to: 5.3.1 Patch 5
Fixed in: 5.3.1 Patch 6

When you upgrade from version 5.3.0 to version 5.3.1 Patch 5, pods might run out of scratch space, which results in those pods being unhealthy. The following error message is shown:

Usage of EmptyDir volume "scratch" exceeds the limit "500Mi".

Workaround: Remove the limit on scratch space. The entry in the Volumes: section of the ConfigMap should look as follows:

- name: scratch
  emptyDir: {}

watsonx.data Presto connector can't write to watsonx.data as a Service

The IBM watsonx.data Presto connector cannot write to IBM watsonx.data as a Service Lite in version 2.3.

IBM watsonx.data Presto connector can not write to S3 storage when using IAM role authentication

In watsonx.data you can connect to S3 storage using the IAM role authentication method. The watsonx.data Presto will not be able to write data to the catalog associated with that storage.

Disabling SSL certificate validation requires extra steps

If users want to explicitly disable the SSL certificate validation, users will have to perform extra steps to ensure that it has been disabled.

Procedure
When creating a new connection or editing an existing connection in the user interface:

1. Select the option to Validate the SSL certificate.
2. Unselect the option to Validate the SSL certificate.
3. Save the connection.

This will need to be performed for any connection where users want to disable the SSL certificate validation.

The watsonx.data and Presto connections do not work with specific data types

Applies to: 5.3.0

The watsonx.data Presto and Presto connections currently do not work with the following data types:

• TIME

The TIME and TIMESTAMP data types are not supported when the Presto C++ engine is used.