Known issues and limitations for Watson Discovery
The following known issues and limitations apply to the Watson Discovery service.
- OpenSearch permission denied while copying certificate during initialization
- CR gets stuck after the node reboot
- CR gets stuck In Progress after the node reboot
- Error is displayed after applying the temporary patch
- Watson Discovery is not accessible after upgrading
- Custom resources are not accessible from the Teach domain concepts section after upgrading
- During shutdown the DATASTOREQUIESCE field does not update
- Upgrade fails due to existing Elasticsearch 6.x indices
- UpgradeError is shown after resizing PVC
- Disruption of service after upgrading, restarting, or scaling by updating scaleConfig
OpenSearch permission denied while copying certificate during initialization
Applies to: 5.2.2 and later
- Error
-
When Watson Discovery is installed on clusters that use NetApp Trident storage, OpenSearch pods might not initialize successfully and can remain in a
CreateContainerErrorstate due to permission denied errors while copying certificates during initialization.You see the OpenSearch pod status similar to the following:
wd-discovery-opensearch-client-000 0/1 CreateContainerError 0If the OpenSearch pods show a
CreateContainerErrorstate, check the OpenSearch custom resource by running the following command:oc describe clusters.opensearch.cloudpackopen.ibm.com/wd-discovery-opensearchLook for messages such as the following, which indicate permission denied errors during container initialization while mounting certificates:
type: Progressing nodePoolPhase: client: phase: Degraded reason: 0 of 2 available and failed with Pod wd-discovery-opensearch-client-001 failed with container opensearch due to CreateContainerError: container create failed: error during container init: error mounting "/workdir/opensearch/config/certificate/http/default": permission denied data: phase: Degraded reason: 1 of 2 available and failed with Pod wd-discovery-opensearch-data-000 failed with container opensearch due to CreateContainerError: permission denied master: phase: Available phase: Degraded
- Solution
-
To resolve this issue, run the following commands to force OpenSearch to use ephemeral storage:
oc patch cm wd-discovery-opensearch-client-config --patch='{"data": {"forceEphemeral": "true"}}' oc patch cm wd-discovery-opensearch-data-config --patch='{"data": {"forceEphemeral": "true"}}' oc patch cm wd-discovery-opensearch-master-config --patch='{"data": {"forceEphemeral": "true"}}'
CR gets stuck after the node reboot
Applies to: 5.2.1
Fixed in: 5.2.2
- Error
-
After rebooting the node, the Watson Discovery CR gets stuck in the
In Progressstate. These pods are not in theRunningstate.wd-discovery-enrichment-flow-669f897ccb-v5jdj 0/1 Init:0/2 0 8m52s wd-discovery-gateway-645794958b-6k62t 0/2 Init:1/2 0 8m21s wd-discovery-ingestion-api-575d75c5b6-zzptp 0/2 Init:1/2 0 7m49s wd-discovery-po-box-86644547f9-nqq7w 0/1 Init:0/1 0 7m18s wd-discovery-training-rest-7599db58d4-zwh82 0/2 Init:1/2 0 5m17s wd-discovery-webhook-connector-6b9db6db79-j7ccr 0/1 Init:1/2 0 4m46sReviewing the RabbitMQ container logs for these pods shows that RabbitMQ is not initialized.
For example:oc logs wd-discovery-training-rest-7599db58d4-zwh82 -n ${PROJECT_CPD_INST_OPERANDS} -c rabbitmq-checkWaiting for RabbitMQ to start. count=1 Waiting for RabbitMQ to start. count=2 Waiting for RabbitMQ to start. count=3 Waiting for RabbitMQ to start. count=4 Waiting for RabbitMQ to start. count=5 … - Cause
- This issue is caused when Watson Discovery is unable to connect to RabbitMQ after a reboot.
- Solution
- Delete the RabbitMQ pod, so that the Watson
Discovery operator can re-create it.
oc delete pod wd-rabbitmq-discovery-0 -n ${PROJECT_CPD_INST_OPERANDS}
CR gets stuck In Progress after the node reboot
Applies to: 5.2.1
- Error
-
After rebooting the node, the Watson Discovery CR gets stuck in the
In Progressstate. Some or all of the pods that begin with the following names are not in theRunningstate.wd-discovery-cnm-api wd-discovery-enrichment-model-copy wd-discovery-entity-suggestion wd-discovery-entity-training wd-discovery-management wd-discovery-ranker-master wd-discovery-ranker-rest wd-discovery-serve-rankerReviewing the logs for the S3-check container in some or all of these pods shows that MCG did not come up properly after the node reboot.Verifying MCG. Unable to list bucket contents Waiting for MCG to start. count=1 Unable to list bucket contents Waiting for MCG to start. count=2 Unable to list bucket contents Waiting for MCG to start. count=3 Unable to list bucket contents Waiting for MCG to start. count=4 - Cause
- This issue is caused when MCG does not come up properly after a node reboot.
- Solution
- Delete the nooba pods so that the nooba operator can re-create
it.
oc delete pod noobaa-core-0 -n openshift-storage oc delete pod noobaa-db-pg-0 -n openshift-storage
Error is displayed after applying the temporary patch
Applies to: 5.2.1 and later
- Error
-
After applying the temporary patch, an error is displayed.
oc get temporarypatches.oppy.ibm.comNAME READY READYREASON UPDATING UPDATINGREASON DEPLOYED VERIFIED AGE temporary-patch False Errored False Errored 1/1 1/1 4sAfter a while, an error is displayed in the status as well.oc get wdNAME VERSION READY READYREASON UPDATING UPDATINGREASON DEPLOYED VERIFIED QUIESCE DATASTOREQUIESCE AGE wd 5.2.1 False ConfigError False Errored 24/24 24/24 NOT_QUIESCED NOT_QUIESCED 11h - Cause
- This issue is caused when the Watson Discovery operator deletes the spec field of the temporary patch.
- Solution
- Apply the same patch again to fill the spec field. Then, delete the Watson Discovery operator to restart
it.
oc delete pod -l icpdsupport/addOnId=discovery,icpdsupport/app=operator -n ${PROJECT_CPD_INST_OPERATORS}
Watson Discovery is not accessible after upgrading
Applies to: 5.2.0 and later
- Error
-
After upgrading to 5.1.0 or later, the Watson Discovery service is healthy, but not accessible from IBM Cloud Pak for Data. This error is likely to occur if the service was originally installed as Version 4.0.x.
- Cause
-
This issue is caused due to a version parameter in the spec.
- Solution
-
- Check the zen extensions to ensure the following are
present:
If these zen extensions are missing, there might be an override in the Watson Discovery CR.oc get zenextensions/discovery-gw-addon zenextensions/wd-discovery-watson-gateway-gw-instance -
Verify the version of Watson Gateway currently in use.
oc get watsongatewaysNAME VERSION READY READYREASON UPDATING UPDATINGREASON AGE wd-discovery-watson-gateway main True Stable False Stable 117d - If the result shows the version as
main, perform the following steps:- Edit the Watson Discovery
CR.
oc edit wd wd - Locate the relevant
versionline in thespec.Gateway: size: small version: main - Remove
version: main, then save and exit. - Monitor the gateway object for changes. This can take several minutes. After it shows as using
the
defaultversion, attempt to access Watson Discovery through IBM Cloud Pak for Data.
The response should appear similar to:oc get watsongatewaysNAME VERSION READY READYREASON UPDATING UPDATINGREASON AGE wd-discovery-watson-gateway default True Stable False Stable 19d
- Edit the Watson Discovery
CR.
- Check the zen extensions to ensure the following are
present:
Custom resources are not accessible from the Teach domain concepts section after upgrading
Applies to: Upgrading from 4.7.1 and 4.7.2 to any later version including 5.2.0
- Error
-
In rare cases, a resource clean-up job might invalidate resources in certain projects when upgrading Watson Discovery. Invalidated resources lead to issues such as dictionaries and entity extractors not being accessible from the Teach domain concepts section of the Improvement tools panel on the Improve and customize page.
- Cause
-
An issue with the resource clean-up job in 4.7.1 and 4.7.2 invalidates the project resources, resulting in this issue.
- Solution
- Scale down the
wd-cnm-api podbefore upgrading Watson Discovery from 4.7.1 and 4.7.2.
After completing the upgrade process, either scale up the pod to its default value or scale the pod to a specific number of replicas instead of the default value.oc -n ${namespace} patch wd wd --type=merge --patch '{"spec": {"cnm": {"apiServer": {"replicas": 0}}}}'To scale up the pod to its default value, run the following command:oc -n ${namespace} patch wd wd --type=merge --patch '{"spec": {"cnm": {"apiServer": {"replicas": 1}}}}' oc -n ${namespace} patch wd wd --type=json --patch '[{"op":"remove","path":"/spec/cnm"}]'To scale the pod to a specific number of replicas, run the following command:oc -n ${namespace} patch wd wd --type=merge --patch '{"spec": {"cnm": {"apiServer": {"replicas": 1}}}}' oc -n ${namespace} patch wd wd --type=merge --patch "{\"spec\": {\"cnm\": {\"apiServer\": {\"replicas\": ${num_of_replicas}}}}}"
During shutdown the DATASTOREQUIESCE field does not update
- Error
- After you run the
cpd-cli manage shutdowncommand, theDATASTOREQUIESCEstate in the Watson Discovery resource is stuck inQUIESCING.Theshutdowncommand completes successfully. However, when you check the status of theWatsonDiscovery wdcustom resource (oc get WatsonDiscovery wd -n "${PROJECT_CPD_INST_OPERANDS}"), the command returns:NAME VERSION READY READYREASON UPDATING UPDATINGREASON DEPLOYED VERIFIED QUIESCE DATASTOREQUIESCE AGE wd 4.7.3 True Stable False Stable 24/24 24/24 QUIESCED QUIESCING 16h
- Cause
- Due to the way quiescing Postgres works, the Postgres pods are still running in background. This results in the metadata not updating in the Watson Discovery resource.
- Solution
- There is no fix for this. However, the state being stuck in
QUIESCINGdoes not affect the Watson Discovery operator.
Upgrade fails due to existing Elasticsearch 6.x indices
Applies to: 5.2.0 and later
- Error
- If the existing Elasticsearch cluster has indices created with Elasticsearch 6.x, then upgrading
Watson
Discovery to Version 5.0.0 and later
fails.
> oc get wd wd NAME VERSION READY READYREASON UPDATING UPDATINGREASON DEPLOYED VERIFIED QUIESCE DATASTOREQUIESCE AGE wd 4.8.0 False InProgress True VerifyWait 2/24 1/24 NOT_QUIESCED NOT_QUIESCED 63m - Cause
- Watson Discovery checks for existence of deprecated version of indices in the Elasticsearch cluster when upgrading to Version 5.0.0 and later.
- Solution
- To determine whether existing Elasticsearch 6.x indices are the cause of the upgrade failure,
verify the log of the
wd-discovery-es-detect-indexpod using the following command:
UpgradeError is shown after resizing PVC
Applies to: 5.2.0 and later
- Error
- After you edit the custom resource to change the size of a persistent volume claim for a data store, an error is shown.
- Cause
- You cannot change the persistent volume claim size of a component by updating the custom resource. Instead, you must change the size of the PVC on the persistent volume claim node after it is created.
- Solution
- To prevent the error, undo the changes that were made to the YAML file. For more information about the steps to follow to change the persistent volume claim size successfully, see Scaling an existing persistent volume claim size.
Disruption of service after upgrading, restarting, or scaling by
updating scaleConfig
Applies to: 5.2.0 and later
- Error
- After upgrading, restarting, or scaling Watson
Discovery by updating
the
scaleConfigparameter, the Elasticsearch component might become non-functional, resulting in disruption of service and data loss. - Cause
- The Elasticsearch component uses a quorum of pods to ensure availability when it completes search operations. However, each pod in the quorum must recognize the same pod as the leader of the quorum. The system can run into issues when more than one leader pod is identified.
- Solution
- To determine if confusion about the quorum leader pod is the cause of the issue, complete the
following steps:
- Log in to the cluster, and then set the namespace to the project where the Discovery resources are installed.
- Check each of the Elasticsearch pod with the role of
masterto see which pod it identifies as the quorum leader.
Each pod must list the same pod as the leader.oc get pod -l icpdsupport/addOnId=discovery,app=elastic,role=master,tenant=wd \ -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | while read i; do echo $i; oc exec $i \ -c elasticsearch -- bash -c 'curl -ksS "localhost:19200/_cat/master?v"'; echo; doneFor example, in the following result, two different leaders are identified. Pods1and2identify pod2as the leader. However, pod0identifies itself as the leader.wd-ibm-elasticsearch-es-server-master-0 id host ip node 7q0kyXJkSJirUMTDPIuOHA 127.0.0.1 127.0.0.1 wd-ibm-elasticsearch-es-server-master-0 wd-ibm-elasticsearch-es-server-master-1 id host ip node L0mqDts7Rh6HiB0aQ4LLtg 127.0.0.1 127.0.0.1 wd-ibm-elasticsearch-es-server-master-2 wd-ibm-elasticsearch-es-server-master-2 id host ip node L0mqDts7Rh6HiB0aQ4LLtg 127.0.0.1 127.0.0.1 wd-ibm-elasticsearch-es-server-master-2
If you find that more than one pod is identified as the leader, contact IBM Support.
Limitations
- Formulas that are embedded as images, especially those containing division bars (horizontal fractions) or other complex notations, are not reliably recognized or extracted by Watson Discovery. As a result, these formulas might be omitted, misinterpreted, or rendered incorrectly in the extracted output. This limitation stems from how the SDU pipeline handles embedded images, and currently affects all versions of Watson Discovery that use SDU.
- The service supports single-zone deployments; it does not support multi-zone deployments.
- You cannot upgrade the Watson
Discovery service by using the
service-instance upgradecommand from the Cloud Pak for Data command-line interface. - You cannot use the Cloud Pak for Data OpenShift® APIs for Data Protection (OADP) backup and restore utility to do an offline backup and restore the Watson Discovery service. Online backup and restore with OADP is available.