Known issues and limitations for watsonx.data integration

1. Replace the Classification operator with an Extract operator, and select all document classes.
2. Remove any additional Extract operators that appear later in the flow.

curl --insecure -X POST "/udp/v1/flows/{flow_id}/deepcopy" -H "Authorization: Bearer <bearer_token>" -H "Content-Type: application/json" -d '{"container_kind": "catalog", "container_id": "<catalog_id>", "target_container_kind": "project", "target_container_id": "<project_id>"}'

1. When designing the flow in a project, create a local parameter or a parameter set for the document library ID.
2. Assign this parameter in the property panel of the document set operator, instead of directly entering the value of the document library ID.
3. Promote the flow to a space when ready.
4. Create the document library in space.
5. When executing the flow in space, pass the document library as a parameter or a parameter set.

• Reading binary data failed from connection (Extract operator)
• No results from runtime (Extract operator)

Node Document set failed and caused aborting the branch execution: Please check if the MinIO bucket associated with the catalog, and the service route has been created or not. 

• Access to the associated metadata store bucket

  Each metadata store (for example, Hive or SQL) is associated with an S3 or Cloud Object Storage bucket. The user executing the operators must have access permissions to this underlying bucket. If the access is not already granted, you must add the required user or group to the bucket using the watsonx Infrastructure Manager Console. Without bucket access, the operators are not able to read or write data to the metadata store.

• Handling the default MinIO bucket (Non-production usage)
  For exploratory or non-production scenarios, watsonx.data includes a default MinIO bucket that is automatically associated with the metadata store. However, this default bucket uses an internal S3 endpoint that is not accessible from external systems such as Unstructured Data Integration. If you plan to use this default MinIO bucket, you must expose the endpoint externally so that it can be accessed by outside systems.

  Note: Creating the edge route exposes the MinIO console externally, allowing external clients to interact with it.
  Follow these steps to expose the MinIO bucket:

  1. Access the MinIO Console.
  2. Create an edge route to expose the MinIO service:

     oc create route edge ibm-lh-lakehouse-minio-console --service=ibm-lh-lakehouse-minio-svc --port=9000

  3. Retrieve the route host for the MinIO service:

     oc get routes ibm-lh-lakehouse-minio-console

     You will now see that the route is port forwarded and is accessible from external systems.
  4. Extract the access and secret keys if needed:

     oc extract secret/ibm-lh-config-secret --to=- --keys=env.properties | grep -E "LH_S3_ACCESS_KEY|LH_S3_SECRET_KEY"

  This step is only required when using the default internal MinIO bucket for testing or non-production purposes. Production-grade metadata stores already use S3 or COS buckets with external endpoints, and do not require port forwarding.

1. If Data Observability is not required:

   oc -n ${PROJECT_CPD_INST_OPERANDS} patch watsonxdataintegration watsonxdataintegration-cr --patch '{"spec":{"enableDataObservability":false}}' --type=merge

2. Delete the current custom resource, if you don’t want to install Data Observability:

   oc -n ${PROJECT_CPD_INST_OPERANDS} delete databandinstaller databand

1. If StreamSets is not required:

   oc -n ${PROJECT_CPD_INST_OPERANDS} patch watsonxdataintegration watsonxdataintegration-cr --patch '{"spec":{" enableRealtimeStreaming ":false}}' --type=merge

2. Delete the current custom resource, if you don’t want to install StreamSets:

   oc -n ${PROJECT_CPD_INST_OPERANDS} delete sdiinstaller streamsets-sdi