Known issues and limitations for Watson Knowledge Catalog

Searching platform connections might not return results

Applies to: 4.7.0 and 4.7.1
Fixed in: 4.7.2

If you search for a connection on the Platform connections page, no results might be returned. Only the displayed connections are searched, although there might be more connections.

Workaround: Click Show more until all connections are loaded and rerun your search.

Can't run lineage imports after upgrading from 4.6.2 to 4.7.0

Applies to: 4.7.0
Fixed in: 4.7.1

After the upgrade from 4.6.2 to 4.7.0, the manta-dataflow pod might be stuck in CrashLoopBackOff state. As a result, you can't run lineage imports.

Workaround: As a user with sufficient permissions, complete the following steps:

1. Generate a bearer token for authentication as described in Creating a CPD bearer token in the Watson Data API documentation.

2. Log in to Red Hat OpenShift Container Platform and start a remote shell for the manta-admin-gui pod:

   oc rsh "$(oc get pods -o name | grep manta-admin-gui)"
   

3. Reset corrupt database files by running the following REST API commands exactly as shown:

   curl -k -v --location --request PUT 'https://manta-configuration-service:8083/private/configuration/v1/unset'
   --header 'Authorization: Bearer '
   --header 'Content-Type: application/json'
   --data-raw '{
   "setBy": "upgrade_h2_workaround",
   "validFrom": "2022-06-30T15:26:35.561+00:00",
   "key": "repository.script-metadata.db",
   "applicationScope": "MANTA_FLOW_SERVER",
   "coponentScope": "REPOSITORY"
   }'
   

   curl -k -v --location --request PUT 'https://manta-configuration-service:8083/private/configuration/v1/unset'
   --header 'Authorization: Bearer '
   --header 'Content-Type: application/json'
   --data-raw '{
   "setBy": "upgrade_h2_workaround",
   "validFrom": "2022-06-30T15:26:35.561+00:00",
   "key": "repository.usage-stats.db",
   "applicationScope": "MANTA_FLOW_SERVER",
   "coponentScope": "REPOSITORY"
   }'
   

   curl -k -v --location --request PUT 'https://manta-configuration-service:8083/private/configuration/v1/unset'
   --header 'Authorization: Bearer '
   --header 'Content-Type: application/json'
   --data-raw '{
   "setBy": "upgrade_h2_workaround",
   "validFrom": "2022-06-30T15:26:35.561+00:00",
   "key": "repository.storage.neo4j-logs",
   "applicationScope": "MANTA_FLOW_SERVER",
   "coponentScope": "REPOSITORY"
   }'
   

   curl -k -v --location --request PUT 'https://manta-configuration-service:8083/private/configuration/v1/unset' \
   --header 'Authorization: Bearer <bearer-token>' \
   --header 'Content-Type: application/json' \
   --data-raw '{
   "setBy": "upgrade_h2_workaround",
   "validFrom": "2022-06-30T15:26:35.561+00:00",
   "key": "repository.storage.neo4j",
   "applicationScope": "MANTA_FLOW_SERVER",
   "coponentScope": "REPOSITORY"
   }'
   

4. Restart the manta-dataflow pod.

After upgrading to 4.7.0, the workflow pod might not start

Applies to: 4.7.0
Fixed in: 4.7.1

After upgrading 4.7.0, the wkc-workflow-service pod does not come up. The liveness probe does not respond, and the pod keeps restarting over and over again. In the pod log, you'll see an exception stack trace. The last Caused by entry in the trace is similar to this one: LockException: Could not acquire change log lock.”`

Workaround: As a user with sufficient permissions, complete the following steps:

1. Log in to Red Hat OpenShift Container Platform. Then, log in to the Db2 pod that runs the workflow database:

   oc rsh c-db2oltp-wkc-db2u-0
   

   sudo su db2inst1
   

   Alternatively, open a bash shell:

   oc exec -it -n ${PROJECT_CPD_INST_OPERANDS} c-db2oltp-wkc-db2u-0 -- bash
   

2. Connect to the workflow database:

   db2
   

   connect to WFDB
   

3. Check the current change log locks:

   select * from FLW_EV_DATABASECHANGELOGLOCK;
   

   This command should return 1 record at maximum.

4. If there is an entry, clear the lock:

   delete from FLW_EV_DATABASECHANGELOGLOCK;
   

   COMMIT;
   

5. Close the connection to Db2 and restart the workflow service pod, for example, by deleting it.

When installing or upgrading to 4.7.0, the wdp-profiling-iae-init job might fail

Applies to: 4.7.0
Fixed in: 4.7.1

When installing or upgrading Watson Knowledge Catalog to version 4.7.0 on separation-of-duties (SOD) configured environments, the wdp-profiling-iae-init job might fail.

The following is an example of the error:

wdp-profiling-iae-init-2j7sz                                      0/1     Error       0

And here is an example of the pod log output when this happens:

oc logs -f wdp-profiling-iae-init-2j7sz
storage_class_name: managed-nfs-storage
wkc_tenant_namespace: tc
wkc_dataplane_namepsace: dp
wkc_cpd_instance_namespace: cpd-instance
Tenant, dataplace and cpd_instance namespaces are not empty/equal and SoD mode is enabled. Override volume_name_with_namespace value
Final volume_name_with_namespace value is set to dp::ProfStgIntrnl
[19-June-2023 09:14 UTC] INFO: usermgmt-svc get profile_user response: {"uid":"1000331002","username":"__internal_profiler__","displayName":"Profile manager","email":"profile.manager@test.com","approval_status":"approved","permissions":["sign_in_only","create_project","create_space","author_governance_artifacts","view_governance_artifacts","can_provision","access_catalog","manage_information_assets","manage_discovery"],"user_roles":["User","Data Engineer","Developer"],"current_account_status":"enabled","internal_user":true,"deletable":true,"authenticator":"default","created_timestamp":1686901467959,"last_modified_timestamp":1686901468183,"misc":{"sessionInfo":[],"last_session_ended_timestamp":"","last_session_start_timestamp":"","dark_mode":false},"role":"User","profile_picture":"","is_custom_picture":false,"groups":[{"group_id":10000,"name":"All users","description":"All users are implicitly part of this group","created_by":"","created_at":"","updated_at":"","misc":{},"members_count":"1"}],"group_roles":[]}
[19-June-2023 09:14 UTC] INFO: uid: 1000331002
[19-June-2023 09:14 UTC] INFO: access_token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImlPUGZxR2ptSFV5bEFDRXpJcVl2S3lEZG5hWmsyRVhzR1NVR0VkcWJjcXciLCJ0eXAiOiJKV1QifQ.eyJhdWQiOiJEU1giLCJleHAiOjE2ODcxNjYzOTIsImlhdCI6MTY4NzE2NjA5MiwiaXNzIjoiS05PWFNTTyIsInN1YiI6Il9faW50ZXJuYWxfcHJvZmlsZXJfXyIsInVzZXJuYW1lIjoiX19pbnRlcm5hbF9wcm9maWxlcl9fIiwidWlkIjoiMTAwMDMzMTAwMiIsImRpc3BsYXlfbmFtZSI6IkNvcmUgQVBJIEludGVybmFsIFVzZXIiLCJyb2xlIjoiQWRtaW4iLCJwZXJtaXNzaW9ucyI6WyJtYW5hZ2VfY2F0ZWdvcmllcyIsImNvbmZpZ3VyZV9wbGF0Zm9ybSIsIm1hbmFnZV9wcm9qZWN0IiwibWFuYWdlX2NhdGFsb2ciLCJtb25pdG9yX3BsYXRmb3JtIiwiYWNjZXNzX2NhdGFsb2ciLCJzaGFyZV9zZWNyZXRzIiwiZ2xvc3NhcnlfYWRtaW4iLCJjb25maWd1cmVfYXV0aCIsImFkZF92YXVsdHMiLCJ2aWV3X2dvdmVybmFuY2VfYXJ0aWZhY3RzIiwibWFuYWdlX3NwYWNlIiwibWFuYWdlX2dsb3NzYXJ5IiwiYWRtaW5pc3RyYXRvciIsImNyZWF0ZV9wcm9qZWN0Iiwidmlld19wbGF0Zm9ybV9oZWFsdGgiLCJtYW5hZ2VfdmF1bHRzX2FuZF9zZWNyZXRzIiwibWFuYWdlX3JlcG9ydGluZyIsIm1vbml0b3JfcHJvamVjdCIsImNyZWF0ZV9zcGFjZSIsImF1dGhvcl9nb3Zlcm5hbmNlX2FydGlmYWN0cyIsImV2YWx1YXRlX3BvbGljeV9kZWNpc2lvbiIsIm1vbml0b3Jfc3BhY2UiLCJzaWduX2luX29ubHkiLCJtYW5hZ2VfZGlzY292ZXJ5IiwibWFuYWdlX3NlcnZpY2VfaW5zdGFuY2VzIiwibWFuYWdlX2dvdmVybmFuY2Vfd29ya2Zsb3ciLCJtYW5hZ2VfZ3JvdXBzIiwiY2FuX3Byb3Zpc2lvbiIsIm1hbmFnZV91c2VycyJdLCJncm91cHMiOlsxMDAwMF0sImF1dGhlbnRpY2F0b3IiOiJkZWZhdWx0In0.TKnnFgmc4-7UNUXAZ7O78LNQw3Rj7Y8ZRSEt4RAyag6f-RfUf4kuwgb8UDkg-7pQzsSt7NjfcKCvyEtRvCmA6bs0vdMP3j1nIQplnANOD9YGvTdUx8aSLNnen1Ur-ojWSjd5cZtKKqf3TPTbES69dI95lsUzGm-5prqa3INsjXLEd4jEmKF-sJnfntW1s-JOvgTaul-Sw_Nzg9QjqFggVRuDi7TFqHKJj76lM_mzXEpKqgGfywBfqOpSu3FjgDGxp0KJ6QCEVZzcmbIKYr52YqRRWa14h_xbCEHymvt09S2DUHZ_MX7qGewfqrhbylCpW71I3cswc3UNm2FEdZrDmg
[19-June-2023 09:14 UTC] INFO: response: {"apiKey":"s2hTDI0vYGbsI0YYvzmCMRenIWmkztbK37HEQrBn","_messageCode_":"success","message":"success"}
[19-June-2023 09:14 UTC] INFO: _messageCode_: success
[19-June-2023 09:14 UTC] INFO: usermgmt_apikey: s2hTDI0vYGbsI0YYvzmCMRenIWmkztbK37HEQrBn
[19-June-2023 09:14 UTC] INFO: auth_response: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImlPUGZxR2ptSFV5bEFDRXpJcVl2S3lEZG5hWmsyRVhzR1NVR0VkcWJjcXcifQ.eyJ1c2VybmFtZSI6Il9faW50ZXJuYWxfcHJvZmlsZXJfXyIsInJvbGUiOiJVc2VyIiwicGVybWlzc2lvbnMiOlsic2lnbl9pbl9vbmx5IiwiY3JlYXRlX3Byb2plY3QiLCJjcmVhdGVfc3BhY2UiLCJhdXRob3JfZ292ZXJuYW5jZV9hcnRpZmFjdHMiLCJ2aWV3X2dvdmVybmFuY2VfYXJ0aWZhY3RzIiwiY2FuX3Byb3Zpc2lvbiIsImFjY2Vzc19jYXRhbG9nIiwibWFuYWdlX2luZm9ybWF0aW9uX2Fzc2V0cyIsIm1hbmFnZV9kaXNjb3ZlcnkiXSwiZ3JvdXBzIjpbMTAwMDBdLCJzdWIiOiJfX2ludGVybmFsX3Byb2ZpbGVyX18iLCJpc3MiOiJLTk9YU1NPIiwiYXVkIjoiRFNYIiwidWlkIjoiMTAwMDMzMTAwMiIsImF1dGhlbnRpY2F0b3IiOiJkZWZhdWx0IiwiZGlzcGxheV9uYW1lIjoiUHJvZmlsZSBtYW5hZ2VyIiwiYXBpX3JlcXVlc3QiOmZhbHNlLCJpYXQiOjE2ODcxNjYwOTIsImV4cCI6MTY4NzIwOTI1Nn0.0Y6uPW1JaTgoyaF92Fcq5vpPbaxCevSrmTcrjQ6qIMhK2THIVrVH3T9uw6SpzvuV8cz3WZWeSChf1fCerboWygQNQR8M8Te4wN8EdmdlmoEujo4ZFfheE4wxi15Q4OJI4AJnu8Hu7Q5HWovxECr8ScA6Q38CciAzZL2q91yr32vlZzYytU-hWOWZpM3nbtwl2lkYUCMXuLQlDcwkRwk4IMGKBrGkcFDgO9W-YO7JmfaPT3eojKGx6gqED_9_Vintbx8ZOtDihdzDmGeu3TeRGNc4JSQg9EaKT8PlFtN9KEETx-cp8tJ71GF4V1XB-sG4XHM7FsKX1SUn-gp6KfBzhA
[19-June-2023 09:14 UTC] INFO: wdp_profiling_gateway_url: internal-nginx-svc.cpd-instance.svc:12443
[19-June-2023 09:14 UTC] INFO: inst_display_name: dp::ProfStgIntrnl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    63  100    63    0     0   1750      0 --:--:-- --:--:-- --:--:--  1750
[19-June-2023 09:14 UTC] INFO: service instance: {"limit":20,"offset":0,"service_instances":[],"total_count":0}
[19-June-2023 09:14 UTC] INFO: service_instance_count: 0
[19-June-2023 09:14 UTC] INFO: inst_display_name: dp::ProfStgIntrnl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    63  100    63    0     0   1702      0 --:--:-- --:--:-- --:--:--  1702
[19-June-2023 09:14 UTC] INFO: service instance: {"limit":20,"offset":0,"service_instances":[],"total_count":0}
[19-June-2023 09:14 UTC] INFO: service_instance_count: 0
[19-June-2023 09:14 UTC] INFO: Retry....1
[19-June-2023 09:15 UTC] INFO: inst_display_name: dp::ProfStgIntrnl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    63  100    63    0     0   1852      0 --:--:-- --:--:-- --:--:--  1852
[19-June-2023 09:15 UTC] INFO: service instance: {"limit":20,"offset":0,"service_instances":[],"total_count":0}
[19-June-2023 09:15 UTC] INFO: service_instance_count: 0
[19-June-2023 09:15 UTC] INFO: Retry....2
[19-June-2023 09:15 UTC] INFO: volume instance dp::ProfStgIntrnl does not exist!!!
[19-June-2023 09:15 UTC] INFO: vol_instance_exist: false
[19-June-2023 09:15 UTC] INFO: Creating volume instance....
[19-June-2023 09:15 UTC] INFO: Function createVolumeInstance()...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   649  100   160  100   489   2162   6608 --:--:-- --:--:-- --:--:--  8770
[19-June-2023 09:15 UTC] INFO: Volume creation response:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
KeyError: 'id'

Workaround: Run the following commands to fix this issue:

1. Run the following to open the job:

   oc debug job/wdp-profiling-iae-init --as-root -n <replace_wkc_namespace>
   

2. Run the following to access the debugging tools:

   source /debug/debug_tools.sh
   

3. Run the following to generate temporary keys:

   cp /wkc/genkeys.sh /tmp/
   

4. Run the following to access the the temporary generated keys:

   chmod 755 /tmp/genkeys.sh
   

5. Run the following to open the temporary keys:

   vi /tmp/genkeys.sh
   

6. Search for the string namespace. You will find 2 entries in the createVolumeInstance() and createInstance() functions as part of CURL JSON fields.

7. Replace the namespace field value with the value wkc_dataplane_namepsace: in the entries you searched previously.

8. Save the file and exit.

9. Run the following shell script:

    /tmp/genkeys.sh
   

10. Make sure that the script runs successfully and it should print a success message. For example:

       SUCCESS
       TVJlIoaZlfZ2VfvmNNhktvxhyaH9WiA0y5RWPHOJ
      1687319111557130
    

11. Exit from this message and you should see a new pod status message. For example:

    oc get po -n <replace_with_wkc_namespace> | grep iae
    wdp-profiling-iae-init-9tgpd                                      0/1     Completed   0
    
    

Upgrading to version 4.7 might fail because of the Db2u container

Applies to: Upgrades to 4.7.0 and later

When upgrading to version 4.7, the upgrade might fail because the DB2COMM value inside the c-db2oltp-wkc-db2u-0 pod or the c-db2oltp-iis-db2u-0 pod is set to NULL.

The Watson Knowledge Catalog wkc-post-upgrade-refresh-users job fails when the common core service (ccs) and Watson Knowledge Catalog operators reconcile in 4.7

Applies to: 4.7.0 and later

You might encounter this issue during the wkc-post-upgrade-refresh-users job if the access advanced mapping permission exists in certain roles. This permission might cause upgrade failures and should be removed before the upgrade.

Before upgrading

Update Roles through the user interface (UI):

1. Access the console UI and expand the left top corner menu, then click on Administrator > Access control
2. Choose the Roles tab
3. The role with the permission is shown
4. Access the permissions from the side menu and highlight the Access advanced mapping permission
5. Click on the Remove button and then confirm the removal

This has to be done for all the roles that have the Access advanced mapping permission.

Failure after the upgrade:

If you encounter the failure after the upgrade, you will need to follow the workaround to re-introduce the Access advanced mapping permission.

Workaround: Apply the missing Access advanced mapping permission to prevent ccs or Watson Knowledge Catalog reconciliation failures by following these steps:

1. Apply the following configmap in the cpd-instance namespace:

   cat <<EOF | oc apply -f -
   apiVersion: v1
   data:
     extensions: |
       [
         {
           "extension_point_id": "zen_permissions",
           "extension_name": "access_advanced_mapping_capabilities",
           "display_name": "{{.global_wkc_access_advanced_mappings}}",
           "match_permissions": "",
           "match_instance_id": "",
           "match_instance_role": "",
           "meta": {},
           "details": {
             "key": "access_advanced_mapping_capabilities",
             "category": "{{.global_wkc_category_catalogs}}",
             "category_description": [
               "{{.global_wkc_category_catalogs_description1}}",
               "{{.global_wkc_category_catalogs_description2}}"
             ],
             "description": [
               "{{.global_wkc_access_advanced_mappings_description}}"
             ],
             "actions": [
               { "description": "{{.global_wkc_action_extension_mappings}}", "tooltip": "{{.global_wkc_extension_mappings_tooltip}}" },
               { "description": "{{.global_wkc_action_import_mappings}}" },
               { "description": "{{.global_wkc_action_import_data_sources}}" }
             ]
           }
         }
       ]
   kind: ConfigMap
   metadata:
     labels:
       app: wkc-lite
       app.kubernetes.io/instance: 0075-wkc-lite
       app.kubernetes.io/managed-by: Tiller
       app.kubernetes.io/name: wkc-lite
       chart: wkc-lite
       helm.sh/chart: wkc-lite
       heritage: Tiller
       icpdata_addon: "true"
       icpdata_addon_version: 4.7.3
       release: 0075-wkc-lite
     name: wkc-permission-extensions-tmp
     namespace: ${PROJECT_CPD_INSTANCE}
   EOF
   
   

2. If the wkc-base-roles-init job fails , force a reconciliation of the ccs operator by running the following:

   oc delete pods -l app.kubernetes.io/name=ibm-cpd-ccs-operator -n ${PROJECT_CPD_OPS}
   

3. If wkc-post-upgrade-refresh-users job fails, force a reconciliation of the Watson Knowledge Catalog operator by running the following:

   oc delete pods -l app.kubernetes.io/name=ibm-cpd-wkc-operator -n ${PROJECT_CPD_OPS}
   

4. Once the operator reconciliation are completed then go to the UI and remove the Access advanced mapping permission from all the roles that has it assigned. Follow the steps outlined in the previous section about the UI.

5. Delete the configmap that you created in the cpd-instance namespace.

6. If you still encounter issues because of other invalid permissions, create a Python script and call it cleanup.py under the /tmp/ folder inside of the failed job related pod, which will either be wkc-base-roles-init or wkc-post-upgrade-refresh-users.

   If the wkc-base-roles-init job fails, run the following:

    oc debug job/wkc-base-roles-init --as-user=122323
   

   If the wkc-post-upgrade-refresh-users job fails, run the following:

    oc debug job/wkc-post-upgrade-refresh-users --as-user=122323
   

   Then, copy the following content for the Python script:

   import json
   import os
   import requests
   import sys
   import argparse
   import urllib3
   urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
   #include extension_name(s) of all the roles that have been modified in your charts for the current release
   #below is an example where I want to refresh the Data Steward and the User roles. This is a list so you can include multiple role extension names.
   service_broker_secret = None
   cpd_namespace = os.environ.get('CPD_NAMESPACE')
   verify_cert = '/etc/wdp_certs/internal-nginx.cert.pem'
   zen_core_api_svc = f'https://zen-core-api-svc.{cpd_namespace}.svc.cluster.local:4444'
   usermgmt_svc = f'https://usermgmt-svc.{cpd_namespace}.svc.cluster.local:3443'
   
   #update the below environment variable to match how you mount the service broker secret in your chart .yaml file
   if os.environ.get('ZEN_SERVICE_BROKER_SECRET') is not None:
       service_broker_secret = os.environ.get('ZEN_SERVICE_BROKER_SECRET')
   elif os.path.isfile('/etc/.secrets/ZEN_SERVICE_BROKER_SECRET'):
       with open('/etc/.secrets/ZEN_SERVICE_BROKER_SECRET') as file:
           service_broker_secret = file.read().strip()
   else:
       sys.exit('could not find zen-service-broker-secret environment variable')
   def get_internal_service_token():
       print('\n>> Requesting internal-service-token from zen-core-api-svc...')
       url = f'{zen_core_api_svc}/internal/v1/service_token'
       headers = {
           'secret': service_broker_secret
       }
       r = requests.get(url, headers=headers, verify=False)
       if r.status_code != 200:
           print('>> Error requesting internal_service_token - status_code - ' + str(r.status_code))
           try:
               print(json.dumps(r.json(), indent=2))
           except Exception:
               print(r.text)
           sys.exit('request to retrieve internal_service_token failed')
       else:
           print('>> Successfully requested internal_service_token - status_code - 200')
           try:
               resp = r.json()
               if resp['token']:
                   return 'Bearer ' + resp['token']
               else:
                   sys.exit('could not parse internal_service_token from the response')
           except Exception:
               sys.exit('could not parse internal_service_token from the response')
   def get_platform_roles(token):
       print('\n>> Requesting platform-roles from usermgmt-svc...')
       url = f'{usermgmt_svc}/v1/roles'
       headers = {
           'Authorization': token
       }
       r = requests.get(url, headers=headers, verify=False)
       if r.status_code != 200:
           print('>> Error retrieving all platform-roles - status_code - ' + str(r.status_code))
           try:
               print(json.dumps(r.json(), indent=2))
           except Exception:
               print(r.text)
           sys.exit('request to retrieve roles failed')
       else:
           print('>> Successfully retrieved all platform-roles - status_code - 200')
           try:
               resp = r.json()
               if resp['rows']:
                   print(json.dumps(resp['rows'], indent=2))
                   return resp['rows']
               else:
                   sys.exit('could not parse roles from the response')
           except Exception:
               sys.exit('could not parse roles from the response')
   def get_permissions(token):
       print('\n>> Getting permissions...')
       url = f'{zen_core_api_svc}/openapi/v1/permissions'
       headers = {
           'Authorization': token
       }
       r = requests.get(url, headers=headers, verify=False)
       if r.status_code != 200:
           print('>> Error retrieving permissions - status_code - ' + str(r.status_code))
           try:
               print(json.dumps(r.json(), indent=2))
           except Exception:
               print(r.text)
           sys.exit('request to retrieve permissions failed')
       else:
           print('>> Successfully retrieved permissions - status_code - 200')
           try:
               resp = r.json()
               if resp['Permissions']:
                   #print(json.dumps(resp['Permissions'], indent=2))
                   return resp['Permissions']
               else:
                   sys.exit('could not parse permissions from the response')
           except Exception:
               sys.exit('could not parse permissions from the response')
   def refresh_user_permissions(token, roles_list, perm_list):
       print('\n>> Verifying permissions of roles from usermgmt-svc...')
       flagger = False
       for role_obj in roles_list:
           if role_obj and type(role_obj) is dict and \
               role_obj['id'] is not None and \
               role_obj['doc'] and type(role_obj) is dict and \
               role_obj['doc']['role_name'] is not None and \
               role_obj['doc']['permissions'] and type(role_obj['doc']['permissions']) is list and len(role_obj['doc']['permissions']) > 0:
               print('>> Verifying permissions for role - ' + role_obj['id'])
               url = f'{usermgmt_svc}/v1/role/' + role_obj['id']
               headers = {
                   'Authorization': token,
                   'Content-Type': 'application/json'
               }
               print('>> Current permissions for role - ' + role_obj['id'])
               print(json.dumps(role_obj['doc']['permissions'], indent=2))
               role_permissions = role_obj['doc']['permissions']
               valid_permissions = list(filter(lambda p: (p in perm_list),role_permissions))
               invalid_permissions = list(filter(lambda p: not(p in perm_list), role_permissions))
               payload_data = {
                   'role_name': role_obj['doc']['role_name'],
                   'description': role_obj['doc']['description'] or '',
                   'permissions': valid_permissions
               }
               r = requests.put(url, headers=headers, data=json.dumps(payload_data), verify=False)
               if r.status_code != 200:
                   print('>> Error refreshing role with extension_name - ' + role_obj['id'] + ' - status_code - ' + str(r.status_code))
                   try:
                       print(json.dumps(r.json(), indent=2))
                   except Exception:
                       print(r.text)
                   flagger = True
               else:
                   if len(invalid_permissions) == 0:
                       print('>> Nothing to purge for role with extension_name - ' + role_obj['id'])
                   else:
                       print('>> Successfully purged invalid permissions from role with extension_name - ' + role_obj['id'] + ' - status_code - 200')
                       print('>> Purged permissions:')
                       print(json.dumps(invalid_permissions, indent=2))
           else:
               continue
       if flagger == True:
           sys.exit('some roles were not refreshed - exit and retry')
       return
   def main():
       bearer_token = get_internal_service_token()
       role_extensions = get_platform_roles(bearer_token)
       permissions_list = get_permissions(bearer_token)
       refresh_user_permissions(bearer_token, role_extensions, permissions_list)
   if __name__ == '__main__':
       print("========== Refresh user-profiles with roles that contain modified permissions ==========")
       main()
   
   

   If vi does not work, you can run:

   /debug/debug_tools.sh
   

   which will enable vi.

   Another option is using the cat command:

   cat <> /tmp/cleanup.py
   

7. Set the following environment variable before running the python script:

   export ZEN_SERVICE_BROKER_SECRET=cat /etc/.secrets/ZEN_SERVICE_BROKER_SECRET
   

   Run the python script from the failed job related pod:

   python3 /tmp/cleanup.py --uninstall
   

After upgrading to 4.7.1 or later, no assets might be imported when you run a lineage import

Applies to: Upgrades to 4.7.1 and later

After upgrading to version 4.7.1 or later, lineage imports might fail because the trust store in MANTA Automated Data Lineage is corrupted.

Workaround: To fix the issue, complete these steps:

1. Open the administration GUI for MANTA Automated Data Lineage. Enter the following URL in a web browser replacing hostname with your Cloud Pak for Data cluster URL:

   https://<hostname>/manta-admin-gui/
   

2. From the menu bar, select Configuration.

3. From the menu, select Common > Common Config and click Edit.

4. In the SSL section, click Edit next to MANTA Flow CLI System Connectors Settings.

5. In the Edit truststore settings window, click Recreate.

6. For the Use generated password option, select false.

7. In the Password and Confirm Password fields, enter the default password mantaConnectorsTruststore and click Confirm.

8. Save the new configuration.

During installs of 4.7.1 the wkc/init.sh might fail

Applies to: 4.7.1
Fixed in: 4.7.2

During new installs in some clusters, there can be a timing issue that can cause the SCC to be created before the global parameter from the configmap is picked up.

Symptoms:

1. The oc get cm db2u-product-cm -n{CPD_OPS_NAMESPACE} -oyaml | grep DB2 command shows DB2U_RUN_WITH_LIMITED_PRIVS: "false"
2. The oc get db2ucluster db2oltp-wkc -n{CPD_INSTANCE_NAMESPACE} -oyaml | grep privileged command shows privileged: true
3. The oc get scc cpd-instance-c-db2oltp-wkc-scc -n{CPD_INSTANCE_NAMESPACE -oyaml | grep Privilege command shows allowPrivilegeEscalation: false and allowPrivilegedContainer: false

Workaround:

1. Delete the Watson Knowledge Catalog custom resource (CR) by running:

   ./cpd-cli manage delete-cr --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} --components=wkc -vvv
   

2. Create the Watson Knowledge Catalog CR to override the default setting of setKernalparameters. On your client workstation, create a file called install-options.yml in the cpd-cli-workspace/olm-utils-workspace/work directory with the following content:

   custom_spec:
     wkc:
       wkc_db2u_set_kernel_params: True
       iis_db2u_set_kernel_params: True
   
   

   If you already used a install-options.yml file for the Watson Knowledge Catalog installation before, then add wkc_db2u_set_kernel_params: True and iis_db2u_set_kernel_params: True to the YAML file.

3. Then run the apply-cr command to install Watson Knowledge Catalog and specify --param-file=/tmp/work/install-options.yml. The apply-cr command should be:

   cpd-cli manage apply-cr \
   --release=${VERSION} \
   --cpd_operator_ns=${PROJECT_CPD_INST_OPERATORS} \
   --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
   --components=wkc \
   --block_storage_class=${STG_CLASS_BLOCK} \
   --file_storage_class=${STG_CLASS_FILE} \
   --license_acceptance=true \
   --param-file=/tmp/work/install-options.yml
   

After upgrading to version 4.7.3, the data quality page and data quality SLA rules might not be usable

Applies to: 4.7.3

When you upgrade to Cloud Pak for Data 4.7.3, it can happen that a minor database schema change fails due to a delayed database start. As a result, the data quality SLA rules feature might not work properly, and error messages and notifications similar to the following ones might be displayed:

The activation status could not be loaded

Failed to load SLA activity status

Workaround: Restart the wkc-data-rules pod. To do so, you can scale the pod to zero and then back to its previous value:

oc scale deploy wkc-data-rules --replicas=0 
oc scale deploy wkc-data-rules --replicas=N  

When uninstalling, terminating PVCs might get stuck

Applies to: 4.7.3

During the uninstall of Watson Knowledge Catalog, the PVCs c-db2oltp-wkc-meta and c-db2oltp-iis-meta might get stuck in the Terminating state.

When inspecting the PVCs, it might show which pods are still using it, which prevents the PVC from being deleted. For example:

Used By:       c-db2oltp-wkc-11.5.8.0-cn1-to-11.5.8.0-cn5-f6d4j
               c-db2oltp-wkc-11.5.8.0-cn1-to-11.5.8.0-cn5-qhwzm
               db2u-ssl-rotate-db2oltp-wkc-k78kg

In the example, the c-db2oltp-wkc-meta PVC is still being used.

Workaround: To ensure the PVC is properly deleted, the completed pods and jobs that are still mounting the PVC must be manually deleted.

Follow these steps to manaully delete the PVCs:

1. Delete completed jobs db2u-ssl-rotate-db2oltp-iis and db2u-ssl-rotate-db2oltp-wkc if they exist:

   oc delete job db2u-ssl-rotate-db2oltp-iis -n ${PROJECT_CPD_INST_OPERANDS} --ignore-not-found
   oc delete job db2u-ssl-rotate-db2oltp-wkc -n ${PROJECT_CPD_INST_OPERANDS} --ignore-not-found
   

2. Delete the completed upgrade pods, if they exist:

   oc delete po c-db2oltp-wkc-11.5.8.0-cn1-to-11.5.8.0-cn5-f6d4j -n ${PROJECT_CPD_INST_OPERANDS}
   oc delete po c-db2oltp-wkc-11.5.8.0-cn1-to-11.5.8.0-cn5-qhwzm -n ${PROJECT_CPD_INST_OPERANDS}
   

When upgrading Watson Knowledge Catalog to version 4.7.3, the upgrade might get stuck

Applies to: 4.7.3

During the upgrade of Watson Knowledge Catalog to version 4.7.3, the NodePort services are removed. This puts the whole upgrade process in a NotReady state and the upgrade becomes stuck.

Workaround: Contact IBM support and open a case.

When the Analytics Engine is upgraded as part of Watson Knowledge Catalog, the upgrade might fail

Applies to: 4.7.0 and 4.7.1
Fixed in: 4.7.2

When users are upgrading from Cloud Pak for Data 4.5.x or 4.6.x to 4.7.0 and 4.7.1, upgrading the Analytics Engine as part of the Watson Knowledge Catalog upgrade might fail on clusters.

This is because the blockStorageClass parameter is not getting passed as expected from the wkc operator.

Workaround:

1. Choose the blockStorageClass parameter from the zen service and update the ae custom resource (cr).

2. Edit the analyticsengine cr and add the blockStorageClass parameter in the spec section of the cr as follows:

   oc edit ae analyticsengine-sample -n
   

3. Update the analyticsengine cr. Here is an example of an updated cr:

   oc get ae analyticsengine-sample -n cpd-instance -o yaml
   
   
    apiVersion: v1
    items:
    - apiVersion: ae.cpd.ibm.com/v1
      kind: AnalyticsEngine
      metadata:
        creationTimestamp: "2023-12-05T22:05:51Z"
        finalizers:
        - ae.cpd.ibm.com/finalizer
        generation: 2
        labels:
          app.kubernetes.io/instance: ibm-cpd-ae-operator
          app.kubernetes.io/managed-by: ibm-cpd-ae-operator
          app.kubernetes.io/name: ibm-cpd-ae-operator
        name: analyticsengine-sample
        namespace: zen
        resourceVersion: "469177"
        uid: 38b31279-a280-4266-ade2-68de81672fe9
      spec:
        blockStorageClass: ocs-storagecluster-ceph-rbd
        cloudpakfordata: true
        license:
          accept: true
   

Resource ID invalid for connected data assets from certain connections

Applies to: 4.7.0 and 4.7.1
Fixed in: 4.7.2

If you import a connected data asset into a catalog that has data from one of these connections, the Resource ID will be rendered as ||/<file_path>.

Affected connections:

• Amazon S3
• Generic S3
• MinIO

Workaround:
Edit the connection. Use the full URL with the https:// prefix for the Endpoint URL.

Missing previews

Applies to: 4.7.0 and later

You might not see previews of assets in these circumstances:

• In a catalog or project, you might not see previews or profiles of connected data assets that are associated with connections that require personal credentials. You are prompted to enter your personal credentials to start the preview or profiling of the connection asset.
• In a catalog, you might not see previews of JSON, text, or image files that were published from a project.
• In a catalog, the previews of JSON and text files that are accessed through a connection might not be formatted correctly.
• In a project, you cannot view the preview of image files that are accessed through a connection.

Missing default catalog and predefined data classes

Applies to: 4.7.0 and later

The automatic creation of the default catalog after installation of the Watson Knowledge Catalog service can fail. If it does, the predefined data classes are not automatically loaded and published as governance artifacts.

Workaround: Ask someone with the Administrator role to follow the instructions for creating the default catalog manually.

Publishing a Cognos Dashboard with more than one image creates multiple attachments with the same image data

Applies to: 4.7.0 and later

When you publish a Cognos dashboard to a catalog and choose to add more than one preview of the dashboard, all attachments in the catalog show the image added last. This issue occurs when you select multiple images at a time and drag them into the publish page and when you add files one at a time.

In addition, when you browse for files in the publishing step, you can select only one file. To add further images, you must drag them to the publish page.

Whenever you publish the same dashboard again, it will have the images from the previously published assets as well as the newly added images. For example, if you publish dashboard A with images 1, 2 and 3, it will have 3 screen captures of image 3. If you publish dashboard A again with images 4, 5, 6, it will have 5 screen captures, 3 with image 3 and 2 with image 6.

Catalog UI does not update when changes are made to the asset metadata

Applies to: 4.7.0 and later

If the Catalog UI is open in a browser while an update is made to the asset metadata, the Catalog UI page will not automatically update to reflect this change. Outdated information will continue to be displayed, causing external processes to produce incorrect information.

Workaround: After the asset metadata is updated, refresh the Catalog UI page at the browser level.

A blank page might be rendered when you search for terms while manually assigning terms to a catalog asset

Applies to: 4.7.0 and later

When you search for a term to assign to a catalog asset and change that term while the search is running, it can happen that a blank page is shown instead of any search results.

Workaround: Rerun the search.

Cannot delete a custom relationship definition for catalog assets

After adding a custom relationship of a type to a catalog, you will not be able to delete it on Asset and artifacts definitions page.

Workaround: To delete a custom relationship definition, you need to delete all other existing relationships of that type first.

Applies to: 4.7.0 and later

Special or double-byte characters in the data asset name are truncated on download

Applies to: 4.7.0 and later

When you download a data asset with a name that contains special or double-byte characters from a catalog, these characters might be truncated from the name. For example, a data asset named special chars!&@$()テニス.csv will be downloaded as specialchars!().csv.

The following character sets are supported:

• Alphanumeric characters: 0-9, a-z, A-Z
• Special characters: ! - _ . * ' ( )

Canceling import job does not prevent processing

Applies to: 4.7.0 and later

Canceling an import job using cpd-cli will not stop the job from being processed.

Workaround: Restart the portal-job-manager pod using the command: kubectl rollout restart deployment portal-job-manager.

Unauthorized users might have access to profiling results

Applies to: 4.7.0 and later

Users who are collaborators with any role in a project or a catalog can view an asset profile even if they don't have access to that asset at the data source level or in Watson Query.

Workaround: Before you add users as collaborators to a project or a catalog, make sure they are authorized to access the assets in the container and thus to view the asset profiles.

Duplicate columns in files will not be displayed in the columns table

Applies to: 4.7.0 and later

If a CSV or other structured file type contains duplicate columns with the same name, only the first instance of each column will be displayed in the columns table on the asset Overview page.

Duplicate action fails when IP address changes

Applies to: 4.7.0 and later

Duplicate actions can fail during connection creation if the connection is using a hostname with a dynamic IP address.

Catalog and Global Search bulk jobs fail when unable to connect

Applies to: 4.7.0
Fixed in: 4.7.1

Bulk jobs for wkc-search-reindexing-cronjob are configured to connect to Elasticsearch on port 9300 by default instead of port 443, causing the bulk job to fail.

Workaround: To configure the connection to Elasticsearch on port 443 complete the following steps:

1. Run oc patch ccs ccs-cr --type merge --patch '{"spec": {"ignoreForMaintenance": true}}' -n <namespace>. to put the CCS operator in maintenance mode.This will prevent any changes from being overwritten.
2. Update wkc-search-reindexing-cronjob cronjob to set the correct port number using the following command oc set env cronjob wkc-search-reindexing-cronjob elasticsearch_uri_port_config=443 -n <namespace>
3. Run the job as usual.
4. To take the CCS operator out of maintenance mode, run oc patch ccs ccs-cr --type merge --patch '{"spec": {"ignoreForMaintenance": false}}' -n <namespace>.

Cannot run export operations on a container package exported from another Cloud Pak for Data cluster

Applies to: 4.7.0

When importing a container package exported from another Cloud Pak for Data cluster, permissions must be configured on the archive to allow export operations on the target cluster to access the files within the archive.

Workaround: To extract the export archive and modify permissions, complete the following steps:

1. Create a temporary directory by running mkdir temp_directory
2. Extract the archive by running tar -xvf cpd-exports-<export_name>-<timestamp>-data.tar --directory temp_directory.
3. Clients will need to run the following command on the target cluster: oc get ns $CLUSTER_CPD_NAMESPACE -o=jsonpath='{@.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}'. Example output: 1000700000/10000.
4. The first part of the output of the previous step (ex. 1000700000) will need be applied as the new ownership on all files within the archive. Example: cd temp_directory/chown -R 1000700000:1000700000 <export_name>
5. Archive the fixed files with the directory, using the same export name and timestamp as the original exported tar: tar -cvf cpd-exports-<export_name>-<timestamp>-data.tar <export_name>/.
6. Upload the archive.

Cannot add duplicates when publishing assets to the default catalog

Applies to: 4.7.0 and later

When the default catalog doesn’t have a set configuration for duplicate asset handling, Add duplicate won’t appear as an option when publishing an asset to the catalog.

Workaround: To allow for publishing assets to the default catalog as duplicates, complete the following steps:

1. On the default catalog's Settings page, change the Duplicate asset handling setting to anything other than Allow duplicates.
2. Immediately change the setting back to Allow duplicates. Now, when you publish to the default catalog, the UI for publishing assets will display the Add duplicate option.

No authentication methods listed when authenticating asset preview

Applies to: 4.7.1 and later

No authentication methods are available when unlocking a connection for an asset on the Asset page.

Workaround: Navigate to the Profile page and select an authentication method from the dropdown.

Relationships are not removed when assets are deleted from a catalog

Applies to: 4.7.1 and later

When deleting an asset that is part of a column-to-column or asset-to-column relationship, the relationship is not removed.

Workaround: Manually remove the relationship from the remaining asset or column. A failure message might appear even if the relationship is successfully removed.

Custom relationships don’t show on search results

Applies to: 4.7.1 and later

Custom relationships named with more than one word won’t show in search results on Manage asset and artifact definitions page.

Workaround: Search the relationship using only the first word of relationship’s name.

Data protection rules do not apply to column names that contain spaces

Applies to: 4.7.0 and later

If a column name contains trailing or leading spaces during import, the column cannot be masked using data protection rules.

Workaround: When importing columns, ensure column names do not contain trailing or leading spaces.

Viewers see edit options for metadata import assets

Applies to: 4.7.3

Catalog Viewers see the property edit options and the Reimport button for metadata import assets but will receive a permission error when attempting to edit.

Metadata file import asset name and description look editable to all users

Applies to: 4.7.3

Metadata file import asset name and description look editable to Asset members who are not Catalog Editors. When edits are attempted, permission errors prevent changes from being saved.

Masked column icon does not appear for asset previews

Applies to: 4.7.3

The shield icon used to identify masked columns does not appear for asset previews.

Catalog asset search only returns exact search term matches

Applies to: 4.7.0 and later

When searching for assets in a catalog, partial matches do not appear in the search results. Only assets that contain the search term at the beginning of the asset name appear in the search results.

Synchronize the data policy service (DPS) category caches

Applies to: 4.7.0 and later

For performance purpose, the data policy service (DPS) keeps a copy of glossary categories in caches. When categories are created, updated, or deleted, the glossary service publishes RabbitMQ events to reflect these changes. The DPS listens to these events and update the caches. However, in some rare occasions, the message might be lost when RabbitMQ service is down or too busy. The DPS provides a REST API utility to update the cache.

You can run the following REST API utility during downtime that has no category changes to help avoid unexpected enforcement results during the run and also avoid inaccurate cache updates:

curl -v -k -X GET --header "Content-Type: application/json"
  --header "Accept: application/json"
  --header "Authorization: Bearer ${token}"
    "${uri}/v3/enforcement/governed_items/sync/category"

This REST API is available in Watson Knowledge Catalog version 4.6.5 or later.

Masked data is not supported in data visualizations

Applies to: 4.7.0 and later

Masked data is not supported in data visualizations. If you attempt to work with masked data while generating a chart in the Visualizations tab of a data asset in a project the following error message is received: Bad Request: Failed to retrieve data from server. Masked data is not supported.

Cannot use CSV to move data class between Cloud Pak for Data instances

Applies to: 4.7.0

If you try to export data classes with matching method Match to reference data to CSV, and then import it into another Cloud Pak for Data instance, the import fails.

Workaround: For moving governance artifact data from one instance to another, especially data classes of this matching method, use the ZIP format export and import. For more information about the import methods, see Import methods for governance artifacts.

Error Couldn't fetch reference data values shows up on screen after publishing reference data

Applies to: 4.7.0

When new values are added to a reference data set, and the reference data set is published, the following error is displayed when you try to click on the values:

Couldn't fetch reference data values. WKCBG3064E: The reference_data_value for the reference_data which has parentVersionId: <ID> and code: <code> does not exist in the glossary. WKCBG0001I: Need more help?

When the reference data set is published, the currently displayed view changes to Draft-history as marked by the green label on the top. The Draft-history view does not allow to view the reference data values.

Workaround: To view the values, click Reload artifact so that you can view the published version.

You can't preview a watsonx.data asset that is masked with a data protection rule with row filtering

Applies to: 4.7.3 and later

The preview of an asset that is masked with a data protection rule with row filtering is not available. The issue occurs when you apply data protection rules with row filtering on assets that are imported from the watsonx.data connection.

HTTP method PATCH might not be supported in custom workflows

Applies to: 4.7.0 and later

Custom workflow templates might call a REST API by using the HTTP task activity offered by the Flowable workflow engine. The HTTP task activity in version 6.5.0 of the embedded Flowable workflow engine that is used in Watson Knowledge Catalog 3.0, 3.5, and 4.0 does not support the HTTP method PATCH. Trying to call a REST API using that method results in a "requestMethod is invalid" error. GET, POST, PUT, and DELETE methods work fine.

Workaround: Modify your REST API call to use the POST method instead, and add this special header to your request:

X-HTTP-Method-Override: PATCH

For this workaround to actually work, the called service must understand and correctly interpret this header field. Calls to REST APIs provided by the wkc-glossary-service service have worked properly.

Column information might not be available for data assets imported through lineage import

Applies to: 4.7.0 and later

When a metadata import is configured to get lineage from multiple connections and databases with the same name exist in these data sources, the tables from these databases are imported but no column information.

Workaround: Configure a separate metadata import for each connection pointing to same-named databases.

Running concurrent metadata import jobs on multiple metadata-discovery pods might fail

Applies to: 4.7.0 and later

When you run several metadata import jobs in parallel on multiple metadata-discovery pods, an error might occur, and an error message similar to the following one is written to the job run log:

Error 429: CDICW9926E: Too many concurrent user requests: 50

Workaround: You can resolve the issue in one of these ways:

• Increase the maximum number of concurrent requests allowed per user. In the wdp-connect-connection pod, change the value of the MAX_CONCURRENT_REQUESTS_PER_USER environment variable, for example:

  MAX_CONCURRENT_REQUESTS_PER_USER: 100
  

• If you don't have enough resources to increase the number of concurrent requests per user, reduce the number of threads connecting to the source. By default, 20 worker threads in a metadata-discovery pod access the wdp-connect-connection pod concurrently. If you define 4 pods for metadata import, 80 worker threads will access the data source at the same time. In a metadata-discovery pod, change the value of the discovery_create_asset_thread_count environment variable. For example:

  discovery_create_asset_thread_count: 10
  

Metadata import jobs might be stuck due to issues related to RabbitMQ

Applies to: 4.7.0 and later

If the metadata-discovery pod starts before the rabbitmq pods are up after a cluster reboot, metadata import jobs can get stuck while attempting to get the job run logs.

Workaround: To fix the issue, complete the following steps:

1. Log in to the OpenShift console by using admin credentials.
2. Go to Workloads > Pods.
3. Search for rabbitmq.
4. Delete the rabbitmq-0, rabbitmq-1, and rabbitmq-2 pods. Wait for the pods to be back up and running.
5. Search for discovery.
6. Delete the metadata-discovery pod. Wait for the pod to be back up and running.
7. Rerun the metadata import job.

Deleting all assets from a metadata import at once might not work

Applies to: 4.7.0 and later

To delete all imported assets from a metadata import at once, you can use the Select all option. However, if the number of assets that you want to delete is very large, deletion might fail.

Workaround: Delete the assets in batches, for example, by using the Select all on page option.

Asset list might be incomplete for an imported metadata import asset

Applies to: 4.7.1
Fixed in: 4.7.2

If a metadata import asset with a scope of more than 20 assets is imported from an exported project .zip file, not all assets might be shown in the asset list of the metadata import although all assets are added to the project.

Workaround: To have the asset list in the metadata import asset updated, rerun the metadata import.

Data assets might not be imported when running an ETL job lineage import for DataStage flows

Applies to: 4.7.2 and later

When you create and run a metadata import with the goal Get ETL job lineage where the scope is determined by the Select all DataStage flows and their dependencies in the project option, data assets from the connections associated with the DataStage flows are not imported.

Workaround: Explicitly select all DataStage flows and connections when you set the scope instead of using the Select all DataStage flows and their dependencies in the project option.

In some cases, you might not see the full log of a metadata enrichment job run in the UI

Applies to: 4.7.0 and later

If the list of errors in a metadata enrichment run is exceptionally long, only part of the job log might be displayed in the UI.

Workaround: Download the entire log and analyze it in an external editor.

Schema information might be missing when you filter enrichment results

Applies to: 4.7.0 and later

When you filter assets or columns in the enrichment results on source information, schema information might not be available.

Workaround: Rerun the enrichment job and apply the Source filter again.

Profiling in catalogs, projects, and metadata enrichment might fail for Teradata connections

Applies to: 4.7.0 and later

If a generic JDBC connection for Teradata exists with a driver version before 17.20.00.15, profiling in catalogs and projects, and metadata enrichment of data assets from a Teradata connection fails with an error message similar to the following one:

2023-02-15T22:51:02.744Z - cfc74cfa-db47-48e1-89f5-e64865a88304 [P] ("CUSTOMERS") - com.ibm.connect.api.SCAPIException: CDICO0100E: Connection failed: SQL error: [Teradata JDBC Driver] [TeraJDBC 16.20.00.06] [Error 1536] [SQLState HY000] Invalid connection parameter name SSLMODE (error code: DATA_IO_ERROR)

Workaround: Complete these steps:

1. Go to Data > Platform connections > JDBC drivers and delete the existing JAR file for Teradata (terajdbc4.jar).
2. Edit the generic JDBC connection, remove the selected JAR files, and add SSLMODE=ALLOW to the JDBC URL.

Running primary key or relations analysis doesn't update the enrichment and review statuses

Applies to: 4.7.0 and later

The enrichment status is set or updated when you run a metadata enrichment with the configured enrichment options (Profile data, Analyze quality, Assign terms). However, the enrichment status is not updated when you run a primary key analysis or a relationship analysis. In addition, the review status does not change from Reviewed to Reanalyzed after review if new keys or relationships were identified.

Metadata enrichment job is stuck in running state

Applies to: 4.7.0
Fixed in: 4.7.1

If the metadata enrichment service manager can't process the events for tracking the metadata enrichment job status, the job can get stuck.

Workaround: Restart the metadata enrichment service manager pod. To restart the wkc-mde-service-manager pod, you can scale the pod to zero and then back to its previous value:

oc scale deploy wkc-mde-service-manager --replicas=0 
oc scale deploy wkc-mde-service-manager --replicas=N  

When running enrichment on a large number of assets, profiling might fail for a subset

Applies to: 4.7.0, 4.7.1, and 4.7.2
Fixed in: 4.7.3

When metadata enrichment is run on a large number of data assets, processing might fail for a subset of the data assets. For each asset that couldn't be enriched, the following error message is written to the log of the metadata enrichment job:

DataProfile cannot be created as a profile already exists in the DataAsset with id \"<asset_ID>\"

Workaround: Rerun enrichment on the assets for which processing failed.

Restarts of the profiling pod might keep the metadata enrichment job in running state

Applies to: 4.7.0 and later

When metadata enrichment is run on a large number of data assets, the wdp-profiling pod might restart several times, which can cause the metadata enrichment job to be stuck in running state.

Workaround: Scale up the wdp-profiling pod to 2 replicas and rerun the job.

Project import doesn't retain default output settings for metadata enrichment

Applies to: 4.7.0 and later

When you import a previously exported project, the data quality output settings that were defined in the default enrichment settings in the originating project are no longer available. Instead, an error message is issued when you open the enrichment settings in the imported project.

Workaround: Edit the enrichment settings and reconfigure the data quality output settings as required.

When running metadata enrichment with all enrichment options, term assignment might fail for a subset of assets

Applies to: 4.7.0, 4.7.1, and 4.7.2
Fixed in: 4.7.3

When you run a metadata enrichment that includes term assignment, enrichment might fail for a subset of the assets. The following error is written to the log:

An unexpected error occurred while running the term assignment profile of data asset

Workaround: Rerun enrichment for the assets in question:

1. In the UI, use the filter option Enrichment failed.
2. Select the result set and rerun enrichment.

Running metadata enrichment jobs from the Jobs page might fail

Applies to: 4.7.0
Fixed in: 4.7.1

Running a metadata enrichment job from the Jobs page** might fail in these cases:

• If you start a Metadata Enrichment job run by clicking the run button ()) while another run of the same job is still in progress

• If you start a run of a Publish Metadata Enrichment Assets job

In either case, the job run isn't started and the error message 403 forbidden is displayed.

Workaround: Wait until the job run in progress is complete, then start a new Metadata Enrichment job run. Trigger any Publish Metadata Enrichment Assets job runs from within the metadata enrichment asset.

Metadata import or metadata enrichment no longer work after an associated job is deleted on the project's Job page

Applies to: 4.7.0, 4.7.1, and 4.7.2
Fixed in: 4.7.3

If you delete a job that is associated with a metadata import or a metadata enrichment by using the Delete option for the job on the project's Jobs page, the respective metadata import or metadata enrichment is no longer functional.

Workaround: To delete such job:

1. Open the metadata import or metadata enrichment asset.

2. In the About side panel, click the link for the job that you want to delete. This action takes you to the Job Details page.

   If the About side panel is hidden, you can show it by clicking the information icon.

3. To delete the job, click the delete icon in the toolbar.

Changes to the metadata enrichment scope might not be possible

Applies to: 4.7.0
Fixed in: 4.7.1

When you create a metadata enrichment and initially select an entire metadata import asset as the scope, you can't update the scope by using the Add assets to data scope option. When you click the link, a blank page is shown instead of the asset browser, and you can't proceed.

Workaround: Save and run the metadata enrichment with just the metadata import asset once. After the initial run is complete, edit the metadata enrichment and change the scope as required.

In environments upgraded from a version before 4.7.1, you can't filter relationships in the enrichment results by assigned primary keys

Applies to: 4.7.1 and later

Starting in Cloud Pak for Data 4.7.1, you can use the Primary key filter in the key relationships view of the enrichment results to see only key relationships with an assigned primary key. This information is not available in upgrade environments if you upgraded from a version before 4.7.1. Therefore, the filter doesn't work as expected.

Workaround: To generate the required information, you can rerun primary key analysis or update primary key assignments manually.

In reruns of metadata enrichment, key relationship analysis is always run on the entire data scope

Applies to: 4.7.1
Fixed in: 4.7.2

When you rerun a metadata enrichment that has the enrichment option Set relationships selected, key relationship analysis is always run on all assets in the data scope even if the scope for reruns is set to New or modified assets. As a consequence, key relationship analysis might run although no new or modified assets were detected, which has an impact on computing resources in Cloud Pak for Data and the data source.

Relationship analysis might report incorrect errors

Applies to: 4.7.1
Fixed in: 4.7.2

Relationship analysis checks only columns with specific data types to detect primary or foreign key candidates. Columns with String or Integral data types, such as CHAR, NCHAR, VARCHAR, NVARCHAR, CLOB, INTEGER, BIGINT, SMALLINT, TINYINT, are checked; columns of type Double or Binary are ignored.

If a data asset doesn't contain any columns of a supported data type, for example, if all columns have the type Double, the following error message is written to the job-run details log:

The data asset (Id=<data asset id>, name=<data asset name>) was not analyzed because it has not been profiled yet

In the described case, this message is not correct. You can ignore it.

Can't configure a custom service for term assignment

Applies to: 4.7.1
Fixed in: 4.7.2

If you select the Custom service option in the metadata enrichment settings to configure a custom ML model for term assignment, the settings aren't saved, and an error message is displayed. Clear the Custom service check box to have the settings saved.

Workaround: To enable the use of a custom service for term assignment, update the metadata enrichment settings by using the Watson Data API:

1. Retrieve the existing project settings:

   GET /v2/metadata_enrichment/metadata_enrichment_area/settings?project_id=
   

   The project ID is part of the URL when you open the project.

   The returned JSON should look similar to this example:

   {
     "term_assignment": {
       "term_assignment_threshold": 0.5,
       "term_suggestion_threshold": 0.4,
       "class_based_assignments": true,
       "ml_based_assignments": true,
       "name_matching": true,
       "ml_based_assignments_default": true,
       "ml_based_assignments_custom": false,
       "evaluate_negative_assignments": true,
       "default_ml_configuration": {
         "scope": "project",
         "catalog_id": "abc"
       }
     },
     "structured_profiling": {
       "null_threshold": 0.05,
       "uniqueness_threshold": 0.95,
       "constant_threshold": 0.99,
       "quality_score_threshold": 0.8,
       "data_class_assignment_threshold": 0.75,
       "data_class_suggestion_threshold": 0.25
     }
   }
   

2. To define the custom service, edit the returned JSON. Set the ml_based_assignments_custom property to true and add the wml_configuration section as follows:

   "ml_based_assignments_custom": true,
   "wml_configuration": {
         "enabled": true,
         "deployment_id": <deployment_id>,
         "deployment_name": <deployment_name>,
         "space_id": <space_id>,
         "space_name": <space_name>,
         "version": <deployment_API_version>,
         "input_transformation_expression": <input_transformation_expression>
         "output_transformation_expression": <output_transformation_expression>
   }
   

   To get the required information:

   1. Go to Deployments > Spaces and click the name of your deployment space.
   2. On the Deployments page, click the name of your deployment to see the deployment details, such as the deployment name, the deployment ID, and the deployment version.
   3. On the Manage page of your deployment space, you find the space name and the space ID.

   The input and output transformation expressions are string values. You must escape any double quotation marks in your expressions with a backslash (\).

   "ml_based_assignments_custom": true,
   "wml_configuration": {
     "enabled": true,
     "deployment_id": "e94e1323-8d54-4048-9b59-b4771ccdf379",
     "deployment_name": "demo_tp_scoring_deployment",
     "space_id": "69670b81-ec64-4d53-8665-c95ad9905bb9",
     "space_name": "TestDeployment",
     "version": "2023-07-12",
     "input_transformation_expression": "{\"input_data\":[{\"values\":$append([ [$$.metadata.name, \"\"] ], $$.entity.data_asset.columns.[[$$.metadata.name, name]])}]}",
     "output_transformation_expression": "{\"term_assignments\": predictions[0].values ~> $map(function($x){function($z){$count($z) > 1? $z : [$z]}($x[0] ~> $zip($x[1]) ~> $map(function($y){{\"term_id\": $y[0], \"confidence\": $y[1]}})) })}" 
   },
   

3. Update the metadata enrichment settings. Submit the following API request with the updated JSON as request body.

   PUT /v2/metadata_enrichment/metadata_enrichment_area/settings?project_id=
   

Minimum database software version for writing metadata enrichment output

Applies to: 4.7.1 and later

If you want to write data quality output generated by metadata enrichment to an Apache Hive database, the required minimum software version is release 3.0.0.

Running overlap analysis on data imported from another project might not update results as expected

Applies to: 4.7.3

When data assets are imported from an exported project that also includes results of previous overlap analyses, the relationships are correctly shown. However, running a new overlap analysis on these data assets might not update existing results but create a second entry for the same relationshop in reverse order.

For example, the imported results show a relationship with Asset A - column a as base asset and Asset B - column b as paired asset. Running overlap analysis creates a new entry for this relationship with Asset B - column b as base asset and Asset A - column a as paired asset.

Workaround: Delete redundant relationships.

Metadata enrichment or profiling can't be run on data from connections that use Cloud Pak for Data credentials

Applies to: 4.7.3

You can't profile or run metadata enrichment on data assets from connections that are configured to use the user's platform login credentials to authenticate to the data source

Workaround: Update the connection to not use the platform login credentials or select a different authentication method, for example, API key.

Rules run on columns of type timestamp with timezone fail

Applies to: 4.7.0 and later

The data type timestamp with timezone is not supported. You can't apply data quality rules to columns with that data type.

Rules fail because the job's warning limit is exceeded

Applies to: 4.7.0 and later

For some rules, the associated DataStage job fails because the warning limit is reached. The following error message is written to the job log:

Warning limit 100 for the job has been reached, failing the job.

The default limit for jobs associated with data quality rules is 100.

Workaround: Edit the configuration of the DataStage job and set the warning limit to 1,000. Then, rerun the job.

Rule output for data sources connected through Generic JDBC and those connected through a native connector might differ

Applies to: 4.7.0
Fixed in: 4.7.1

For data assets in a data source that is connected through a Generic JDBC connector, the rule output might be different from the output that is generated if the data source is connected through the native connector. The difference is due to the fact that rows where all values are NULL are removed from the output if the data source is connected through a Generic JDBC connector.

Random sampling with a percentage of 100 is not supported for use with the Oracle (optimized) connector

Applies to: 4.7.0
Fixed in: 4.7.1

If you configure and run a rule with random sampling and a percentage of 100 for a data source that is connected through the Oracle (optimized) connector, the associated DataStage flow fails. An error message similar to the following one is issued:

"SAMPLE percentage must be in the range [0.000001,100)"

Workaround: With the Oracle (optimized) connector, use a percentage in the range 1%-99%. If you want your sample to include all records, you can alternatively work with sequential sampling.

Data quality information for an asset might not be updated if a data quality rule is deleted

Applies to: 4.7.0
Fixed in: 4.7.1

If a data quality rule is deleted, its associated data quality issues sometimes might not get deleted. In this case, assets can have orphaned issues attached that still have an impact on the data quality score.

Workaround: Delete the orphaned issues by using the Watson Data API Create a data quality issue.

Rules with multiple joins might return incorrect results for data assets from Apache Cassandra, Apache Hive, or MongoDB data sources

Applies to: 4.7.1 and later

A data quality rule that is created from one or more data quality definitions and contains multiple joins might return incorrect results when is is run on data assets from Apache Cassandra, Apache Hive, or MongoDB data sources that are connected through a Generic JDBC connection.

Dates are not filled properly if the language is set to Japanese

Applies to: 4.7.2 and later

If the browser language is set to Japanese, date input fields in data quality rules are not filled as expected.

Workaround: Set the browser language to English when you need to set a date.

The wkc-data-rules pod goes into status Completed or ContainerStatusUnknown

Applies to: 4.7.2
Fixed in: 4.7.3

The wkc-data-rules pod goes into status Completed or ContainerStatusUnknown. To verify the status, run:

oc get pods | grep wkc-data-rules

Workaround: Update the ephemeral storage limit of wkc-data-rules deployment to 2 GB:

1. Verify the existing limit with the following command:

   oc get deployment wkc-data-rules --output="jsonpath={.spec.template.spec.containers[*].resources.limits.ephemeral-storage}" && echo -e "\n"
   

2. If the limit is set to 1Gi, change the setting to 2Gi with the following command:

   oc patch wkc wkc-cr -n wkc --type merge -p '{"spec":{"wkc_data_rules_resources":{"requests":{"cpu":"100m","memory":"800Mi","ephemeral-storage":"50Mi"},"limits":{"cpu":1,"memory":"2048Mi","ephemeral-storage": "2Gi" }}}}'
   

Metadata import jobs for getting lineage might take very long to complete

Applies to: 4.7.0 and later

If multiple lineage scans are requested at the same time, the corresponding metadata import jobs for getting lineage might take very long to complete. This is due to the fact that MANTA Automated Data Lineage workflows can't run in parallel but are executed sequentially.

Chrome security warning for Cloud Pak for Data deployments where MANTA Automated Data Lineage for IBM Cloud Pak for Data is enabled

Applies to: 4.7.0 and later

When you try to access a Cloud Pak for Data cluster that has MANTA Automated Data Lineage for IBM Cloud Pak for Data enabled from the Chrome web browser, the message Your connection is not private is displayed and you can't proceed. This is due to MANTA Automated Data Lineage for IBM Cloud Pak for Data requiring an SSL certificate to be applied and occurs only if a self-signed certificate is used.

Workaround: To bypass the warning for the remainder of the browser session, type thisisunsafe anywhere on the window. Note that this code changes every now and then. The mentioned code is valid as of the date of general availability of Cloud Pak for Data 4.6.0. You can search the web for the updated code if necessary.

Rerun of a lineage import fails if assets were deleted from the source

Applies to: 4.7.0 and 4.7.1
Fixed in: 4.7.2

When you rerun an lineage import after assets were deleted from the data source, the reimport fails with an error message similar to this one:

message" : "This error occurred while accessing the connectors service: The assets request failed: CDICO2005E: Table could not be found: SCHEMA.TABLE. If the table exists, then the user might not be authorized to see it.

In ETL job files, the name of the connection folder in the root structure must not be the technology name

Applies to: 4.7.0, 4.7.1, and 4.7.2

Fixed in: 4.7.3

The ETL job files that provide the data scope for metadata imports with the Get ETL job lineage option have a prescibed format. The root input folder follows this convention:

input/<technologyName>/<connectionName>

If the name of the connectionName folder is the same as the name of the technologyName folder, the lineage import fails.

Workaround: Make sure that the folder names are different. If required, you can decompress the .zip file and update the folder name. Then, compress the ETL job file again.

Metadata imported from Apache Hive, PostgreSQL, and Qlik Sense connections is not reimported after upgrading Watson Knowledge Catalog from 4.7.2 to 4.7.3

Applies to: 4.7.3

In 4.7.2 version, you created a metadata import with lineage from these connections: Apache Hive, PostgreSQL, and Qlik Sense. After the upgrade of Watson Knowledge Catalog to 4.7.3 version, when you try to reimport the data, either no data is reimported, or it is reimported only partially. In MANTA Automated Data Lineage, the workflow for this process is finished with error.

Workaround: From the manta-admin-gui pod, run the following command for each connection.

curl -k -i -X PUT 'https://manta-configuration-service:8083/private/configuration/v1/unset' --header 'Authorization: Bearer <bearer_token>' --header 'Content-Type: application/json' --data-raw '{
    "setBy": "WKC",
    "validFrom": "2023-04-04T14:14:00.000+00:00",
    "key": "<manta_keystore_password_property_name>",
     "applicationScope": "MANTA_FLOW_CLI",
     "componentScope": "<manta_Technology>"
}'

Provide values for the following properties. See table below the list for specific values.

• Bearer: replace <bearer_token> with the bearer token for the IBM Cloud Pak for Data platform.
• "key": replace the <manta_keystore_password_property_name> with the name of the keystore password property for the specific MANTA Automated Data Lineage technology.
• "componentScope": replace <manta_Technology> with the name of the specific MANTA Automated Data Lineage technology.

Columns are displayed as numbers for a DataStage job lineage in the catalog

Applies to: 4.7.3 and later

The columns for a lineage that was imported from a DataStage job are not displayed correctly in the catalog. Instead of column names, column numbers are displayed. The issue occurs when the source or target of a lineage is a CSV file.

Assets connected through promoted lineage flows appear on column lineage

Applies to: 4.7.0 and later

To improve performance of lineage on assets higher in hierarchy, business lineage promotes lineage flows from low levels to higher levels. Other assets connected through promoted lineage flow can appear on column lineage.

Lineage graph exported to PDF file might be incomplete

Applies to: 4.7.0 and later

If your lineage graph resolution is more than 1451  × 725, you won’t be able to export the entire generated graph to PDF file.

Workaround: In the mini map, select Fit graph to screen, and download your graph.

Links on the side panel on Lineage page don’t work with DataStage job assets

Applies to: 4.7.2

Link Go to asset’s technical data lineage and Go to asset’s flow in project will redirect you to a wrong page.

Workaround: To see the asset’s flow in project, go to the Projects page, open the project in which the DataStage metadata import asset was created and choose the imported DataStage flow asset. This asset won’t appear in the asset's technical data lineage view and can’t be opened.

Links on the side panel on Lineage page don’t work with DataStage job assets

Applies to: 4.7.2

Link Go to asset’s technical data lineage and Go to asset’s flow in project will redirect you to a wrong page.

Workaround: To see the asset’s flow in project, go to the Projects page, open the project in which the DataStage metadata import asset was created and choose the imported DataStage flow asset. This asset won’t appear in the asset's technical data lineage view and can’t be opened.

An empty graph occurs after trying to show mappings

Applies to: 4.7.2

The lineage graph will show an empty graph state after selecting Show mappings on a mapping group asset.

Workaround: Refresh the page at the browser level to show your lineage graph again.

Lineage graph fails to display lineage relationship mapping

Applies to: 4.7.3

When the relationship mapping is imported between two assets, and you try to display the mapping in the lineage graph, the graph fails to load with the loops not allowed error similar to this:

Unknown error
{"trace":"fc442491-44e6-4806-96fe-4a92b08ce865","errors":[{"code":"bad_request","message":"loops not allowed"}],"statusCode":400,"transactionId":"eyvw633jh8fm3bgdvsjrh7ytr"}

Workaround: Try reimporting the lineage relationship mapping.

The name of lineage relationship mapping displays path

Applies to: 4.7.2

When creating lineage relationship mapping asset, the name of the mapping contains path extracted from a mapping group. For example, relationship mapping file with a name mapping1 will be displayed as /root/mappinggroup/mapping1.

Lineage metadata don’t show on Knowledge Graph after upgrading

Applies to: 4.7.2

After upgrading to 4.7.2, an unknown error appears on the lineage tab.

Workaround: To start seeing the Knowledge Graph, you need to resync catalogs' metadata, see Resync of lineage metadata.

Visualized asset’s name not available

Applies to: 4.7.3

The name of visualized asset is not available when viewing lineage graph.

Workaround: Resync your lineage metadata following steps described in Resync of lineage metadata.

No link to technical data lineage for IBM Db2 assets

Applies to: 4.7.3

After selecting an IBM Db2 asset in lineage graph, there’s no technical data lineage on the side panel.

Workaround: Search for the asset in MANTA Automated Data Lineage.

Business lineage between data assets and IBM Cognos Analytics assets is incomplete

Applies to: 4.7.0 and later

After you import metadata from IBM Cognos Analytics with its source data assets, business data lineage does not include these source data assets.

Parent topic: Limitations and known issues in Cloud Pak for Data