Table of contents

Known issues for Watson Studio and supplemental services

These known issues apply to Watson Studio and the services that require Watson Studio.

Also see Cluster imbalance overwhelms worker nodes.

General issues

UI display might not load properly

If the UI does not load properly, the Watson Studio administrator must restart redis.

Workaround

If the client behaves unexpectedly, for example, enters a redirection loop or parts of the user interface fail to load, then complete the following steps to restart redis:

  1. Log in to the OpenShift cluster.
  2. Restart the pods for redis using the following command:
     oc delete po -n <project> | oc get po -n <project> -l component=redis -o jsonpath="{.items[*].metadata.name}"
    
    

Connections

OData or SAP OData: Target connections and target connected data assets are not supported

  • Jobs for Data Refinery flows that use an OData or a SAP OData target connection or a target connected data asset will fail.
  • In SPSS Modeler, exporting data to an OData or to a SAP OData connection or to a connected data asset will result in error.

OData or SAP OData for Data Refinery: Cannot select partitioned data as a source for a Data Refinery flow

If you want to create a Data Refinery flow from an OData or a SAP OData partitioned data set, create a connected data asset to the file. For instructions see Adding data from a connection to an analytics project. You can then use the connected data asset as the source for the Data Refinery flow.

Planning Analytics for Data Refinery: Target connections and target connected data assets are not supported

Jobs for Data Refinery flows that use a Planning Analytics target connection or a target connected data asset will fail.

Personal credentials are supported in Data Refinery only through connected data assets

You cannot select a personal connection as a source, target, or for a Join operation in Data Refinery. To refine data, you must create a connected data asset from the personal connection first, then use that connected data asset for the source, target, or in a Join operation.

Workaround:

  1. Create a connected data asset for the data you want to refine. Instructions. The target location for the output file of the Data Refinery flow must also be a connected data asset.
  2. Go to the Assets page and click the connected data asset.
  3. Click Refine.
    If the preview of a target connected data asset does not refresh, go to the data asset directly by browsing to the connection on Data Refinery:
    1. Go to the project’s Assets tab and click New Data Refinery flow in the Data Refinery flows section.
    2. From Connections, select the connection and drill down to the data asset.

Authorization for Hadoop Execution Engine connections in a Watson Studio project or a Watson Studio catalog

When you create a Hadoop Execution Engine connection, you do not enter credentials. Instead, select the “Use Watson Studio authorization for the connection” checkbox in the Connection Details. Similarly, if the Hadoop Execution Engine connection is published to a catalog, any user, including the owner, needs to select the “Use Watson Studio authorization for the connection” checkbox.

Assets

Limitations for previews of assets

You can’t see previews of these types of assets:

  • Folder assets associated with a connection with personal credentials. You are prompted to enter your personal credentials to start the preview or profiling of the connection asset.
  • Connected data assets for image files in projects.
  • Connected assets with shared credentials of text and JSON files are incorrectly displayed in a grid.
  • Connected data assets for PDF files in projects.

Can’t load files to projects that have #, %, or ? characters in the name

You can’t create a data asset in a project by loading a file that contains a hash character (#), percent sign (%), or a question mark (?) in the file name.

Hadoop integration

On certain HDP Clusters, the Execution Engine for Apache Hadoop service installation fails

The installation fails during the Knox Gateway Configuration step. The issue is because the Knox gateway fails to start and happens on some nodes. See https://issues.apache.org/jira/browse/KNOX-1987 for more information.

The following error occurs:

Failed to configure gateway keystore Exception in thread "main" Caused by: java.lang.NoSuchFieldError: DEFAULT_XML_TYPE_ATTRIBUTE

The workaround is to remove the org.eclipse.persistence.core-2.7.2.jar file from the installation directory by using the following command:

mv /opt/ibm/dsxhi/gateway/dep/org.eclipse.persistence.core-2.7.2.jar /tmp/

Cannot stop jobs for a registered Hadoop target host

When a registered Hadoop cluster is selected as the Target Host for a job run, the job cannot be stopped. As a workaround, view the Watson Studio Local job logs to find the Yarn applicationId; then, use the ID to manually stop the Hadoop job on the remote system. When the remote job is stopped, the Watson Studio Local job will stop on its own with a “Failed” status. Similarly, jobs that are started for registered Hadoop image push operations cannot be stopped either.

Cross platform between Watson Studio Local and a remote Hadoop cluster is not supported for virtual environments

For virtual environments only, Watson Studio Local and Hadoop must have matching architectures. For example, a virtual Watson Studio Local on POWER environment can work only with a virtual Hadoop POWER environment.

Notebooks

Error in notebooks when rendering data from Cloudera Distribution for Hadoop

When running Jupyter notebooks against Cloudera Distribution for Hadoop 5.16 or 6.0.1 with Spark 2.2, the first dataframe operation for rendering cell output from the Spark driver results in a JSON encoding error.

Workaround:

When running Jupyter notebooks against CDH 5.16 or 6.0.1 with Spark 2.2, the first dataframe operation for rendering cell output from the Spark driver results in a JSON encoding error. To workaround this error, do one of the following procedures:

  • For interactive notebook sessions

    Manually re-run the first cell that renders data.

  • For non-interactive notebook sessions

    Add the following non-intrusive code after establishing the Spark connection to trigger the first failure:

	%%spark
	import sys, warnings
	def python_major_version ():
	    return(sys.version_info[0])
	with warnings.catch_warnings(record=True):
         print(sc.parallelize([1]).map(lambda x: python_major_version()).collect())

Database connection added from a global connection to a project in Jupyter Notebook or JupyterLab results in an error

If you’re using a database connection that was added from a global connection to the project in a Jupyter Notebook or JupyterLab, the following error might occur if the connection uses SSL:

com.ibm.db2.jcc.am.DisconnectNonTransientExceptionPyRaisable: com.ibm.db2.jcc.am.DisconnectNonTransientException: [jcc][t4][2034][11148][4.26.14] Execution failed due to a distribution protocol error that caused deallocation of the conversation.

To work around the issue you can do one of the following:

  • Create the connection directly in the project as opposed to adding a global connection to the project.
  • Adjust the generated code that was inserted into the notebook by replacing:
    if(ssltestconnection_credentials.get('ssl,'false') == 'true'):
    

    with

    if(str(ssltestconnection_credentials.get('ssl,'false')).lower() == 'true'
    

Missing pyspark modules in migrated notebooks

If you attempt to run an existing Spark notebook that you created in a previous version of Cloud Pak for Data, you might receive an missing pyspark module error.

To run migrated Spark notebooks:

  1. Install the Analytics Engine powered by Apache Spark service, if it isn’t already installed. See Analytics Engine powered by Apache Spark.
  2. Associate your notebook with the Default spark python 3.6 environment. See Change your environment.
  3. If necessary, restart the runtime kernel in your notebook.

Notebook loading considerations

The time that it takes to create a new notebook or to open an existing one for editing purposes might vary. If no runtime container is available, a container needs to be created and only after it is available, the Jupyter notebook user interface can be loaded. The time it takes to create a container depends on the cluster load and size. Once a runtime container exists, subsequent calls to open notebooks will be significantly faster.

Kernel not found when opening a notebook imported from a Git repository

If you import a project from a Git repository that contains notebooks that were created in JupyterLab, and try opening the notebooks from the project Assets page, you will see a message stating that the required notebook kernel can’t be found.

The reason is that you are trying to open the notebook in an environment that doesn’t support the kernel required by the notebook, for example in an environment without Spark for a notebook that uses Spark APIs. The information about the environment dependency of a notebook that was created in JupyterLab and exported in a project is currently not available when this project is imported again from Git.

Workaround:

You need to associate the notebook with the correct environment definition. You can do this:

  • From the notebook opened in edit mode by:

    1. Clicking the Notebook Info icon (Notebook Info icon) from the notebook toolbar and then clicking Environment.
    2. Selecting the correct environment definition for your notebook from the list under Environments.
  • Before you open the notebook, from the project Assets page by:

    1. Selecting the notebook and unlocking it if it is locked. You can only change the environment of a notebook if the notebook is unlocked.
    2. Clicking Actions > Change Environment and selecting the correct environment definition for your notebook.

Cannot load exported GPU notebooks when imported to a different cluster

If you export a project with a notebook that use a GPU environment from one cluster and import the project to another cluster, you will see a message stating that the notebook cannot be loaded when you try opening it in the imported project. The reason for this is that the imported GPU environment definition associated with the notebook cannot be identified by the new cluster.

Workaround:

You need to create a GPU environment definition in the imported project and associate the imported notebook with this new environment definition. To do this:

  1. Create a GPU environment definition. See Creating environment definitions.
  2. Change the environment of the notebook to the GPU environment that you created. See Changing the environment.

Environment runtime can’t be started because the software customization failed

If your Jupyter notebook runtime can’t be started and a 47 killed error is logged, the software customization process could not be completed because of lack of memory.

You can customize the software configuration of a Jupyter notebook environment by adding conda and pip packages. However, be aware that conda does dependency checking when installing packages which can be memory intensive if you add many packages to a customization.

To complete a customization successfully, you must make sure that you select an environment with sufficient RAM to enable dependency checking at the time the runtime is started.

If you only want packages from one conda channel, you can prevent unnecessary dependency checking by excluding the default channels. To do this, remove defaults from the channels list in the customization template and add nodefaults.

Anaconda Repository with IBM

Channel names for Anaconda Repository with IBM don’t support double-byte characters

When you create a channel in Anaconda TE, you can’t use double-byte characters or most special characters. You can use only these characters: a-z 0-9 - _

Rstudio

Running job for R script and selected RStudio environment results in an error

When you’re running a job for an R Script and a custom RStudio environment was selected, the following error occurs if the custom RStudio environment was created with a previous release of Cloud Pak for Data: The job uses an environment that is not supported. Edit your job to select an alternative environment.

To work around this issue, delete and re-create the custom RStudio environment with the same settings.

Git integration broken when RStudio crashes

If RStudio crashes while working on a script and you restart RStudio, integration to the associated Git repository is broken. The reason is that the RStudio session workspace is in an incorrect state.

Workaround

If Git integration is broken after RStudio crashed, complete the following steps to reset the RStudio session workspace:

  1. Click on the Terminal tab next to the Console tab to create a terminal session.
  2. Navigate to the working folder /home/wsuser and rename the .rstudio folder to .rstudio.1.
  3. From the File menu, click Quite Session… to end the R session.
  4. Click Start New Session when prompted. A new R project with Git integration is created.

RStudio doesn’t open although you were added as project collaborator

If RStudio will not open and all you see is the endless spinner, the reason is that, although you were added as collaborator to the project, you have not created your own personal access token to the Git repository associated with the project. To open RStudio with Git integration, you must select your own access token.

To create your own personal access token, see Collaboration in RStudio.

Data Refinery

User with the Editor role cannot save a Data Refinery flow (First-time use)

You must have the Admin role to save a Data Refinery flow (or to save and create a job for a Data Refinery flow) in the first project after installation. Thereafter, a user with the Editor role can save a Data Refinery flow in any project.

CSV file limitations

Be sure that CSV files are correctly formatted and conform to the following rules:

  • Files can’t have some rows ending with null values and some columns containing values enclosed in double quotation marks.
  • Two consecutive commas in a row indicate an empty column.
  • If a row ends with a comma, an additional column is created.

Column name limitations

Be sure that column names conform to the following rules:

  • The column names are unique within the data set.
  • The column names are not reserved words for R.
  • The column names are not numbers. A workaround is to enclose the column names in double quotation marks (“”).

System-level schemas aren’t filtered out

When creating a connection to IBM Db2 Warehouse on Cloud (previously named IBM dashDB), system-level schemas aren’t filtered out.

Target connection limitations

When you save the Data Refinery flow output (target data sets) to connections, Data Refinery flows that have a target on a Compose for MySQL connected data asset are not supported.

Target file limitations

If you save Data Refinery flow output (the target data set) to a file, do not change the file format if the file is an existing data asset.

White space characters are considered as part of the data

If your data includes columns that contain white space (blank) characters, Data Refinery considers those white space characters as part of the data, even though you can’t see them in the grid. Be aware that some database tools might pad character strings with white-space characters to make all the data in a column the same length and this change will affect the results of Data Refinery operations that compare data.

Jobs

Excluding days when scheduling a job causes unexpected results

If you select to schedule a job to run every day of the week excluding given days, you might notice that the scheduled job does not run as you would expect. The reason might be due to a discrepancy between the timezone of the user who creates the schedule, and the timezone of the master node where the job runs.

This issue only exists if you exclude days of a week when you schedule to run a job.

Error occurs when jobs are edited

You cannot edit jobs that were created prior to upgrading to Cloud Pak for Data version 3.0. An erorr occurs when you edit those jobs. Create new jobs after upgrading to Cloud Pak for Data version 3.0.

Watson Machine Learning

Watson Machie Learning might require manual rescaling

By default, the small installation of Watson Machine Learning comes up with one pod. When the load on the service increases, you may experience these symptoms, indicating the need to manually scale the wmlrepository service:

  1. wmlrepository service pod restarts with an Out Of Memory error
  2. wmlrepository service request fails with this error:
    Generic exception of type HttpError with message: akka.stream.BufferOverflowException: Exceeded configured max-open-requests value of [256]. This means that the request queue of this pool  has completely filled up because the pool currently does not process requests fast enough to handle the incoming request load. Please retry the request later. See http://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information.

Use this command to scale the repository:

  ./cpd-linux scale -a wml --config medium -s server.yaml  -n <namespace>

  medium.yaml
  commands:
  - scale --replicas=2 deployment wmlrepository

Do not import/export models between clusters running on different architectures

When you export a project or space, the contents, including model assets, are included in the export package. You can then import the project or space to another server cluster. Note that the underlying architecture must be the same or you might encounter failures with the deployment of your machine learning models. For example, if you export a space from a cluster running the Power platform, then import to a cluster running x86-64, you may be unable to deploy your machine learning models.

Display issues on some Firefox versions

Some parts of the Watson Machine Learning interface will not render properly on older versions of Firefox. If you encounter display issues, upgrade to the latest version of Firefox to resolve the problems.

AutoAI character support

Currently, AutoAI supports double-byte characters (UTF-8) in the content of a model, but not in the file name.

Deleting model definitions used in Deep Learning experiments

Currently, users can create create model definition assets from the Deep Learning Experiment Builder but cannot delete a model definition. They must use REST APIs to delete model definition assets.

No visual indicator for promoted assets

Currently, there is no indication of which assets have already been promoted to a space. As a result, multiple copies of the asset can be promoted to a space if users promote more than once. This does not cause problems but users might find it confusing.

Manually delete project tag from space

When a project is deleted, the associated space can get into a state where you cannot load the spae or associate it with another project unless you first manually remove the project tag from the space metadata. To do so, use the update_space method from the Watson Machine Learning API.

Deprecation of space assets in the API

The assets array in the Space resource response API is deprecated in Cloud Pak for Data 3.0 because of performance problems fetching the assets in a space. To view assets in deployment spaces, use one of these methods:

  1. To see meta data for all spaces, use query parameters such as include=name or get_spacefor an individual space.

  2. All specific asset types in a space can be fetched via Watson Machine Learning APIs. For example, GET /v4/models?space_id={space_id} will fetch all the models in the given space.

AutoAI notebook can’t access data from catalog

If you are working in a JupyterLab project and save an AutoAI pipeline as a notebook, the notebook will be unable to load the input data if the data asset was added from a knowledge catalog. To workaround this limitation, download the data asset from the catalog and manually upload it to the project.

AutoAI online deployment can generate misleading error for schema mismatch

If you create an online dpeloyment for an AutoAI model but you submit input data with missing values, the resulting error does not indicate the corect fields. Instead, the error will subsititute generic field names, such as “f1”, “f2”, and so on. You must review the schema and make sure you are supplying the correct input data to correct the problem.

Decision optimization

Decision Optimizaton batch jobs creation

While the existing batch deployment interface show details about existing Decision Optimization batch deployment job runs, there is no interface for new Decision Optimization batch deployment job run creation. Users must use APIs or other clients to create Decision Optimization batch deployment job runs.

Manually remove deployments to avoid resource consumption

Decision Optimization deployments use long running pods to execute jobs submitted to them. Over time, if the long running pods are not cleaned up, they will continue to use resources as long as they are running. This can result in insufficient resources being available for other deployments. To avoid this issue, delete Decision Optimization deployments that are no longer needed to clean up the long running pods and free up resources.