Administering Cloud Pak for Data clusters

Register a remote cluster

Sign in as the administrator, click the menu icon ( ) and click Administration > Configurations > Systems integration to register your remote clusters. Click New integration. Assign the registration a name and provide the Service URLs and Service User ID that you received from the remote cluster admin.

Deleting a remote cluster registration

If you need to delete a registration, be aware that user assets that depend on this registration will no longer work properly. This includes connectors, environments, jobs, and notebooks that depend on the environments.

If a registration is later created with the same ID, users must still re-create the environment, and update all jobs and notebooks to reference the newly created environment in order for the assets to work properly. If the registration is created with a different ID, users must update connections to ensure that the URL that is referenced is correct in addition to updates needed for the jobs and notebooks.

If you need to refresh the registration, such as if you re-installed on the remote cluster for Execution Engine for Apache Hadoop, select the registration. Refresh the certificates first, and then wait a few minutes to allow the dependent pod to be re-created. Refresh the endpoints to ensure all configurations are refreshed.

View details about the registered cluster

In the Details page of each registration, you can view the endpoints, the edge nodes in the high availability setup, and runtimes.

• For Hadoop remote clusters, depending on the services that are exposed when you’re installing and configuring the Execution Engine for Apache Hadoop service, the details page lists the WebHDFS, WebHCAT, Livy spark 2, and JEG.

Push runtime images to the registered remote cluster

Data scientists can leverage the Python packages and custom libraries that are installed in a Jupyter Python environment when they’re working with models on remote cluster. These remote images can be used to run notebooks, notebook jobs, and Python script jobs (for Hadoop clusters only)

To push a runtime image to the remote cluster from its Details page, an admin uses the push operation to instantiate the process.

• For remote Hadoop clusters, an image archive is first built on the Cloud Pak for Data cluster, and then pushes that archive as-is to HDFS on the target Hadoop cluster. This operation is useful when the architecture and GCC library versions on the Hadoop cluster nodes are compatible with those on the Cloud Pak for Data cluster nodes.

After the image is pushed, a subset of libraries that are available for the Python environments are local to Cloud Pak for Data. The subset of libraries can be sent over to the remote Anaconda instances, and then be created. They are created using the specified Anaconda channels. Users on Cloud Pak for Data can then leverage the new Anaconda environment to run their notebooks, providing a similar experience to running the notebook on local Cloud Pak for Data environment.

The set of Python libraries are filtered so that internal IBM packages are not exported.

Pushing the image can take a long time. If the node goes down, retry and push the image again.

If you modified any of the runtime images locally, you can update it on the remote cluster by clicking Replace Image next to it.

Runtimes can have the following statuses:

• Available on Cloud Pak for Data, but not pushed to the registered remote cluster. Users can either push or refresh the environment to the registered remote cluster.
• Pending transfer from Cloud Pak for Data to the registered remote cluster.
• Failed transfer from Cloud Pak for Data to the registered remote cluster.
• Push succeeded on the registered remote cluster.