Updating instance groups to use an updated Spark version or notebook package

When an updated Apache Spark version or notebook package becomes available, upon adding them to your system to use with IBM® Spectrum Conductor, you can then update existing instance groups to use the upgraded packages.

Before you begin

  • You must be a cluster administrator, consumer administrator, or have the Instance Groups Configure permission.
  • The Spark version and notebook packages must be added to the system (see Adding Spark versions or Adding notebooks). The built-in Spark and notebook versions are installed with IBM Spectrum Conductor. You are not required to add those packages, unless you removed them.
  • The instance group must be in the Registered, Ready, Register Error, or Deploy Error state. If the instance group is running workload, stop the instance group and all associated notebooks before you update it. See Stopping instance groups and Stopping notebooks in an instance group.

About this task

When an updated Spark version or notebook package is added, that package becomes available for use with your instance groups. To enable an existing instance group to use the updated packages, that Spark version or notebook package must be deployed to the instance group.

Procedure

  1. From the cluster management console, select Workload > Instance Groups.
  2. When Spark version and notebook package updates are available for the instance group, make your selection as follows:
    Note: For updated notebook packages, the notebook is undeployed and the new version is deployed. Therefore, if you specified the notebook base data directory to be either in the same location or under the notebook's deployment directory, the base data directory is removed. To retain your data, manually back up the contents of the base data directory before you update the instance group configuration.
    • To only update the packages, under the Updates column, click the update for a instance group. When prompted, click Update.
    • To update the packages and change instance group settings, select the instance group to update and click Configure. When prompted, click Continue. For details on changing instance group settings, see Modifying instance groups.

Results

The updated packages are deployed to hosts of the instance group that use those packages.

What to do next

Start the instance group. See Starting instance groups.