Provisioning native Spark engine

IBM® watsonx.data allows you to provision native Spark engine to run complex analytical workloads.

About this task

To provision a native Spark engine, complete the following steps.

Procedure

  1. Log in to watsonx.data console.
  2. From the navigation menu, select Infrastructure Manager.
  3. To provision an engine, click Add component and select IBM Spark.
  4. Click Next.
  5. In the Add component - IBM Spark window, enter the Display name for your Spark engine.
  6. Select Create a native Spark engine, and do the following:
    1. Specify the storage volume that is considered as Engine home, which stores the Spark events and logs that are generated while running spark applications. You can either select an existing storage volume or specify details to create a new storage volume. Choose one of the following options:
      Note: To store Spark application, create a different storage volume. To create storage volume , see Creating a storage volume.
      • Option1: Select an existing volume. To do that, specify the following fields:
        • Existing volume: Select the option to associate a storage volume that is already available in the cluster. To create storage volume , see Creating a storage volume.
        • Select volume: To use an existing volume, select the storage volume from the list.
      • Option2 : Create a new storage volume. To do that, specify the following fields:
        • New Volume: Select the option to create a new storage volume and use it.
        • Volume name: Enter a name for the new storage volume.
        • Storage Class: Select the class to which the storage volume belongs.
        • Size of the new storage volume: Slide to select the volume size in GB. You can select values between 5 GB and 1024 GB.
        Restriction: Use storage classes that provision file storage rather than block storage. If you try to use a storage class that provisions block storage, you might encounter an error when you try to create storage volumes.
        Note: You must have user role with the Create service instances permission in IBM Software Hub to create Storage volumes. If you do not have the permission, the Administrator must create a storage volume and grant you write access permission. To create storage volume and grant access permission, see Creating a storage volume.
    2. Select the Spark runtime version that must be considered for processing the applications.
    3. Select the catalogs that must be associated with the engine from the Associated catalogs(optional) field.
  7. Click Create. An acknowledgment message is displayed.

    Related API: For information on related API, see

    Create Spark engine.