Installing Analytics Engine powered by Apache Spark

Information you need to complete this task

Review the following information before you install Analytics Engine powered by Apache Spark:

Version requirements

All of the components that are associated with an instance of Cloud Pak for Data must be installed at the same release. For example, if the Cloud Pak for Data control plane is installed at Version 4.7.4, you must install Analytics Engine powered by Apache Spark at Version 4.7.4.

Environment variables

The commands in this task use environment variables so that you can run the commands exactly as written.

• If you don't have the script that defines the environment variables, see Setting up installation environment variables.
• To use the environment variables from the script, you must source the environment variables before you run the commands in this task. For example, run:

  source ./cpd_vars.sh

Security context constraint

Analytics Engine powered by Apache Spark works with the default Red Hat® OpenShift® Container Platform security context constraint:

• On Version 4.10, the default SCC is restricted.
• On Version 4.12, the default SCC is restricted-v2

Storage requirements

You don't need to specify storage information when you install Analytics Engine powered by Apache Spark. However, you do need to specify storage when you provision an instance of Analytics Engine powered by Apache Spark.

Before you begin

This task assumes that the following prerequisites are met:

Procedure

Complete the following tasks to install Analytics Engine powered by Apache Spark:

1. Specifying installation options
2. Installing the service
3. Validating the installation
4. What to do next

Analytics Engine powered by Apache Spark parameters

################################################################################
# Analytics Engine powered by Apache Spark parameters
################################################################################

# ------------------------------------------------------------------------------
# Analytics Engine powered by Apache Spark service configuration parameters
# ------------------------------------------------------------------------------
#analyticsengine_spark_adv_enabled: true
#analyticsengine_job_auto_delete_enabled: true
#analyticsengine_kernel_cull_time: 30
#analyticsengine_image_pull_parallelism: "40"
#analyticsengine_image_pull_completions: "20"
#analyticsengine_kernel_cleanup_schedule: "*/30 * * * *"
#analyticsengine_job_cleanup_schedule: "*/30 * * * *"
#analyticsengine_skip_selinux_relabeling: false
#analyticsengine_mount_customizations_from_cchome: false

# ------------------------------------------------------------------------------
# Spark runtime configuration parameters
# ------------------------------------------------------------------------------
#analyticsengine_max_driver_cpu_cores: 5          # The number of CPUs to allocate to the Spark jobs driver. The default is 5.  
#analyticsengine_max_executor_cpu_cores: 5        # The number of CPUs to allocate to the Spark jobs executor. The default is 5.
#analyticsengine_max_driver_memory: "50g"         # The amount of memory, in gigabytes to allocate to the driver. The default is 50g.
#analyticsengine_max_executor_memory: "50g"       # The amount of memory, in gigabytes to allocate to the executor. The default is 50g. 
#analyticsengine_max_num_workers: 50              # The number of workers (also called executors) to allocate to spark jobs. The default is 50.
#analyticsengine_local_dir_scale_factor: 10       # The number that is used to calculate the temporary disk size on Spark nodes. The formula is temp_disk_size = number_of_cpu * local_dir_scale_factor. The default is 10.

Analytics Engine powered by Apache Spark service configuration parameters

The service configuration parameters determine how the Analytics Engine powered by Apache Spark service behaves.

Property	Description
analyticsengine_spark_adv_enabled	Specify whether to display the job UI. Default value true Valid values false Do not display the job UI. true Display the job UI.
analyticsengine_job_auto_delete_enabled	Specify whether to automatically delete jobs after they reach a terminal state, such as FINISHED or FAILED. The default is true. Default value true Valid values true Delete jobs after they reach a terminal state. false Retain jobs after they reach a terminal state.
analyticsengine_kernel_cull_time	The amount of time, in minutes, idle kernels are kept. Default value 30 Valid values An integer greater than 0.
analyticsengine_image_pull_parallelism	The number of pods that are scheduled to pull the Spark image in parallel. For example, if you have 100 nodes in the cluster, set: analyticsengine_image_pull_completions: "100" analyticsengine_image_pull_parallelism: "150" In this example, at least 100 nodes will pull the image successfully with 150 pods pulling the image in parallel. Default value "40" Valid values An integer greater than or equal to 1. Increase this value only if you have a very large cluster and you have sufficient network bandwidth and disk I/O to support more pulls in parallel.
analyticsengine_image_pull_completions	The number of pods that should be completed in order for the image pull job to be completed. For example, if you have 100 nodes in the cluster, set: analyticsengine_image_pull_completions: "100" analyticsengine_image_pull_parallelism: "150" In this example, at least 100 nodes will pull the image successfully with 150 pods pulling the image in parallel. Default value "20" Valid values An integer greater than or equal to 1. Increase this value only if you have a very large cluster and you have sufficient network bandwidth and disk I/O to support more pulls in parallel.
analyticsengine_kernel_cleanup_schedule	Override the analyticsengine_kernel_cull_time setting for the kernel cleanup CronJob. By default, the kernel cleanup CronJob runs every 30 minutes. Default value "*/30 * * * *" Valid values A string that uses the CronJob schedule syntax.
analyticsengine_job_cleanup_schedule	Override the analyticsengine_kernel_cull_time setting for the job cleanup CronJob. By default, the job cleanup CronJob runs every 30 minutes. Default value "*/30 * * * *" Valid values A string that uses the CronJob schedule syntax.
analyticsengine_skip_selinux_relabeling	Specify whether to skip the SELinux relabeling. To use this feature, you must create the required MachineConfig and RuntimeClass definitions. For more information, see Enabling MachineConfig and RuntimeClass definitions for certain properties. Default value false Valid values false Do not skip the SELinux relabeling. true Skip the SELinux relabeling.
analyticsengine_mount_customizations_from_cchome	Specify whether to you want to enable custom drivers. These drivers need to be mounted from the cc-home-pvc directory. Common core services This feature is available only when the Cloud Pak for Data common core services are installed. Default value false Valid values false You do not want to use custom drivers. true You want to enable custom drivers.

Spark runtime configuration parameters

The runtime configuration parameters determine how the Spark runtimes generated by the Analytics Engine powered by Apache Spark service behave.

Property	Description
analyticsengine_max_driver_cpu_cores	The number of CPUs to allocate to the Spark jobs driver. Default value 5 Valid values An integer greater than or equal to 1.
analyticsengine_max_executor_cpu_cores	The number of CPUs to allocate to the Spark jobs executor. Default value 5 Valid values An integer greater than or equal to 1.
analyticsengine_max_driver_memory	The amount of memory, in gigabytes to allocate to the driver. Default value "50g" Valid values An integer greater than or equal to 1.
analyticsengine_max_executor_memory	The amount of memory, in gigabytes to allocate to the executor. Default value "50g" Valid values An integer greater than or equal to 1.
analyticsengine_max_num_worker	The number of workers (also called executors) to allocate to Spark jobs. Default value 50 Valid values An integer greater than or equal to 1.
analyticsengine_local_dir_scale_factor	The number that is used to calculate the temporary disk size on Spark nodes. The formula is: temp_disk_size = number_of_cpu * local_dir_scale_factor Default value 10 Valid values An integer greater than or equal to 1.

Installing the service

To install Analytics Engine powered by Apache Spark:

1. Run the cpd-cli manage login-to-ocp command to log in to the cluster as a user with sufficient permissions to complete this task. For example:

   cpd-cli manage login-to-ocp \
   --username=${OCP_USERNAME} \
   --password=${OCP_PASSWORD} \
   --server=${OCP_URL}

   Tip: The login-to-ocp command takes the same input as the oc login command. Run oc login --help for details.
2. Run the following command to create the required OLM objects for Analytics Engine powered by Apache Spark in the operators project for the instance:

   cpd-cli manage apply-olm \
   --release=${VERSION} \
   --cpd_operator_ns=${PROJECT_CPD_INST_OPERATORS} \
   --components=analyticsengine

   Wait for the cpd-cli to return the following message before you proceed to the next step:

   [SUCCESS]... The apply-olm command ran successfully

   If the apply-olm fails, see Troubleshooting the apply-olm command during installation or upgrade.

3. Create the custom resource for Analytics Engine powered by Apache Spark.

   Run the appropriate command to create the custom resource.

   Default installation (without installation options)

   cpd-cli manage apply-cr \
   --components=analyticsengine \
   --release=${VERSION} \
   --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
   --license_acceptance=true

   Custom installation (with installation options)

   cpd-cli manage apply-cr \
   --components=analyticsengine \
   --release=${VERSION} \
   --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
   --param-file=/tmp/work/install-options.yml \
   --license_acceptance=true

Validating the installation

[SUCCESS]... The apply-cr command ran successfully

If you want to confirm that the custom resource status is Completed, you can run the cpd-cli manage get-cr-status command:

cpd-cli manage get-cr-status \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--components=analyticsengine

What to do next

Complete the following tasks in order before users can access the service:

1. For additional properties that you can specify in the custom resource, see Specifying additional configurations for Analytics Engine powered by Apache Spark.
2. An instance administrator can set the scale of the service to adjust the number of available pods. See Scaling services.
3. Before you can submit Spark jobs by using the Spark jobs API, you must provision a service instance. See Provisioning the service instance.
4. The service is ready to use. See Spark environments.