Changing instance resource quota in Analytics Engine Powered by Spark

Working with resource quotas for an instance using the V4 API

To enable using the V4 resource quota APIs, the Cloud Pak for Data Scheduling service must be installed on the cluster and enabled to programmatically enforce the quotas that you set on a service instance.

Required services: The Scheduling service must be installed. You can install the scheduling service along with the Cloud Pak for Data platform. See Installing or upgrading the scheduling service.

Required role: You must be an Openshift administrator or Openshift project administrator to make changes to the Analytics Engine custom resource (CR).

You can enable using the V4 resource quota APIs in provisioned instances in one of two ways:

• Either by using the following patch command:

  oc patch AnalyticsEngine analyticsengine-sample --namespace ${PROJECT_CPD_INST_OPERANDS} --type merge --patch '{"spec":{"serviceConfig":{"schedulerForQuotaAndQueuing":"ibm-cpd-scheduler"}}}'
  

  Wait about 5-10 minutes for the operator to reconcile and reach Completedstate.

• Or by carrying out the following steps:

  1. Log in to the Cloud Pak for Data cluster.
  2. Update the spec.serviceConfig.schedulerForQuotaAndQueuing property in the Analytics Engine CR YAML file that was used to set up Analytics Engine powered by Apache Spark.
  3. Then apply the changes to the existing deployed CR using the following command:

     oc apply -f cr.yaml -n ${PROJECT_CPD_INST_OPERANDS}
     

  4. Wait for the Analytics Engine CR to be in Completed state:

     oc get analyticsengine -n ${PROJECT_CPD_INST_OPERANDS}
     

The following V4 resource quota APIs are available:

• Set resource quota limits
• Get the quota limits that were set
• Get the quota consumption for the instance
• Remove quota limits

curl -X GET \
  "$CPD_URL/v4/analytics_engines/$INSTANCE_ID/resource_consumption_limits" \
  --header "Authorization: ZenApiKey ${TOKEN}"

{
  "max_cores": "50",
  "max_memory": "50G"
}

curl -X GET \
  "$CPD_URL/v4/analytics_engines/$INSTANCE_ID/current_resource_consumption" \
  --header 'Authorization: ZenApiKey ${TOKEN}"

{
  "running": {
    "cores": "2000mi",
    "memory": "2048Gi"
  },
  "pending": {
    "cores": "1000mi",
    "memory": "1024Gi"
  }
}

curl -X DELETE \
  "$CPD_URL/v4/analytics_engines/$INSTANCE_ID/resource_consumption_limits" \
  --header "Authorization: ZenApiKey ${TOKEN}"

Working with resource quotas for an instance using the V3 API (deprecated)

V3 APIs for resource quota is deprecated and it will be removed in upcoming releases.

You can see the total quota for an instance and edit resource quota details if required.

1. From the Spark jobs V3 endpoint, get the instance ID. See Managing Analytics Engine powered by Apache Spark instances. The format of the endpoint is: https://<CloudPakforData_URL>/v2/spark/v3/instances/<INSTANCE_ID>/spark/applications.

2. To edit the resource quota details of an existing instance to 200 CPU and 800 GB for example, use the following cURL command. Insert the instance ID you got from the Spark jobs endpoint:

   curl -iX PUT -k -H "Content-Type: application/json" -H "Authorization: ZenApiKey ${TOKEN}" https://<CloudPakforData_URL>/v2/spark/v3/instances/<INSTANCE_ID>/resource_quota -d '{
           "cpu_quota": 200,
           "memory_quota_gibibytes": 800
   }"
   

3. To get the total CPU and memory quota along with the available resources for that instance, use the following command:

   curl -iX GET -ivk -H "Content-Type: application/json" -H "Authorization: ZenApiKey ${TOKEN}" https://<CloudPakforData_URL>/v2/spark/v3/instances/<INSTANCE_ID> 
   

If you try to submit a job when there aren't enough instance resources available, the job API fails with a 400 Bad Request error indicating that the resources requested by the job exceeded the instance quota.

Updating instance resource quota limits using Cloud Pak for Data user interface

To update the resource quota limits, do the following steps:

1. Log in to the Cloud Pak for Data cluster.
2. From the Navigation menu on the IBM Cloud Pak for Data user interface, select Services > Instances, then select the Analytics Engine powered by Apache Spark instance.
3. In the Instance resource quota details section, click the Edit icon () to update the resource quota details.
4. Select any one of the following options from the Quota managed by field:
   • Analytics Engine powered by Apache Spark: Indicates that the instance resource quota is managed by Analytics Engine powered by Apache Spark. Selecting this option internally invokes V3 API. You must have the pre-requisites for using the V3 resource quota APIs. For more information, see Working with resource quotas for an instance using the V3 API.
   • Scheduling service: Indicates that the instance resource quota is managed by the scheduler. Selecting this option internally invokes V4 API. You must have the pre-requisites for using the V4 resource quota APIs. For more information, see Working with resource quotas for an instance using the V4 API.
5. Increase or decrease the CPU allocation (by using the + or - icon) from the CPU limit field.
6. Increase or decrease the memory allocation (by using the + or - icon) from the Memory limit (GB) field.
7. Click Save. The quota limits are saved.

Parent topic: Administering Analytics Engine powered by Apache Spark