How to use the RESTful API

Body

IBM Platform Conductor for Spark (Platform Conductor) V1.1 can be configured and maintained through RESTful APIs.

These RESTful APIs are managed by two web servers in your Platform Conductor cluster.

After installing Platform Conductor, you can find the RESTful API reference at the following default URLs:

• Conductor RESTful API: https://<hostname>:8643/platform/rest/conductor/v1

• EGO RESTful API: https://<hostname>:8543/platform/rest/ego

To determine the base URL of the RESTful API, run the following commands:

• egosh client view REST_BASE_URL_1
• egosh client view ASCD_REST_BASE_URL_1

The description in the command results shows the base URL location of the RESTful API.

Client SSL enablement

By default, Platform Conductor is set up to use SSL to transport data for RESTful API calls. The REST client, therefore, must support SSL. Platform Conductor supplies the CA certification in the following install location:

• EGO_TOP/wlp/usr/shared/resources/security/cacert.pem

For example, if you are using a Java client, the certificate must be installed in the JDK keystore:

1. Go to JAVA_HOME/jre/lib/security.
2. Run keytool –import –v –keystore cacerts –storepass changeit –file cacert.pem.

Managing the Spark instance group lifecycle with RESTful APIs

As an example, let’s discuss how a Spark instance group can be managed using the RESTful API.

Prerequisite:

• Install Platform Conductor v1.1 on a server.
• Install cURL on a client.

Procedure:

1. Start Platform Conductor on the server:
   egosh ego start
    
2. Deploy a service package to the top consumer on the server:
   soamdeploy add testPackage -p testPackage.tar -c /
    
3. Authenticate the client to the Platform Conductor server:
   curl –k -u <userid>:<password> https://<hostname>:8643/platform/rest/conductor/v1/auth/logon
   Adding the –k option to the cURL command allows connections to SSL sites without CA certification.
    
4. Register a Spark instance group:
   curl –k -u <userid>:<password> -H "Content-Type:application/json" -H "Accept:application/json" -X POST --data-binary @create_instance_group.json https://<hostname>:8643/platform/rest/conductor/v1/instances

Content of create_instance_group.json:
{     

       "sparkversion": "1.4.1",

       "consumerpath": "/",

       "conductorinstancename": "MyInstance",

       "parameters": {

              "sparkms_batch_rg_param": "ComputeHosts",

              "sparkms_notebook_rg_param": "ComputeHosts",

              "sparkhs_rg_param": "ComputeHosts",

              "driver_rg_param": "ComputeHosts",

              "executor_rg_param": "ComputeHosts",

              "execution_user": "root",

              "deploy_home": "/opt/ibm/2"

       },

       "notebooks": [{

              "rg": "ComputeHosts",

              "name": "Zeppelin",

              "version": "0.5",

              "basedatadir": "",

              "deploydir": "",

              "executionuser": "root",

              "baseport": 8380,

              "consumer": "spark"

       }],

       "dependentpkgs": ["/testPackage"],

       "sparkparameters": {

              "SPARK_MASTER_PORT": 7078,

              "spark.master.rest.port": 6067,

              "SPARK_MASTER_WEBUI_PORT": 8081,

              "spark.shuffle.service.port": 7338,

              "SPARK_EGO_LOGSERVICE_PORT": 28083

              }

}

This command returns a Spark instance group ID, which will be used in the subsequent steps.

5. Deploy the Spark instance group:
   curl –k -u <userid>:<password> -X PUT https://<hostname>:8643/platform/rest/conductor/v1/instances/<spark_instance_group_id>/deploy
    
6. Start the Spark instance group:
   curl –k -u <userid>:<password> -X PUT https://<hostname>:8643/platform/rest/conductor/v1/instances/<spark_instance_group_id>/start
    
7. Stop the Spark instance group:
   curl –k -u <userid>:<password> -X PUT https://<hostname>:8643/platform/rest/conductor/v1/instances/<spark_instance_group_id>/stop
    
8. Unregister the Spark instance group:
   curl –k -u <userid>:<password> -X DELETE https://<hostname>:8643/platform/rest/conductor/v1/instances/<spark_instance_group_id>

You can try out the Spark instance group sample on IBM BlueMix DevOps Services, illustrating how to manage a Spark instance group from the RESTful API.

This is a quick introduction to the Conductor RESTful API. If you have any questions, let us know in the forum! For more information, see the IBM Knowledge Center.