SampleApp: Developing an asynchronous IBM Spectrum Symphony Python client

The purpose of an asynchronous client is to get the output as soon as it is available. The client thread does not need to be blocked once the input data is sent and can perform other actions. In this tutorial, you learn how to convert a synchronous client into asynchronous.

1. Package the sample service.

   To deploy the service, you first need to package it.

   1. Change to the directory where the service is located:

      cd $SOAM_HOME/version/samples/Python/SampleApp/Service

   2. Create and deploy the service package:

      ./makepackage.sh

2. Check the list of deployed services with the soamdeploy command:

   soamdeploy view -c /SampleAppPython

   You should be able to see the package you just deployed.

3. If you are using a Python 3.8.0 or 3.9.0, or 3.10.0 environment, set the PYTHONPATH environment variable on the service side by updating the application profile:
   • On Windows, use a semicolon (;) as the separator, but do not include a semicolon at the end of the PYTHONPATH value as that impacts the execution of soamapiversion.py. For example:

     Updating the application profile for IBM Spectrum Symphony Developer Edition

     <Servicedescription="Python Sample Service"name="SampleServicePython"packageName="SampleServicePython">
             <osTypes>
                 <osType name="all"startCmd="python ${SOAM_DEPLOY_DIR}/MyServiceContainer.py"workDir="${SOAM_HOME}/work">    
                  <env name="PYTHONPATH">${SOAM_DEPLOY_DIR};${SOAM_HOME}/${VERSION_NUM}/${EGO_MACHINE_TYPE}/lib64/pythonapi_3.8.0;${SOAM_HOME}/${VERSION_NUM}/${EGO_MACHINE_TYPE}/lib64</env>
                  <env name="PATH">c:/python3.8.0</env>
             </osType>
     </Service>

     Updating the application profile for IBM Spectrum Symphony

     <Servicedescription="Python Sample Service"name="SampleServicePython"packageName="SampleServicePython">
             <osTypes>
                 <osType name="all"startCmd="python ${SOAM_DEPLOY_DIR}/MyServiceContainer.py"workDir="${SOAM_HOME}/work">    
                  <env name="PYTHONPATH">${SOAM_DEPLOY_DIR};${SOAM_HOME}/${VERSION_NUM}/${EGO_MACHINE_TYPE}/lib64/pythonapi_3.8.0;${SOAM_HOME}/${VERSION_NUM}/${EGO_MACHINE_TYPE}/lib64;${SOAM_HOME}/../ego_version/lib</env>
                  <env name="PATH">c:/python3.8.0</env>
             </osType>
     </Service>

   • On Linux®, use a colon (:) as the separator, but do not include a colon at the end of the PYTHONPATH value as that impacts the execution of soamapiversion.py. For example:

     Updating the application profile for IBM Spectrum Symphony Developer Edition or IBM Spectrum Symphony

     <Service description="Python Sample Service" name="SampleServicePython" packageName="SampleServicePython">
             <osTypes>
                 <osType name="all" startCmd="python ${SOAM_DEPLOY_DIR}/MyServiceContainer.py" workDir="${SOAM_HOME}/work">
     		<env name="PYTHONPATH">${SOAM_DEPLOY_DIR}:${LD_LIBRARY_PATH}:${SOAM_HOME}/${VERSION_NUM}/${EGO_MACHINE_TYPE}/lib64/pythonapi_3.8.0:${SOAM_HOME}/${VERSION_NUM}/${EGO_MACHINE_TYPE}/lib64</env>
                  <env name="PATH">/usr/local/python38</env>
                 </osType>
             </osTypes>
     </Service>

4. Navigate up one directory and register the application with the soamreg command.
   Note: If your Python binary is not in the default location (/usr/local/bin), in the PATH environment variable in the application profile SampleAppPython.xml, provide the directory that contains the Python binary (for example, <env name="PATH">/usr/bin</env>.

   To register the application profile, run:

   cd ../; soamreg SampleAppPython.xml

   The application is registered and enabled.

5. If you are using a Python 3.8.0 or 3.9.0, or 3.10.0 environment, set the PYTHONPATH environment variable on the client side:
   • On Windows, use a semicolon (;) as the separator, but do not include a semicolon at the end of the PYTHONPATH value as that impacts the execution of soamapiversion.py. For example:

     Setting the environment variable for IBM Spectrum Symphony Developer Edition to run as the client

     set PYTHONPATH=%SOAM_HOME%\soam_version\w2k3_x64-vc7-psdk\lib64\pythonapi_3.8.0;%SOAM_HOME%\soam_version\w2k3_x64-vc7-psdk\lib64;%SOAM_HOME%\soam_version\w2k3_x64-vc7-psdk\ego_lib

     Setting the environment variable for IBM Spectrum Symphony to run as the client

     set PYTHONPATH=%SOAM_HOME%\soam_version\w2k3_x64-vc7-psdk\lib64\pythonapi_3.8.0;%SOAM_HOME%\soam_version\w2k3_x64-vc7-psdk\lib64;%SOAM_HOME%\..\ego_version\lib

     Setting the environment variable for the IBM Spectrum Symphony client to run workload

     set PYTHONPATH=%SOAM_HOME%\lib64\pythonapi_3.8.0;%SOAM_HOME%\lib64

   • On Linux, use a colon (:) as the separator, but do not include a colon at the end of the PYTHONPATH value as that impacts the execution of soamapiversion.py. For example:

     Setting the environment variable for IBM Spectrum Symphony Developer Edition to run as the client

     export PYTHONPATH=$SOAM_HOME/soam_version/linux-x86_64/lib64/pythonapi_3.8.0:$SOAM_HOME/soam_version/linux-x86_64/lib64:$SOAM_HOME/soam_version/linux-x86_64/ego_lib64

     Setting the environment variable for IBM Spectrum Symphony to run as the client

     export PYTHONPATH=$SOAM_HOME/soam_version/linux-x86_64/lib64/pythonapi_3.8.0:$SOAM_HOME/soam_version/linux-x86_64/lib64:${SOAM_HOME}/../ego_version/linux-x86_64/lib

     Setting the environment variable for the IBM Spectrum Symphony client to run workload

     export PYTHONPATH=$SOAM_HOME/lib64/pythonapi_3.8.0:$SOAM_HOME/lib64

6. Run the sample client and service.

   To run the service, you run the client application. The service a client application uses is specified in the application profile.

   1. Change to the Python sample SampleApp directory:

      cd $SOAM_HOME/version/samples/Python/SampleApp/

   2. Source the environment:
      • (csh) source cshrc.sampleapp
      • (bash) . profile.sampleapp
   3. Change to AsyncClient directory and run the application:

      cd AsyncClient/
      python AsyncClient.py

      You should see output on the command line as work is submitted to the system. The client starts and the system starts the corresponding service. The client displays messages indicating that it is running.

7. Walk through the code.

   You review the sample client application code to learn how you can understand the differences between a synchronous client and an asynchronous client.

   1. Locate the code samples:

      Files	Location of code sample
      Client	$SOAM_HOME/version/samples/Python/SampleApp/AsyncClient
      Message object	$SOAM_HOME/version/samples/Python/SampleApp/Common
      Service code	$SOAM_HOME/version/samples/Python/SampleApp/Service
      Application profile	The service required to compute the input data along with additional application parameters are defined in the application profile: $SOAM_HOME/version/samples/Python/SampleApp/SampleAppPython.xml

   2. Understand what the sample does.

      The client application sample sends ten input messages with the data Hello Grid!! and retrieves the results.

      Results are returned asynchronously with a callback interface provided by the client to the API. Methods on this interface are called from threads within the API when certain events occur. In the sample, the events are as follows:

      • When there is an error at the session level
      • When results return from IBM Spectrum Symphony
   3. Note considerations for asynchronous clients:

      Synchronization

      Because results can come back at any time, it is probable that your callback code needs synchronization between the callback thread and the controlling thread. The controlling thread needs to know when work is complete.

      Order of results

      Results are not sent back in order. If order of results is important, the client application must sort the results.

   4. Note code differences between synchronous and asynchronous clients.
      An asynchronous client is very similar to a synchronous client. The only differences are as follows:

      • You need to specify a callback when creating a session
      • You specify a different flag to indicate asynchronous when you create a session
      • Retrieval of replies

      Let us look at the steps to create synchronous and asynchronous clients and highlight the differences. Steps in bold indicate differences. Everything else is the same as the synchronous client.

      
   5. Declare the message object and implement.

      As in the synchronous client tutorial, declare the message object and implement your own message object.

      If you have not done so already, take a look at the synchronous client application tutorial Synchronous IBM Spectrum Symphony Python client and service tutorial for details on the Message object, specifically:

      • Input and output: declare the message object
      • Implement the MyMessage object
   6. Declare and implement your callback object.

      Perform this step after declaring the Message object and implementing the MyMessage object.

      In MySessionCallback.py, we create our own callback class from the SessionCallback class, and we implemented on_response() to retrieve the output for each input message that we send.

      Note:

      • on_response() is called every time a task completes and output is returned to the client. The task output handle allows the client code to manipulate the output.
      • is_successful() checks whether there is output to retrieve.
      • If there is output to retrieve, populate_task_output() gets the output. Once results return, we print them to standard output and return.

      
      import soamapi
      import MyMessage
      import threading
      import pickle
      
      class MySessionCallback(soamapi.SessionCallback):
      
          def __init__(self, tasks_to_receive):
              self.lock = threading.Lock()
              self.exception_occurred = False
              self.received_tasks = 0
              self.tasks_to_receive = tasks_to_receive
      
          '''
          This handler is called once a message is returned
          from the system when a task completes.
          '''
          def on_response(self, task_output_handle):
              print('onres')
              if task_output_handle.is_successful():
                  out_msg = MyMessage.MyMessage()
                  task_output_handle.populate_task_output(out_msg)
                  
                  # Display content of reply
                  prt_msg = "Task Succeeded [" + task_output_handle.get_id() + "]\n"
                  prt_msg += "Integer Value : " + str(out_msg.get_int()) + "\n"
                  prt_msg += out_msg.get_string()
                  print(prt_msg)
              else:
                  ex = task_output_handle.get_exception()
                  prt_msg = "Task Not Succeeded : " + str(ex)
                  print(prt_msg)
              task_output_handle = None
              self.lock.acquire()
              try:
                  self.received_tasks = self.received_tasks + 1
              finally:
                  self.lock.release()
      
          '''
          This handler is called once any exception occurs
          within the scope of the session.
          '''
          def on_exception(self, exception):
              print(exception)
              self.lock.acquire()
              try:
                  self.exception_occurred = True
              finally:
                  self.lock.release()
      
          def isComplete(self):
              self.lock.acquire()
              try:
                  if self.received_tasks == self.tasks_to_receive:
                      return True
                  elif self.exception_occurred:
                      return True
                  else:
                      return False
              finally:
                  self.lock.release()   

   7. Create a session to group tasks.

      In AsyncClient.py, perform this step after you have connected to the application.

      When creating an asynchronous session, you need to specify the session attributes by using the SessionCreationAttributes object. In this sample, we create a SessionCreationAttributes object called session_attr and set four parameters in the object.

      The first parameter is the session type. The session type is optional. You can leave this parameter blank or not make the API call at all. When you do this, system default values are used for your session.

      The second parameter is the session name. This is optional. The session name can be any descriptive name you want to assign to your session. It is for information purposes, such as in the command line interface.

      The third parameter is the session flag, which we specify as SessionFlags.PARTIAL_ASYNC. You must specify it as shown. This indicates to Symphony that this is an asynchronous session.

      The fourth parameter is the callback object.

      We pass the attributes object to the create_session() method, which returns a pointer to the session:

      
      session_attr = soamapi.SessionCreationAttributes()
      session_attr.set_session_type("RecoverableAllHistoricalData")
      session_attr.set_session_name("MySessionName")
      session_attr.set_session_flags(soamapi.SessionFlags.PARTIAL_ASYNC)
      session_attr.set_session_callback(my_callback)
      session = connection.create_session(session_attr)
      prt_msg = "Created session:" + session.get_id()
      print(prt_msg)

   8. Synchronize the controlling and callback threads.

      Perform this step after sending the input data to be processed.

      Since our client is asynchronous, we need to synchronize the controlling thread and the callback thread. In this example, the controlling thread blocks until all replies have come back.:

      while (my_callback.isComplete() == False):
          time.sleep(1)
      print("Finished receiving all task results")