Shared services

A shared service is a predefined pattern that is deployed and shared by multiple client application deployments, such as virtual applications and virtual systems, in the cloud. A shared service provides certain runtime services to multiple applications or services to the user on behalf of multiple applications.

You usually find only one single reference to the shared service per cloud group, which is a physical group of hardware that defines a cloud. That shared service can be used by all application deployments in the cloud group, which means the shared service must provide multitenant access.

Shared service infrastructure

The infrastructure provides a common administration framework and registry services. A single reference to a shared service is allowed in each cloud group. UI panels enable deployment and management of shared services. You can create two types of shared services, one with the virtual machines created in the cloud group and the other, which is an external reference to a service outside the cloud group.

A registry contains information about deployed shared services in each cloud group. The client of a shared service can query the registry for information about the shared service it wants to interact with. The shared service implementation determines what information the client requires and also what client API versions it can support.

Creating a shared service implementation

A shared service is developed similar to plug-in development for virtual applications. There is extra metadata and capabilities in some scenarios. The following list outlines the steps to follow when you develop a service.

Create the predefined application model and property metadata.
The appmodel.json file represents the serialization of the model that is defined in the Pattern Builder user interface for a regular virtual application. Components (nodes) and links, along with user-specified property values, are represented. For shared services, the properties must be predefined. In addition to nodes, links and other attributes of an application model, the following attributes are required for a shared service:
app_type

Set to "service".

serviceversion

Unique shared service application model version in VRMF format. For example, 1.0.0.0.

servicesupportedclients
List of supported client versions that can use this shared service.
Example patterns are:

*: matches all versions

[a,b]: matches all versions between a and b, including a and b

(a,b): matches all versions between a and b, excluding a and b

[*,b]: matches all versions up to b, including b

[a,*]: matches all versions a and greater
servicedisplayname

Display name for similarly grouped services.

servicename

Name of the service. Used as the name of the service registry documents.

id

Set to "sharedservice". This attribute is an attribute on a node.

type

Unique ID that links the transformer and appmodel metadata to this node.

servicetype

Set to External in the appmodel.json to identify a service as an external shared service.
The following example shows the use of these attributes:
```
{
   "model":{
		"name":"Shared Service",
		"app_type":"service",
		"patterntype":"foundation",
		"version":"1.0",
		"serviceversion":"1.0.0.0",
		"servicesupportedclients":"[0.0,1.0]",
		"servicedisplayname":"servicegroup",
		"servicename":"service",
		"description":"comments",
		"nodes":[{
			 "attributes":{...},
			"id":"sharedservice",
			"type":"uniqueComponentOSGIID"
		}],
		"links":[]
	}
}
```
The appmodel/metadata.json file describes the components, links, and policies that are implemented by the plug-in. Shared services use the same attribute design. Default attributes can be set by setting the specific attribute inside the predefined appmodel or by using the sampleValue field inside the metadata.json.
Define a registry provider class.
The shared service registry contains information that clients can look up to find information that is shared by the service about itself. The infrastructure provides this ability through the shared service specific implementation of the com.ibm.maestro.iaas.RegistryProvider class. The following method allows the shared service to return information to the client based on its model and deployment configuration.

public JSONArtifact getRegistry(String ClientVersion, Map<String, JSONObject> deploymentInfo throws HttpException;

deploymentInfo contains the appmodel (appmodel.json), deployment (deployment.json), topology (topology.json) and registry (registry.json) documents.
Create the topology template for the appmodel.
Transformers are services that convert the application model from a logical description into a topology document that is used to deploy the virtual application. Shared services (like other plug-ins) can define a template for a topology document and transformers that convert the template to an actual topology document during deployment. The following attribute must be provided by a shared service to reference the shared service registry provider class:
```
 "service-registry": [{
        "type": "<RegistryProvider implementation>"
    
}],
```
Do not include the vm-templates attribute section for an external shared service topology template since it is pointing to an external resource implementation of the shared service.
Create the shared service lifecycle scripts.
For more information about developing lifecycle scripts, see the Developing lifestyle scripts section of the plug-in development guide. When you develop lifecycle scripts for shared services, consider functions such as being able to stand up and recover from failures, providing administrative operations for the service, scalability, and other similar functions.
Optional. Make a shared service public certificate available for client access.
The infrastructure provides a central location for certificates that a shared service must make available securely to deployments that act as clients to it. The following com.ibm.maestro.iaas.RegistryService APIs can be called by the shared service to manage its certificates:

public void putCertificate(RestClient restClient, String putObj, String cloudGroup, String sharedServiceName, String sharedServiceVersion) throws CredentialExpiredException, MaestroSecurityException, HttpException;

public void deleteCertificate(RestClient restClient, String cloudGroup, String sharedServiceName, String sharedServiceVersion) throws CredentialExpiredException, MaestroSecurityException, HttpException;
Expose administrative HTTP calls for the clients to interact with the service.
Each shared service must expose an HTTP administrative interface for the clients to easily register and interact with the service (reserve resources on service). Then, the usage of the service can be custom to that service (by using the reserved resources over non-HTTP interaction). This step is where the client version helps determine the client contract for the administrative HTTP interaction before it uses the service.

The Shared Service Infrastructure framework provides a helper feature to easily create the http administrative interface if the shared service would like to use the framework more. For more information, see the section "Generic Shared Service REST API support and client interaction".

Shared service client development

Application deployments can enable a shared service consumer model:

By directly exposing a resource through the virtual application modeler (An example is the routing policy that enables a deployment to be an On Demand Router based Load Balancer shared service client).
Indirectly by using a setting that implies usage of a shared service. Scaling policy implies HTTP session caching and enables a deployment to be a client to the Caching shared service.
By using the appmodel transformation process that injects the client. This process is done by adding the client plug-in packages, parameters, service references, and other elements, while the process creates the topology document.

The infrastructure supports predeployment service reference checks that are done on behalf of the shared service client and can stop the deployment from creating resources in cloud.

The client can indicate the services that it references and if the availability of the service determines whether the deployment proceeds. The following tag in the client topology document indicates that this client requires the shared service it references to support a client version of 3.0 to be available in the cloud group it is being deployed to. If not, the client deployment is to be failed.
```
   "service-templates": [
		{
			"sharedservice": {
				"client-version": "3.0", 
				"name": "shared service name>", 
				"required": true
			}
		}
    ]
```
The client can define a ServiceProvisioner to provision required resources. It can interact with the shared service by retrieving its registry information and providing input to the topology document for later use. The deployment can be failed at this stage as well by the client. The client obtains information about the shared service by calling the com.ibm.maestro.iaas.RegistryService interface getRegistry method: public JSONArtifact getRegistry(RestClient restClient, String sharedServiceName, String clientCloudGroup, String clientVersion, String clientDeploymentId) throws HttpException;
For information about ServiceProvisioner, see the Javadoc documentation in the Plug-in Development Kit.

Post deployment, when the client plug-in packages are extracted on the deployed virtual machines, it can interact with the shared service through its lifecycle scripts. The infrastructure provides a maestro.registry python library for registry and certificate-related method calls from lifecycle scripts.

import maestro
parms=maestro.registry.getRegistry(shared_service_name, shared_service_client_version)
masterIP = parms['<registry parm name>']

The removeRegistryRef call can be used when the client no longer wants to be connected to the shared service, or when the client is about to be deleted:

import maestro
maestro.registry.removeRegistryRef(shared_service_name, shared_service_client_version)

If the shared service exposed a public certificate in the shared service framework, then the client can obtain that certificate with the following command:

import maestro 
maestro.registry.getCertificate(shared_service_name, shared_service_client_version, temp_cert_file)

The following example shows how to use the preceding commands in a lifecycle script. The script displays the IP address of a deployed Caching shared service, downloads the public certificate, and provides a signal to the service that the script is finished running.

import maestro

service_name = "caching"
client_version = "2.0"

# get registry information
registry_info = maestro.registry.getRegistry(service_name, client_version)
ip_address = registry_info['cache-ip']

print "Caching service found at " + ip_address


# this is where the certificate will be stored
certificate_location = "/tmp/caching_certificate"

# download certificate
maestro.registry.getCertificate(service_name, client_version, certificate_location)

# signal the service that we have finished interacting with it
maestro.registry.removeRegistryRef(service_name, client_version)

Shared services providers must clearly document the client interaction model that is based on capabilities the service provides. Document this interaction by using the client version, which becomes a client contract that defines exactly what to expect for that version. Different shared service versions might return different information through the registry to the client. The client service provisioner and lifecycle scripts must understand the right way to interact with the service based on the registry response. For example, consider a shared service version 1.0 returning a host, user name, and password attributes through a getRegistry call to its client of Version 1.0. If a port must be added as well to the response, the service can indicate that it will for clients of Version 2.0, which are written to handle the additional attribute. In its next revision, the service can then support both clients of Version 1.0 and 2.0 providing each with expected responses.

The version of the client also helps the shared service infrastructure determine which shared service versions the client can interact with in a cloud group. For example, if shared service Version 1.0 supports only client Version 1.0, then the infrastructure does not return a registry object to a client of Version 2.0 requesting the registry.

Shared service client tracking and operation invocation

Certain shared services might require the capability to be able to track clients that connect to it. This capability might be needed so that the shared service can communicate its start or delete lifecycle events to the client. The client might choose to stop requesting information from the shared service when the client is deleted, for example. This ability is enabled by including the following metadata in the shared service vm template.

"service-registry": [{
        "type": "<RegistryProvider implementation>"
        "trackclientusage":true
        
}],

If this metadata is specified, the infrastructure maintains a list of clients that request registry information from the service. The infrastructure also provides the capability for the service to call operations on clients that are tracked to be using it. The operation is defined on the client: public void callOperationOnSSClients(RestClient restClient, JSONObject serviceInfo, JSONObject operationParms) throws HttpException;

This example shows you how to call the operation from the shared service ServiceProvisioner

public StateAndDescription deleteService(String serviceReferenceName,
			JSONObject serviceDescription, RestClient restClient)
			throws Exception {
        final String METHOD_NAME = "deleteService";
        if (logger.isLoggable(Level.FINE)) {
            logger.logp(Level.FINE, CLASS_NAME, METHOD_NAME, "deleteService: " + serviceReferenceName + "/" + serviceDescription);
            logger.logp(Level.FINE, CLASS_NAME, METHOD_NAME, "Calling the disconnect operation on shared service clients");
        }
        
        JSONObject operationParms = new JSONObject();
        operationParms.put("role","<role_name>");
        operationParms.put("type", "<type_name>");
        operationParms.put("script", "<scriptname>");
        operationParms.put("method", "<method_name>");
        operationParms.put("parameters", new JSONObject());
        this.registrySvc.callOperationOnSSClients(restClient, serviceDescription, operationParms);

Generic Shared Service REST API support and client interaction

The Generic Shared Service REST infrastructure helps provide a common HTTP-based interaction model for clients to start methods that are exposed by shared services. Shared services can provide operation metadata and python scripts that implement the method. The client can call the generic shared services REST APIs to start the operations. For example, a client might call GET on the REST URL: https://<Master_Agent_VM>:9999/sharedservice/<servicename>/<resource>

Follow these steps to configure the server-side metadata and setup.

Service metadata must be provided in a JSON file and packaged in the shared service nodepart data. For example: plugin/nodeparts/cachingrestapi/cachingrestapi/properties/cachingrestapi.json.
The JSON object in the metadata must contain the following attributes:

servicename

The name of the shared service: "caching".

operations

JSON array of operations that is exposed by the service. Each object in this array defines the following attributes:

type

HTTP operation type: GET, PUT, POST, or DELETE.

parms

JSON array of objects that represents parameters for each call of the type. The object must define the following attributes:

resource

Resource on which operation is carried out. Maps to URL segment after the servicename on the REST URL.

role

The role this operation is to run against.

script

<script_name> <method_name>. The Python script that defines the operation and the method that starts.

clientdeploymentexposure

Optional. Determines whether the call can originate from client deployed virtual machines in addition to from the KernelServices process. The default value is false.

timeOut

Optional. Operation timeout. The default value is 60000. If this attribute is 0, The operation runs synchronously with no timeout. If this attribute is greater than 0, the operation waits the specified amount of time, in milliseconds, for this call to return. The operation responds with an HTTP 202 response if a timeout occurs.

pattern

This attribute can be used to match further segments of the URL to feed further parms into the operation. An entry that is wrapped in curly braces matches as a name-value pair against the incoming URL and passes into the operation as a parms. For example, consider a REST URL of the form: https://<host>:<port>/sharedservices/caching/sessiongrid/gridA and a pattern {gridName}. The name and value pair gridName=gridA feeds into the operation as an extra argument.
Here is an example metadata json:
```
{
	"servicename": "caching", 
	"operations": [
		{
			"type": "PUT", 
			"parms": [
				{
					"resource": "sessionGrid", 
					"role": "Caching-Master.Caching", 
					"script": "restapi.py createSession",
					"timeout": 1200000
				},
				{
					"resource": "simpleGrid", 
					"role": "Caching-Master.Caching", 
					"script": "restapi.py createSimple"
				},
				{
					"resource": "dynamicGrid", 
					"role": "Caching-Master.Caching", 
					"script": "restapi.py createDynamic"
				}
			]
		}, 
		{
			"type":"DELETE",
			"parms":[
				{
					"resource": "sessionGrid", 
					"role": "Caching-Master.Caching", 
					"script": "restapi.py deleteSession",
					"pattern": "{gridname}"
				}, 
				{
					"resource": "simpleGrid", 
					"role": "Caching-Master.Caching", 
					"script": "restapi.py deleteSimple",
					"pattern": "{gridname}"
				},
				{
					"resource": "dynamicGrid", 
					"role": "Caching-Master.Caching", 
					"script": "restapi.py deleteDynamic",
					"pattern": "{gridname}"
				}
			]
		},
		{
			"type":"GET",
			"parms":[
				{
					"resource": "sessionGrid", 
					"clientdeploymentexposure":true,
					"role": "Caching-Master.Caching", 
					"script": "restapi.py gridExists",
					"pattern": "{gridname}"
				},
				{
					"resource": "publicCert", 
					"clientdeploymentexposure":true,
					"role": "Caching-Master.Caching", 
					"script": "restapi.py downloadCert",
					"responseType": "zipFile"
				}
			]
		}
	]
}
```
During virtual machine start of the shared service deployment, the metadata must be copied to the following location on the virtual machine: /0config/sharedservices/restmetadata/. This metadata can be copied by providing a nodepart installation script that runs during the deployment. For example: /plugin/nodeparts/cachingrestapi/common/install/7_cachingrestapi.py that contains
```
restapidir = '/0config/sharedservices/restmetadata'
restapifile = '/cachingrestapi.json'
restapi = '../../cachingrestapi/properties/cachingrestapi.json'

os.popen('mkdir -p ' + restapidir)
os.popen("mv " + restapi+ " "+ restapidir + restapifile)
```

Provide the implementation of the methods in a python script. Define operations that run based on parms from the different sources, input JSON object to the REST call or URL pattern matching. For example: parts/caching.scripts/scripts/CachingMaster/restapi.py. The client can call the APIs through the following convenience methods:

If you call operations on the service from within the KernelServices process (through a Service Provisioner), use the following method that is defined on com.ibm.maestro.iaas.RegistryService

public OperationResponse callOperationOnSharedService(String serviceName, String clientCloudGroup, String clientVersion, 
String serviceIP, String resourceUrl, String operationType, JSONObject operationParms) throws HttpException;

For example:

JSONObject clientJSON = new JSONObject();
    	clientJSON.put("user", user);
    	clientJSON.put("password", password);
    	clientJSON.put("gridname", grid);
    	clientJSON.put("gridcap", cachingCap);

      OperationResponse response = this.registrySvc.callOperationOnSharedService("caching", (String)serviceDescription.get("cloud_group"), "3.0", ip,
      "sessionGrid", "PUT", clientJSON);
      int status = response.getStatusCode();
      JSONObject result = response.getOperationResponse();

For virtual application deployments, the shared services infrastructure provides a utility to interact with shared services. This utility can be accessed within lifecycle scripts.
The utility script provides only one method, callSSrestapi: sharedservices.callSSrestapi(url, method, data=None, filepath_input=None, filepath_output=None). This method makes a REST API call to the specified URL with the specified method. This function returns the HTTP status code of the call and the returned JSON document, if one was provided.

The caller can set the filepath_input parameter to store the returned file at the specified location (assuming the directory exists). This parameter is expected if the shared service returns a compressed file from this call. Note, if filepath_input is not set, any response from the Shared Service will be deleted after it parsed.

If the caller must send a file to the shared service, the filepath_output parameter can be set to the location of this file, but the method argument must be PUT or POST to send the file. If PUT or POST methods are used, either the data or filepath_output argument is expected.
To access this function, you must include the following code in the lifecycle script:
```
import sys
ss_path = '/0config/nodepkgs/helper/scripts'
if not ss_path in sys.path:
    sys.path.append(ss_path)
import sharedservices
```
This example script uses the utility functions to interact with the caching shared service. It retrieves the public certificate and checks whether a named caching grid exists.
```
import sys

ss_path = '/0config/nodepkgs/helper/scripts'
if not ss_path in sys.path:
    sys.path.append(ss_path)
import sharedservices

service = "caching"
version = "2.0"

regInfo = maestro.registry.getRegistry(service, version)
try:
    ipAddr = regInfo['cache-ip']
    print "caching service found at " + ipAddr
except KeyError:
    print "caching service not found"
    sys.exit(1)


cert_zip_location = "cert.zip"

# download zip (that contains the certificate)
cert_response = sharedservices.callSSrestapi("https://" + ipAddr + ":9999/sharedservice/caching/publicCert", 
"GET", filepath_input=cert_zip_location)


# check if a grid exists
grid_name = "SomeGrid"

http_code, grid_response = sharedservices.callSSrestapi("https://" + ipAddr + ":9999/sharedservice/caching/gridExists/", "GET")
ret =  grid_response['OperationResults'][0]['return_value']

if (ret.find(grid_name + ' does not exist') >= 0):
    print grid_name + " does not exist"
else:
    print grid_name + " exists"

# inform the caching service that we have finished interacting with it
maestro.registry.removeRegistryRef(service, version)
```

Manual testing of the API is possible through the following on the deployed virtual machine:

cd /0config  
export $(./get_userdata.sh)  
export header=$(/opt/python-2.6.4/bin/python create_security_header.py)
curl -H "X-IWD-Authorization : $header" -kv -H Content-Type:application/json -X PUT -d '{"input":"hi"}' 
https://<shared service IP:9999/sharedservice/<servicename>/<resourcename>

Virtual system client interaction with shared services

The shared services infrastructure also provides support for virtual system deployments to be shared services clients. The following libraries and APIs can be called through script packages on virtual systems to query for the shared service registry, start REST APIs exposed by the shared service, and other functions.

The shared services plug-in for virtual systems requires that maestro is installed on a deployed virtual system. By default, maestro is not enabled for such deployments. The steps for enabling maestro are:

Click Catalog > System Plug-ins. from the dashboard.
Select the Foundation pattern type.
Select the virtualsystem plug-in.
Click configure.
Set the Plug-ins for virtual systems to enabled.

After you follow these steps, any deployed virtual system patterns install maestro and include the shared services plug-in for virtual systems.

The shared services client is a node package that attempts to mimic the virtual application lifecycle scripts in function. However, there are a few significant differences in their operation and how they are called.

To interact with shared services, you must import the shared service client script:

import sys
ss_path = '/0config/nodepkgs/helper/scripts'
if not ss_path in sys.path:
    sys.path.append(ss_path)
import sharedserviceclient

After you import this script, it can access to the following functions:

sharedserviceclient.getRegistry(service_name, client_version): This function returns registry information of the named shared service, if the client version is supported. Unlike maestro.registry.getRegistry, this function returns the response as a text string.
sharedserviceclient.removeRegistryRef(service_name, client_version): This function is called when a client finishes interacting with the named shared service, which signals the service to clean up any obsolete resources. If this operation is successful, the return value is 0.
sharedserviceclient.restapi(url, method, data=None): This function calls a REST API method, by using curl, to interact with the shared service at the provided URL. Accepted methods are GET, DELETE, PUT, and POST. If the method is PUT or POST the data parameter is required.
The data argument must be a text string. Similarly, this function returns the response from the service as a text string.

The following example shows how to interact with the caching shared service to see whether a named data grid exists:

import sys
ss_path = '/0config/nodepkgs/helper/scripts'
if not ss_path in sys.path:
    sys.path.append(ss_path)
import sharedserviceclient


def find_cache_IP(reg_info):
    """
Finds the Caching Shared Service IP address in the registry information.
    """
    str_info = str(reg_info)
    
    start = 'cache-ip":"'
    end = '",'
    
    start_cip = str_info.find(start)
    end_cip = str_info.find(end, start_cip)
    
    return str_info[start_cip+len(start):end_cip]

def does_grid_exist(grid_name, caching_ip):
    """
Queries the Caching Shared Service to see if a grid exists.
Returns '1' if the grid exists, '0' otherwise.
    """
    str_reply = str(sharedserviceclient.restapi("https://" + cip + ":9999/sharedservice/caching/gridExists/" + grid_name, "GET", ""))
    
    if (str_reply.find(grid_name + ' does not exist') >= 0):
        return 0
    else:
        return 1


# figure out the caching service's IP address
reg = sharedserviceclient.getRegistry("caching", "2.0")
cip = find_cache_IP(reg)

# grid to read data from
grid_name = "WAS_GRID"

# ensure the grid exists
if (does_grid_exist(grid_name, cip)):
    print "Reading data from grid..."
    # read data from it
else:
    print "Grid '" + grid_name + "' doesn't exist"
    # ask somebody to create it

# done with caching server
sharedserviceclient.removeRegistryRef("caching", "2.0")