Managing product binaries in virtual application pattern workloads

Virtual application pattern is a cloud workload deployment model used in IBM® PureApplication™ System, IBM SmartCloud™ Orchestrator, and IBM Workload Deployer. While developing your virtual application pattern plug-ins, you will encounter situations where you need to manage and install product binaries. In this article, you will learn various techniques to manage them.

Share:

Bhadri Madapusi (bhadrim@ca.ibm.com), Advisory Software Developer , IBM

Photo of Bhadri MadapusiBhadri Madapusi is a Software Developer and currently focused on IBM PureApplication System pattern-type development. Bhadri has over 10 years of software development experience. Previously, he worked on cloud computing, Business Process Management, and WebSphere Commerce.



Valentina Birsan (popescu@ca.ibm.com), Senior Software Developer , IBM

Photo of Valentina BirsanValentina Birsan is a Senior Developer in WebSphere and currently focuses on IBM PureApplication System pattern-type development. Previously, Valentina was a technical lead on Rational Application Developer. She was one of the initial members of the Eclipse TPTP open source project, and served as the chair of the TPTP Architecture Group. Valentina was also the lead architect for the Cosmos Service Modeling Eclipse open source project and a member of the Service Management Language open standard.



29 January 2014

Also available in Chinese Japanese

Introduction

When you develop your virtual application pattern workload to be installed on IBM PureApplication System, IBM SmartCloud Orchestrator, or IBM Workload Deployer, you will encounter situations where your plug-in needs to manage product binaries. The product binaries may be yours or from a third party vendor such as IBM, SAP®, or Oracle®. In this article, you will see different strategies that you can follow to manage the product binaries and how your plug-in can access and install the binaries. You will also understand the relative advantages and disadvantages of these different strategies.


Strategies to manage product binaries

The strategies to manage product binaries are classified into the following categories:

  • Using the Storehouse repository
  • Using the shared service repository server

Using the Storehouse repository

The Storehouse is a repository that contains the system artifacts, virtual application pattern workload artifacts, virtual application pattern deployment runtime artifacts, and virtual application pattern troubleshooting artifacts. You can view the artifacts that are stored in the Storehouse repository using the Storehouse Browser.

In IBM Workload Deployer and IBM PureApplication System, you can get to the Storehouse Browser by navigating to Workload console > System > Storehouse Browser. In IBM SmartCloud Orchestrator, you can get to the Storehouse Browser by navigating to SmartCloud Orchestrator console > Administration > Storehouse Browser. Only the users who have access to view the System menu (or Administration menu) can view the Storehouse Browser. Figure 1 illustrates the typical content of the Storehouse Browser.

Figure 1. Storehouse Browser
Storehouse Browser

Click to see larger image

Figure 1. Storehouse Browser

Storehouse Browser

You can use the Storehouse repository to store your product binaries. During workload deployment, your virtual application pattern can download the binary files from the Storehouse, and install and configure them on the virtual machine. In order to upload your product binaries to the Storehouse, you can follow one of the following techniques:

  • Use cURL to upload the binary.
  • Use your pattern-type to upload the binary.
  • Use a separate binary upload pattern-type to upload the binary.

Each of the above techniques has its advantages and limitations. In the following sections, you will learn how to use the above three techniques. We will illustrate the above techniques by uploading the binary file, rhel_rpm.tgz, into the following Storehouse repository location: /admin/files/Sample/Common/.

Use cURL to upload the binary

In this technique, you will use the cURL utility to upload a binary file on to the Storehouse repository as illustrated in Listing 1. The code in Listing 1 uploads the file into the location as illustrated in Figure 2.

Figure 2. File location in the Storehouse browser
File location in the Storehouse browser

Click to see larger image

Figure 2. File location in the Storehouse browser

File location in the Storehouse browser

The advantages of this technique are:

  • It is relatively simple compared to the other two techniques.
  • It does not need any additional artifacts or code to upload the binary.

The limitations of this technique are:

  • Some flavors of cURL have a limitation on the size of the uploaded file.
  • This method may not work on PureApplication System.

Further, you need administrator rights to upload the files to the repository.

Use cURL to upload the product binary
#Upload file to storehouse under /admin/files/Sample/Common.
curl -v -u <user>:<password> -k -X PUT --data-binary @rhel_rpm.tar.tgz -H "Content-Type:
application/octet-stream" https://<server_name>/storehouse/admin/files/Sample/Common/ rhel_rpm.tgz

#Give full access to the current user and group Everyone to download the file.
curl -v -i -u <user>:<password> -k -X POST -H "Content-Type: application/json" 
-X POST -d "{\"AccessRights\":{\"$USER\":\"F\",\"_group_:Everyone\":\"
F\"}}" https://<server_name>/storehouse/admin/files/Sample/Common/rhel_rpm.tgz?meta

If you use this technique on PureApplication System, the file is uploaded to the Storehouse, but you may not be able to download it in your virtual application pattern, as the group access control (as illustrated in the second curl command above) cannot be set to "Everyone".

Use your pattern-type to upload the binary

In this technique, you will package the binary file with the pattern-type that contains your virtual application pattern plug-ins. If you open a built pattern type using any archive extraction tool, it looks as shown in Figure 3. The archive is composed of a patterntypes folder, which contains a pattern type project in archive format, and a plugins folder, which contains a plug-in project in archive format. When you install this pattern type into Workload Deployer, PureApplication System, or SmartCloud Orchestrator, the contents of the patterntypes folder is stored in the Storehouse repository under "/admin/patterntypes", and the contents of the plugins folder is stored in the Storehouse repository under "/admin/plugins".

Figure 3. Expanded built pattern type archive file
Expanded built pattern type archive file

Now that you know how the Storehouse repository works, in order to have a binary file rhel_rpm.tgz under /admin/files/Sample/Common, you need to have an additional folder named files/Sample/Common with the file rhel_rpm.tgz in the built pattern type archive, as illustrated in Figure 4. When you install this modified pattern type, the binary gets stored in the Storehouse repository under /admin/files/Sample/Common as shown in Figure 2.

Figure 4. Expanded modified pattern type archive file with binary
Expanded modified pattern type archive file with binary

To help you package the binary file as shown in Figure 4, we have provided an Ant build file called addStorehouseFiles.xml in the Download section of the article. You can run this utility inside the IBM Workload Deployer Eclipse PDK environment. To use this utility, open your IBM Workload Deployer Eclipse PDK environment and place this Ant file under your pattern type project as shown in Figure 5. You should also create a folder named storehouse/files as displayed in Figure 5. The sub-folder files map to the files folder displayed in Figure 4.

After you build your pattern type using IBM Pattern toolkit > build, right-click on addStorehouseFiles.xml and select Run As > Ant Build. The Ant script un-tars the archive file, /patterntype_project/export/built_pattern_type_version.tgz, which was created in the pattern build step. It re-tars with the folder under /patterntype_project/storehouse. Before you use the sample Ant script, edit addStorehouseFiles.xml and change the value of the patternTypeFile property to reflect your pattern type name and version.

Figure 5. Pattern type PDK project with the addStorehouseFiles.xml utility
Pattern type PDK project with the addStorehouseFiles.xml utility

Click to see larger image

Figure 5. Pattern type PDK project with the addStorehouseFiles.xml utility

Pattern type PDK project with the addStorehouseFiles.xml utility

The advantages of using this technique are:

  • You can upload files of any size you want. You have a 2GB limit if you install the pattern using the workload console user interface. For a size greater than 2 GB, use the workload CLI environment.
  • You can package and deliver the binary and the related plug-ins as a single entity.

If you use this technique, the binary is removed from the Storehouse repository when the pattern type is removed from the system. This results in the following limitations:

  • If the binary files are large, you end up spending a lot of time and network resources every time you update the functional elements of your pattern type.
  • If you want to share the same binary across pattern types, you must retain the pattern type that was used to upload the binary in the system, even if it was not required.

You need administrator rights to upload the pattern types. To use this technique, you must belong to the administrator group.

Use a separate binary upload pattern-type to upload the binary

To address the above limitations, you can decouple the binary files from the pattern type that contains the functional artifacts of the virtual application pattern. In this technique, instead of having one pattern type with both binary and virtual application pattern plug-ins, you will have two pattern types. The first pattern type contains only the binary files and second pattern type contains your virtual application pattern plug-ins. You will create the pattern type the same way you created the pattern type in the previous section. The only difference is this pattern type does not have any virtual application pattern plug-ins. Figure 6 illustrates a sample binary only pattern type. Once you decouple the binary from your virtual application pattern plug-ins, then you can manage the binary files independent of your functional artifacts.

Figure 6. Expanded binary only pattern type
Expanded binary only pattern type

The advantages of using this technique are:

  • You can share the binary files across patterns types.
  • You can update and manage the binaries files and the functional aspects of the pattern, independent of each other.

You still need administrator rights to upload the pattern types. To use this technique, you must belong to the administrator group.


Downloading binary files from the Storehouse repository

Now that you have uploaded the binaries to the Storehouse repository, let us see how to download these files to the VMs created by your virtual application pattern instance. You can download the files from Storehouse repository using the virtual application pattern maestro framework in the parts lifecycle scripts. For details on the virtual application pattern maestro framework and parts lifecycle scripts, refer to the Resources section of this article.

Listing 2 illustrates how you can download the binary files from the Storehouse repository. The call to maestro.filesurl returns the location of /admin/files in an HTTP URL format. You can then append your file location to this URL and use the maestro.downloadx method to download and extract the file to the desired location. Once downloaded to your virtual application pattern virtual machines, you can install and configure the binaries.

Listing 2. Download files from the Storehouse repository
def downloadBinaries():
	logger.debug("enter downloadBinaries")
	fileName = 'Sample/Common/rhel_rpm.tgz'
	installerUrl = urlparse.urljoin(maestro.filesurl, fileName)   
	downloadPath = '/home/virtuser'    
	logger.debug('download ' + installerUrl)
	maestro.downloadx(installerUrl, downloadPath)
	logger.debug("exit downloadBinaries")

With this article, we have provided sample code to showcase how to create a binary only pattern-type (patterntype.binaryonly), and a pattern-type with both binary and functional plug-ins (patterntype.storehouseupload and plugin.storehouseupload).


Using the shared service repository server

Shared services are predefined virtual application patterns that can be shared by multiple workload deployments in the same cloud group, including virtual applications, virtual systems, and virtual appliances. Only one instance of shared service is allowed per cloud group. For more details on shared services and how to develop a shared service, refer to the Resources section.

You can create a shared service repository server that can host the binary files required by your workloads. The shared service repository server can expose the necessary interfaces so that the virtual application pattern workload that requires the binary file can use the exposed interfaces to connect to the repository server and download the appropriate file.

With this article, we have provided a sample shared service repository server that acts as a NFS server, and a virtual application pattern workload client that downloads a binary using NFS mount. At deploy time, the shared service repository server is provisioned with the disk size required to store the binaries. The shared service repository server also provides a post-deploy operation that you can use to upload the necessary binaries from an external system using the HTTP protocol. Figure 7 illustrates this post-deployment operation view.

Figure 7. Shared server operation view
Shared server operation view

Click to see larger image

Figure 7. Shared server operation view

Shared server operation view

The Workload Deployer, PureApplication System, and SmartCloud Orchestrator shared service infrastructure provides a registry that contains information about the deployed shared services. The client can query this registry for information about the shared service it wants to work with. In the sample, the repository server exposes its host name and the NFS mount end point to the clients through this registry. Listing 3 illustrates how to provide the information to the shared service registry using the RegistryProvider interface. A repository client that is interested in the shared service repository server can then use the maesro.registry framework in its life cycle scripts to get the shared service server host name, and the NFS end point from the registry and copy the necessary binary file from the NFS end point. Listing 4 illustrates how a repository client can get the details from the registry exposed by the server.

Listing 3. Code to provide shared service repository server information to the registry
public class SampleSharedServiceRegistryProvider implements RegistryProvider {
  private static final String CLASS_NAME =
   SampleSharedServiceRegistryProvider.class.getCanonicalName();
  private static Logger logger = Logger.getLogger(CLASS_NAME);
  public final String LOCAL_BINARIES_REPOS_PATH="/shared_repository";

  public JSONArtifact getRegistry(String clientVersion,
    Map<String, JSONObject> deploymentInfo) throws HttpException {
    String METHOD = "getRegistry";
    JSONObject registry = new JSONObject();
    JSONObject deploymentDoc = deploymentInfo.get
        (RegistryProviderConstants.DEPLOYMENT_DOC);
    logger.logp(Level.FINE, CLASS_NAME, METHOD, "enter getRegistry ");
    //get instance ip addresses and return in the registry
    if(deploymentDoc != null){
      JSONObject instances = (JSONObject) deploymentDoc.get("instances");
      if (null != instances) {
	for (Object key : instances.keySet()) {
	 String keystr = (String)key;
	 if(keystr.startsWith("SampleSharedService")){
           Object ipAddr = ((JSONObject) instances.get(key)).get("public-ip");
                       String ip = (String)ipAddr;
                       registry.put("serviceHost", ip);
	 }
        }
      }
     }
     registry.put("mountPoint", LOCAL_BINARIES_REPOS_PATH);
     logger.logp(Level.FINE, CLASS_NAME, METHOD,
         "getRegistry exit. Registry return="+registry);
     return registry;
  }
}
Listing 4. Client code to retrieve shared service repository server information from the registry
# Retrieve info on SharedService (IP) so we know where to go
regInfo = maestro.registry.getRegistry("sampleMediaService", "1.0")
serviceHost = regInfo['serviceHost']
mountPoint = regInfo['mountPoint']

The advantage of this method is, if you do not want to share the binaries through the Storehouse, then you can have your own binary repository server. Since we use the shared service registry as a lookup directory, the interaction between the client that needs the binary and the repository server that provides the binary is loosely coupled.

The major limitation of using the shared service repository server is that you can have only one shared service of a type per cloud group. Therefore, all the deployments in the cloud group are forced to share the same repository server. Further, there is additional overhead of shared service server development and maintenance. In order to install and deploy a shared service server, you will need administrator rights.


Working with the samples

The following sample pattern types (with source) are available with this article:

  • patterntype.binaryonly-1.0.0.0.tgz
  • patterntype.storehouseupload-1.0.0.0.tgz
  • ptype.installsharedservice-1.0.0.0.tgz

Working with Storehouse repository samples

When you install patterntype.binaryonly-1.0.0.0.tgz or patterntype.storehouseupload-1.0.0.0.tgz, you see a file named "/admin/files/Sample/Common/ rhel_rpm.tgz" added to the Storehouse as illustrated in Figure 2.

In addition to this, with patterntype.storehouseupload-1.0.0.0.tgz, you can create a sample virtual application pattern as illustrated in Figure 8. When you deploy this virtual application pattern, it downloads and extracts the file that was uploaded into the Storehouse to /home/virtuser/binaries on the virtual machine. The virtual application pattern downloads the file using the code illustrated in Listing 2.

Tips for uploading large binaries

If you use the Workload console or the SmartCloud Orchestrator console user interface to install the pattern type, then you cannot install a pattern type larger than 2 GB. You cannot use the console browser interface to import large binaries files to the Storehouse. To overcome this limitation, you have to use the CLI API to install the pattern type. You can follow these steps to install the pattern type from the command line:

  1. Open a command prompt and change to the directory where your pattern type archive file is located.
  2. Download the CLI zip from the IBM Workload Deployer, IBM PureApplication System, or IBM SmartCloud Orchestrator system. Then unzip it under /pathToCli.

From the command line, run the following, depending on your IBM cloud solution:

/pathToCli/pure.cli/bin/pure -h IPAS_IP -u <user> -p <pwd>

or

/pathToCli/deployer.cli/bin/deployer -h IPAS_IP -u <user> -p <pwd>

Once you get back the CLI prompt, run the following command:

deployer.patterntypes.create('YourPatternType-1.0.0.0.tgz')
Figure 8. Virtual application created using patterntype.storehouseupload
Virtual application created using patterntype.storehouseupload

Click to see larger image

Figure 8. Virtual application created using patterntype.storehouseupload

Virtual application created using patterntype.storehouseupload

Working with the shared service repository pattern type sample

After you install and enable the pattern type ptype.installsharedservice-1.0.0.0.tgz, deploy the shared service Sample Shared Service with the disk size to be provisioned for your binary repository as its input. After deployment of the shared service, you must populate the shared service repository with the binary files using the post-deployment operations provided in the Virtual Application Console. To do this, once the shared service is running, select the running shared service instance and click on the Manage button. The Virtual Application Console opens. In the Virtual Application Console, select Operations > Media Service. You see two operations available as shown in Figure 7. Use the second operation, Import file via HTTP, to populate the binary repository. If you have multiple files, you can run this operation multiple times.

To test the shared service repository, create and deploy a virtual application pattern based on the pattern type sample.installsharedservice 1.0 as illustrated in Figure 9. As explained in the previous section, the virtual application pattern gets the following shared service repository details, the shared service repository IP, and the NFS mount point from the registry (see the code in Listing 4). The client can use this data to mount the endpoint and download the binary.

Figure 9. Virtual application created using sample.installsharedservice
Virtual application created using sample.installsharedservice

Click to see larger image

Figure 9. Virtual application created using sample.installsharedservice

Virtual application created using sample.installsharedservice

Conclusion

In this article, you learned the various techniques to manage the binary files to be used in your virtual application pattern workload. You also learned the relative advantages and the limitations of these techniques. Based on the advantages and limitations, the best available option to manage the binary files is using the Storehouse repository. The best technique to upload the binary files to the Storehouse repository is to use a dedicated binary only pattern type.

Acknowledgements

The authors would like to thank Jim Riordan of IBM for helping with the concept of the shared services repository technique.


Download

DescriptionNameSize
Code samplecode_sample.zip76KB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Cloud computing on developerWorks


  • Bluemix Developers Community

    Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.

  • developerWorks Labs

    Experiment with new directions in software development.

  • DevOps Services

    Software development in the cloud. Register today to create a project.

  • Try SoftLayer Cloud

    Deploy public cloud instances in as few as 5 minutes. Try the SoftLayer public cloud instance for one month.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Cloud computing
ArticleID=961638
ArticleTitle=Managing product binaries in virtual application pattern workloads
publish-date=01292014