Deploy new applications on IBM PureSystems with plug-ins, Part 2

Lessons learned from a project to enable a real-world SugarCRM solution

The introduction of IBM® PureSystems -- an expert, integrated, enterprise-level family of cloud system that encompass applications, services, hardware, and even the expertise (delivered in the form of best practices patterns) -- takes cloud computing to new heights. One way to prepare an application to leverage IBM PureSystems is to craft a plug-in, a bridge between the application package and the system. Part 1 describes the development efforts to enable SugarCRM on IBM PureSystems. This article addresses the lessons learned from those efforts.

Chin Huang, Cloud Solutions Architect, IBM

Chin Huang is a cloud solutions architect specialized in IT and software architecture. His professional experience is in development of advanced solutions, including IaaS platform, SaaS integration, analytics cloud, web services, and IWD plug-ins. Chin is a certified cloud computing solution advisor, and author of more than 10 technical papers. He lives and works in Silicon Valley, California.



Ton Ngo (ton@us.ibm.com), Cloud Solutions Architect, IBM

author photoTon Ngo is a cloud solution architect and senior developer at the IBM Silicon Valley Lab, San Jose, California. Recently he was the architect for the High Performance Computing Cloud for the Nanyang Technological University, Singapore and the Test and Development Cloud for the Royal Bank of Canada. Previously he was a researcher at the IBM T.J. Watson Research Center and Almaden Research Center for 17 years and has published papers on a wide range of subjects.



11 April 2012

Also available in Chinese Russian Japanese Vietnamese

In Part 1 of this series, the IBM Cloud labs team experts describe the development efforts to enable SugarCRM, an independent service vendor application, on IBM PureSystems. SugarCRM is a PHP application that requires the LAMP stack (Linux®, Apache, MySQL, PHP). LAMP is not supported by IBM PureSystems out of the box, so the team developed a new pattern type and a set of plug-ins that support the modeling, deployment, and operation of the application atop the base Linux and IBM AIX® image. This article addresses the lessons learned from the project.

It's helpful, but not neccesary to read Part 1 before continuing with this article.

The lessons learned from developing a plug-in for the existing third-party SugarCRM application cover these topics:

  • Supporting multiple platforms
  • Managing MySQL and DB2
  • Developing scripts
  • Debugging plug-ins
  • Running a life cycle script
  • Deleting an undeletable plug-in
  • Importing individual plug-ins versus packaged plug-ins
  • Using versions to accelerate development
  • Handling large binaries
  • Extending the base image

Supporting multiple platforms

IBM PureSystems plug-ins are by nature platform neutral. The plug-in developers must design an execution plan to support multiple platforms in different phases of the plug-in and application life cycles. Otherwise, the on-going development, maintenance, and debugging can become very difficult rather quickly.

The following areas are particularly important to our project:

  • Declaring platform support and requirements.
  • Organizing files and scripts for multiple platforms.
  • Gathering plug-in-enablement input for multiple platforms.
  • Writing platform-neutral and platform-specific scripts.
  • Managing supported versus non-supported platforms.

Let's look at the areas more closely.

Declaring platform support and requirements

The plug-in configuration file, config.json, has a requires section for each package instance. If a plug-in supports multiple platforms with the same set of files, there is no need to declare platform.

Otherwise, as in our case with vast differences between Linux and AIX files and scripts, the arch clause should be used, along with other attributes such as memory and CPU, to indicate the package definition is limited to support a particular platform. As the results show, our package has two definitions, each with independent system requirements and parts.

Organizing files and scripts for multiple platforms

The folder structure in the plug-in is tightly related to the package definition. For easier maintenance, always consider using the same set of files and scripts for multiple platforms. We chose to clearly indicate platform in our directory names as described in Part 1, Design and development.

Gathering plug-in enablement input for multiple platforms

The plug-in configuration user interface is defined in the plug-in's config_meta.json file. The declarative approach makes it easy to construct the user interface and at the same time restricted in terms of flexibility to support multiple platforms if the requirements are different.

With the list of files described in Part 1, Solution packaging for Linux and the much longer one for AIX, we decided to generically take an input of one packaged file with all prerequisite files compressed in a predefined folder structure for each of our functional plug-ins. This design allows the cloud administrator to collect just enough files for the target platform while our plug-ins support both Linux and AIX.

Writing platform-neutral and platform-specific scripts

The principle is to write the same set of scripts for Linux and AIX platforms wherever possible. We eventually wrote two sets due to the drastically different processes to get PHP up and running.

Still, we learned the importance of knowing the platform limitations in terms of commands and scripts. For example, in AIX, the first line of a shell script should be #!/bin/sh or #!/bin/ksh rather than #!/bin/bash, and gunzip followed by tar should be used because tar command does not have the -z option.

Managing a supported versus non-supported platform

Our initial target platform for cloud deployment is System p; however, due to numerous logistic and scheduling reasons, we found it necessary to support System x as well.

A virtual application pattern is platform independent; therefore the need to support multiple platforms does not affect the modeling aspect, but it does impact the plug-in implementation because it requires that the plug-in provides full support for all platforms.

In our case, fully covering the support matrix was not possible because of our limited resource. This limitation does expose the risk of an unexpected user experience when a pattern fails to deploy in an unsupported platform. Our current workaround for this situation is to provide templates with a name associated with the supported platform, for instance, "SugarCRM with DB2 on AIX."


Managing MySQL and DB2

Technically MySQL and DB2 serve the same purpose of a back-end database for SugarCRM applications. We used the similar Velocity templates for modeling and transformation which shortened our development time. The ownership aspect of the software however made a major impact on the supported topology and implementation in our solution. IBM owns DB2; therefore we naturally designed our solutions to integrate with out-of-box DB2 plug-ins, as described in Part 1, Reuse existing plug-ins.

Contrarily we applied the loosely coupled approach to support existing MySQL.


Developing script

In installing the software stack required by SugarCRM on the base image, we found that a number of packages (RPMs) are not included in the image. As a result, the plug-in needs to include and install these missing RPMs.

One possible solution is to provide a service during activation that automatically satisfies the dependencies and installs the missing packages. The plug-in implementation simply lists the prerequisite packages and does not need to handle them.

An error condition that we encountered involved a race condition between two roles. (A race condition occurs when the output or result of the process is unexpectedly and critically dependent on the sequence or timing of other events.) It is important to note that the life cycle scripts from different roles are invoked in parallel; therefore, care must be taken to avoid conflict when the scripts manage shared resources.

In one instance, we found that a script from one role was stopping Apache while a script from the other role was starting Apache. This led to unexpected behavior where Apache would not operate correctly. It is necessary to identify shared services or resources and design the code to properly manage them.


Debugging plug-ins

In developing the shell scripts, we found that it took too long to go through the cycle of editing the script in Eclipse, building the plug-in, deleting the plug-in in IBM PureSystems, and then redeploying. In addition, the VM and the patterns utilizing the plug-in needed to be deleted and recreated.

In practice, we leveraged the debugging support in the PDK to expedite the script development. This was done by first adding all the necessary artifacts and the basic script files in the plug-in. We deployed the pattern with the debug component to create an environment with the artifacts in the store house and the activation directories containing the JSON files. Then we accessed the VM and invoked the scripts individually for additional development and debugging. The scripts were copied back into the Eclipse projects.


Running a life cycle script

When developing life cycle Python scripts, the developer must pay close attention to the life cycle events that trigger each script to run. Some scripts are run only once in the entire lifespan of a VM and others are executed in various life cycle events.

For example, install.py is run only once for initialization while configure.py, start.py, and stop.py are also invoked on every VM reboot.

We had a specific requirement to execute the SugarCRM install in the database link plug-in's changed.py script after Apache, PHP, and the database are all properly configured. The install script should be run only once to set up and configure SugarCRM contents, not every time changed.py is also invoked for each VM restart.

A role attribute can be introduced in Python to indicate the install status. In changed.py, we check the existence of a persistent file produced by SugarCRM at the end of install to ensure the install is not run multiple times.

The details of life cycle scripts can be found in IBM Workload Deployer Information Center and Plug-in Development Guide.


Deleting an undeletable plug-in

During development, we often face a situation that a plug-in cannot be deleted in IBM PureSystems because some element of the plug-in is still in use. While the error message simply states the plug-in is in use and doesn't provide the information on where or how, we found that doing a thorough clean-up of deployed application instances, application patterns, and application templates related to the plug-in usually gets rid of the error.


Importing individual plug-ins versus packaged plug-ins

Although we eventually packaged all our plug-ins into a pattern type, we learned it is more efficient and timely to import individual plug-ins into IBM PureSystems during development. The primary pattern type still has to be imported separately so that individual plug-ins can be configured and enabled.

This late-binding mechanism allows any of us to change and debug just one plug-in without affecting the pattern type and other plug-ins. The shortened development cycle is extra helpful because some of our plug-ins carry large file sets and take long to re-import and re-enable.

That said, we ran into a situation where we could no longer configure or delete an individually imported plug-in if the primary pattern type is deleted in IBM PureSystems. The scenario can happen when one developer attempts to delete a plug-in and another developer is in the middle of refreshing the pattern type. When the primary pattern type is put back in, the troubled plug-in can be fully managed. The pattern type packaged plug-ins have no such issue because they come and go with the pattern type.

We suggested to the IBM PureSystems product team to allow pattern type deletion only if there are no individually imported, dependent plug-ins; the feature will be delivered in a future release.


Using versions to accelerate development

Each pattern type has a version in the major.minor format. During development, occasionally we wanted to deploy and compare multiple versions of a pattern type or we simply did not want to go through the pattern type clean-up cycle. We found that it was expedient to use a different major version to construct and install the pattern type.

The plug-ins packaged in the pattern type also need to have a new major version and to refer to the new pattern type version. For example, if the currently installed pattern type version is 1.0.0.1 and the new pattern type version is 2.0.0.1, the plug-ins should have the configuration such as the following sample segment in config.json:

"name": "sugarcrm",
"version": "2.0.0.2",
"patterntypes": {
    "primary": {
        "sugarcrm": "2.0"
    }   
},

This technique not only supports quick verification of a new pattern type implementation, but it also allows multiple developers to share the same IBM PureSystems environment.


Handling large binaries

Because of the licensing consideration described previously, we must package some software separately from the plug-in and upload them when the plug-in is deployed. This step is performed only one time in the normal scenario; however, during plug-in development, repeated uploading of large binaries in the range of 500MB can be inhibitive, particularly with a slow network.

A technique used by the plug-in development team is to use a variant of the build process that allows loading binary files separately from the plug-ins. This is possible with an internal development environment. The plug-in development team is working on providing the capability to all plug-in developers.


Extending base image

To provide the set of prerequisite software for an application, a trade-off frequently considered is whether to install the software on the base image or to include the software as part of the base image. Because IBM PureSystems allows the user to extend the base image, we considered creating a new base image with the necessary RPMs for SugarCRM. This simplifies the initial development effort as well as the maintenance effort in the future since the plug-in will be insulated from changes in the base image.

However, because such a customized base image would be common to all patterns, it is not clear if this approach is workable in practice. It appears that some flexibility in associating the base image, perhaps in conjunction with an image-building tool such as ICON, adds new possibilities in plug-in design. In general it is easier for the customer to use one base image. Adding this capability requires careful considerations.


In conclusion

In this article and Part 1, we've been able to demonstrate the design considerations our team used as a template to guide us in creating a plug-in to enable the existing SugarCRM solution to deploy on IBM PureSystems; we also took you through the development efforts we made; and we detailed the lessons we learned from this project.

We hope our hands-on experiences can make writing IBM PureSystems plug-ins for existing applications much easier for you.

After successfully testing the use cases, we've declared that the SugarCRM solution is "IBM PureSystems-ready." Besides gaining valuable experience in developing complex plug-ins, we believe that our project has achieved these significant results:

  • Successful delivery of the first application pattern on IBM PureSystems based on a non-standard software stack. This will contribute to the success of both IBM PureSystems and SugarCRM by enabling an important ISV to offer their software as part of the IBM PureSystems catalog.
  • Clear demonstration that IBM PureSystems is not limited to application patterns provided out of the box. By developing new plug-ins, new applications from customers and ISVs can be integrated to leverage the power of IBM PureSystems.
  • Rigorous exercise of the support for plug-in development in IBM PureSystems. This support must be robust in order to build a rich catalog of application patterns. By exposing the plug-in framework to scenarios that are different from the current set of IBM software, we helped the IBM PureSystems team identify areas that need improvements and the effort has led to numerous new features that will help future development.

IBM PureSystems brings the potential for greatly increasing productivity in cloud by allowing users to focus on their application patterns without concerning the underlying infrastructure. We believe that the proliferation of new plug-ins will ensure the success of IBM PureSystems by giving the users a rich set of choices in application patterns.

Acknowledgements

We'd like to thank the major supporters of this project:

  • Willy Chiu, Larry Hsiung, Thomas Truong, Jeffrey Coveyduc, Kai Young, Nauman Fakhar, Chris Kiernan, Raymond Wong of the cloud labs and HiPODS team
  • Steve Ims, Lin Sun, Ted Kirby of the IBM Workload Deployer development team
  • Ning Wang, Qi Rong Wang of the IBM Database as a Service development team
  • Nasser Momtaheni, Joseph Peterson, Rodney Johnson of the IBM Innovation Center team
  • Stas Malyshev of SugarCRM

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Cloud computing on developerWorks


  • Bluemix Developers Community

    Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.

  • Cloud digest

    Complete cloud software, infrastructure, and platform knowledge.

  • DevOps Services

    Software development in the cloud. Register today to create a project.

  • Try SoftLayer Cloud

    Deploy public cloud instances in as few as 5 minutes. Try the SoftLayer public cloud instance for one month.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Cloud computing, Information Management
ArticleID=809038
ArticleTitle=Deploy new applications on IBM PureSystems with plug-ins, Part 2
publish-date=04112012