Improve your industry solution installation, Part 1: Centralize the logging

Often industry solutions are deployed on complex, distributed topology, where the logs are generated dispersedly and are hard to manage. Logging centralization provides an easier means to read and manage logging. This article describes a best practice approach for logging centralization when deploying industry solutions in a distributed environment.

Share:

Zhao Zhuo, Software Engineer, IBM

Photograph of Zhao ZhuoZhuo Zhao is a software engineer working in the IBM China Development Lab. She has many years of experience in Industry Solution development and deployment. Currently, Zhuo is working on automated deployment for IBM Integrated Information Core.



Jing Fan, Staff Software Engineer, IBM

Photograph of Jing FanJing Fan is a staff software engineer on the Industry Solution Development Team. She is responsible for Integrated Information Core installer development and deployment. Jing is experienced working with the IBM Solution Assembly Toolkit and Install Anywhere.



Brian Lavin, Senior Software Engineer, IBM

Photograph of Brian LavinBrian Lavin is a member of the industry solutions team in the IBM Dublin Software Lab. He is an Java Enterprise developer whose experience includes working with industry solutions installers for the IBM Integrated Information Core product.



Raymond Mahon, Infrastructure Software Engineer, IBM

Photograph of Raymond MahonRaymond Mahon is a member of the industry solutions team in the IBM Dublin Software Lab. He has many years of experience in designing automated solutions and is currently focusing on industry solutions installers development.



15 January 2013

Overview

Target audience

This article is for software engineers who are working on industry solution development and other automatic installations. It focuses on industry solution deployment using the IBM Software Assembly Toolkit. We assume the audience has experience in:

  • Industry solution deployment
  • Java™ development with Ant
  • IBM Software Assembly Toolkit development

Often industry solutions are deployed on a complicated and distributed topology, that involve the installation and integration of multiple middleware, and several resource configurations and enterprise application deployments. This familiar scenario can make the industry solution deployment complex and produces a large number of various loggings during the deployment and configuration. The loggings are generated to different directories and on distributed environments. It can be hard for the customer to find the logging for a specific middleware installation or a specific configuration, and if the deployment failed troubleshooting is more difficult.

Fortunately, it is possible to handle and manage logging in a centralized way during distributed industry solution deployment. In this article, we demonstrate an approach to centralize logging management for industry solution deployment that will not only make the deployment clean and neat, but also reduce overall effort for troubleshooting or deployment verification.


IBM Software Assembly Toolkit

The IBM Software Assembly Toolkit was designed to simplify the process of creating and deploying industry solutions. It provides an open platform that you can use to integrate custom applications, products, and services with middleware components to create and distribute a total business solution as a single deliverable. The toolkit automates the installation and configuration of middleware products and software applications by assembling them into simple, fast, and reliable deployment packages. This helps developers deliver and package a complete business solution to one or more target computers in the way they want.

Advantages

The toolkit provides flexible, easy-to-use packaging and deployment tools for creating and distributing complete solutions across multiple platforms.

You might consider the IBM Software Assembly Toolkit as an industry solution deployment toolkit if you are going to build your own industry solution on top of a complex topology instead of a single deployment, as remote agents are deployed on each target for the life of the installer and deployment. The toolkit also provides support for multiple operation systems, which makes your solution install development flexible and efficient. The Eclipse-based IDE may help you quickly become a "solution deployment master".

The advantages of using the toolkit for industry solution deployment include:

  • You can use built-in remote agent capability, and execute relevant tasks remotely through those remote deployment agents. The remote deployment agent is a very important function in that enables the installer capabilities for distributed deployment, which makes it different from many other toolkits.
  • You can use multiple technical approaches to implement the application deployment accelerators, including Java™, Ant, Shell, Command Line, and IBM Installation Manager.
  • You can use Eclipse-based IDE and GUI interfaces that make both development and testing easier.
  • You can share variables such as user names, installation locations, passwords, and port numbers within the entire installer.

Additionally, it supports multiple platforms, including Windows(reg/>, HP-UNIX, and Linux. Because of these advantages, we chose the IBM Software Assembly Toolkit as our industry solution deployment toolkit. We used the IBM Software Assembly Toolkit V3.2 to develop the samples and code for this article. For more features and details of the IBM Software Assembly Toolkit see Resources.

Drawbacks

Although there are many advantages of using the IBM Software Assembly Toolkit, it is not perfect yet, especially from a logging management perspective. It doesn't manage the install and configuration logs in an centralized way, all the logs generated during deployment are delivered randomly.

Two main drawbacks in the toolkit logging are:

  1. All of the logs are generated by geography. During a multiple-node deployment, each node has its own logs generated according to the tasks running on this node.
  2. The toolkit creates temporary log files randomly, and removes them after installation terminates. You cannot find those files after you terminate or close the installer.

These drawbacks make it difficult to check and manage the toolkit logging during deployment, especially if any troubleshooting work is needed. Now, let's investigate the drawbacks more and learn how you can eliminate them and improve your solution deployment.


Begin with a simple industry solution deployment scenario

You can download a sample installer (see Download) that checks the status of IBM WebSphere Application Server in a WebSphere Application Server Network Deployment distributed environment. Assume that we have two nodes in this environment, one hosts the deployment manager, the other hosts the managed application server.

Figure 1 shows how the sample installer should appear based on this scenario.

Figure 1. The sample installer tasks screen
A screen capture shows how the sample installer should appear based on this scenario.

In the SAMPLE_INSTALLER download included with this article (see Download), there is only one task group named Check Server Status defined, and only one Install Task inside. In this install task, we import the application Sample_Server_Status, and execute server status check.

Listing 1 contains the sample code of the application development accelerator Sample_Server_Status, implemented using Ant.

Listing 1. Sample code of the application development accelerator Sample_Server_Status
<?xml version="1.0" encoding="UTF-8"?>
<project basedir="." default="default">
<!-- Make the properties from the application deployment accelerator available. -->
 <property file="${IRU_propFile}"/>
 <target name="default">
	<echo message="Unpacked directory:  ${IRU_unpacked}" />
    <echo message="Logs directory:  ${IRU_logsDir}" />
    <echo message="Log file name:  ${IRU_logFile}" />
    <echo message="Deployment Accelerator properties file name:  ${IRU_propFile}" />
 	<mkdir dir="${logLocation}"/>
<!-- execute wasadmin with the specified jython script and property file -->
 	<exec executable="${WASHome}/bin/serverStatus.bat">
 		<arg line="-all &gt; '${logLocation}/serverStatus.log'" />
 	</exec>
	
<!--validate execution result-->
 <loadfile property="logcontent"  srcFile="${logLocation}/serverStatus.log"/>
 <condition property="status.finished">
    <or>
 	<contains string="${logcontent}" substring="ADMU0508I" casesensitive="no"/>
 	<contains string="${logcontent}" substring="ADMU0509I" casesensitive="no"/>
    </or>
 	</condition>
 	<echo message="Success=${status.finished}" />
 	<fail message="Failed to verify install task. Please refer to $
{logLocation}/serverStatus.log on target node for detailed information.">
 		<condition>
 			<isfalse value="${status.finished}" />
 		</condition>
 	</fail>
 </target>
</project>

The code in Listing 1 executes the serverStatus.bat command under the WebSphere Application Server home bin directory and generates a log file under a log file folder. After the execution is finished, it loads the log file and validates if there is a successful string output into the log file. If not, the Ant script will notify you to refer to the logging on a remote target node for detailed information. Both WASHome and logLocation are user input variables defined in the application development accelerator.

As shown in Figure 2, the sample installer also has two target groups, the deployment manager node and managed node. Each target node has the Sample_Server_Status application development accelerator running on top of it.

Figure 2. Sample installer targets
A screen capture showing the targets of the sample installer.

Review the install results

After you finish running the sample installer on your environment, a logging file is created.

On your deployment manager you see C:\Samples\ServerStatusLogs\serverStatus.log:

DMU0116I: Tool information is being logged in file: 
C:\Program Files\IBM\WebSphere\AppServer\profiles\SampleDM\logs\serverStatus.log
ADMU0128I: Starting tool with the SampleDM profile
ADMU0503I: Retrieving server status for all servers
ADMU0505I: Servers found in configuration:
ADMU0506I: Server name: dmgr
ADMU0508I: The Deployment Manager "dmgr" is STARTED

On your managed node you see C:\Samples\ServerStatusLogs\serverStatus.log:

ADMU0116I: Tool information is being logged in file: 
C:\Program Files\IBM\WebSphere\AppServer\profiles\SampleManaged\logs\serverStatus.log
ADMU0128I: Starting tool with the SampleManaged profile
ADMU0503I: Retrieving server status for all servers
ADMU0505I: Servers found in configuration:
ADMU0506I: Server name: Sample.AppSvrNode01.0
ADMU0506I: Server name: nodeagent
ADMU0509I: The Application Server " Sample.AppSvrNode01.0" is STARTED
ADMU0509I: The Node Agent "nodeagent" is STARTED.

Note that in our sample, the managed node has two WebSphere instances, the nodeagent and the application server. Sample.AppSvrNode01.0 is the application server name.

The serverStatus.log files are generated because developers explicitly add logger creation statement in their application development accelerator's code, so we call them application related logs. In addition to this type of log, the toolkit temporarily generates log files that record the runtime status and execution process, thus we call them temporary logs.

Temporary logs

There are two kinds of temporary logs generated at runtime. To find their locations, you have to open the deployment task and read the detailed messages, as shown in Figure 3, which is a screen capture of the Sample_Server_Staus task in the sample installer.

Figure 3. Deployment task detailed messages
A screen capture showing the detailed messages displayed in the deployment task.

The unpacked directory outlined first (in red) is named IRU_unpacked, while the log directory outlined second (in green) is named SolutionEnabler.

IRU_unpacked

Before your installer terminates, find a temporary folder named "iru" under c:\tmp. (If you are using a Linux operation system find it under /tmp.) Figure 4 shows the iru directory file structure.

Figure 4. File structure of the iru directory
A screen capture showing the file structure of the iru directory.

In that folder you will find your application script (MainProgram.xml), the ant binaries (iru_ant) and other IBM Software Assembly Toolkit built-in runtime support jar files (extenalSupportJars). The toolkit creates the iru folder during runtime because it is in the runtime environment where your customer's install scripts execute.

Note that you will only find the iru folder while running your installer, after the install tasks are finished, the folder is removed automatically.

SolutionEnabler

Unlike the IRU_unpacked log, the location of SolutionEnabler log is random, and for each time you run a toolkit deployment task, a new SolutionEnabler log folder is generated, with a different number appended. Figure 5 shows the file structure of the SolutionEnabler log.

Figure 5. File structure of the SolutionEnabler log
A screen capture showing the file structure of the SolutionEnabler log.

There is a log folder under SolutionEnabler folder. The log folder contains all the logs that record the progress and execution status of each task.


Identify the problem

At this point you might be confused by the logging mechanism of the IBM Software Assembly Toolkit.

First, all the loggings (including application related logs and temporary logs) are generated by geography. Each node that has the server status check application development accelerator running has serverStatus logging created. Thus you have to navigate among different physical machines frequently to gather or validate logs. The work effort is increased if the installer fails and you need to troubleshoot.

Second, the temporary logs can be troublesome. The IRU_unpacked logs that are created during runtime, are removed after the deployment terminates, regardless on whether the deployment succeeds or fails. The SolutionEnabler logs are created during runtime, but the logs are placed randomly, and are not removed after termination. Both of these problems make it is hard to manage the logs and use to troubleshoot or trace the deployment status.


Improve the deployment with logging centralization features

Generally, most industry solution installers generate and manage loggings using a stage server that dispatches and executes deployment packages. Unfortunately, the logging is dispersed.

Figure 6. A common logging mechanism
A graphic image depicting a staging server dispersing logging packages.

However, we offer an improved sample installer with logging centralization features. In our improved version, a stage server dispatches and executes deployment packages, but all the deployment related logging, including the application related logs, IRU_unpacked logs and SolutionEnabler logs, are retrieved and migrated to a specific logging server, with no logging left dispersedly on target servers.

The logging server can be a separate node, or it can be one of the target machines, as shown in Figure 7.

Figure 7. Suggested logging mechanism
A graphic image depicting a staging server dispersing logging packages, however all the logging is directed to one specific server.

Improved implementation

First you want to collect all the deployment logging on a specific logging server. We recommended that you add an extra user input to the toolkit installer wizard for your logging server. As shown in Figure 8, the variable logHost (highlighted in red) points to the host of the centralized logging server. You can download the sample code to get more about the improved installer.

Figure 8. Tasks of improved sample installer
A screen capture showing variable logHost that points to the host of the centralized logging server.

Next, to add logging centralization feature, you want improve the sample code implementation as shown in Listing 2.

Listing 2. Add logging centralization feature
<?xml version="1.0" encoding="UTF-8"?>
<project basedir="." default="default">
<!-- Make the properties from the application deployment accelerator available. -->
 <property file="${IRU_propFile}"/>
 <target name="default">
	<echo message="Unpacked directory:  ${IRU_unpacked}" />
    <echo message="Logs directory:  ${IRU_logsDir}" />
    <echo message="Log file name:  ${IRU_logFile}" />
    <echo message="Deployment Accelerator properties file name:  ${IRU_propFile}" />
 	<mkdir dir="${logLocation}"/>
<!-- execute wasadmin with the specified jython script and property file -->
 	<exec executable="${WASHome}/bin/serverStatus.bat">
 		<arg line="-all &gt; '${logLocation}/serverStatus.log'" />
 	</exec> 
 	
<!-- get local hostname -->
 	<exec executable="hostname" outputproperty="hostName"/>
<!--4 Steps migrate logging to logging server -->
<!--Step1: migrate Application Related Logging to logging server -->
 	<copy file="${logLocation}/serverStatus.log" todir="//${logHost}/c$
            /Samples/${hostName}Log/"/>
<!--Step2: migrate IRU Unpacked Logging to logging server -->
 	<copy todir="//${logHost}/c$/Samples/${hostName}Log/IRU_Unpacked/">
 	    <fileset dir="${IRU_unpacked}"/>
 	</copy>
<!--Step3: migrate Solution Enabler Logging to logging server -->
 	<copy todir="//${logHost}/c$/Samples/${hostName}Log/Solution_Enabler/">
 	 	<fileset dir="${IRU_logsDir}"/>
 	 </copy>
<!--Step4: clear Logging from target server -->
 <delete dir="${logLocation}"/>
<!--validate execution result -->
 <loadfile property="logcontent"  srcFile="//${logHost}/c$/Samples/
            ${hostName}Log/serverStatus.log"/>
 <condition property="status.finished">
  <or>
 	 <contains string="${logcontent}" substring="ADMU0508I" casesensitive="no"/>
 	 <contains string="${logcontent}" substring="ADMU0509I" casesensitive="no"/>
  </or>
 	 </condition>
 	 <echo message="Success=${status.finished}" />
 	 <fail message="Failed to verify install task. Please refer to //${logHost}
            /${logLocation}/../${hostName}Log/serverStatus.log for detailed information.">
 	 	<condition>
 	 		<isfalse value="${status.finished}" />
 	 	</condition>
 	 </fail>
 </target>
</project>

In this improved implementation, we reuse the status check logic. The Ant script executes wsadmin command to get the server status as before, and then, the logs are generated.

To manage and migrate those logs in an centralized approach, we added a four-step logging migration to copy the logging from target node to logging server, that logging server can be a separate one from the target machines, or it can be one of the target machines.

The loggings on the centralized logging server can be categorized by the folder name. We name the folder by target machine’s host name. That means the logging gathered from different targets are put into different folders.

As commented, we migrate the application related logs, server status logs, to the logging server in step 1. We migrate the IRU_Unpacked logs and SolutionEnabler logs respectively in step 2 and step 3. Then in step 4, the last step, we clear all the logging from target servers.

By doing so, the validation logic can be executed on the centralized logging server, and if the validation fails, users can navigate to the logging server to refer to detailed information about the failuer.

Improved install results

Now, when you finish running the improved sample installer in your environment, the logging files are no longer generated on your target machines. All the loggings are swept and migrated to the centralized logging server.

Figure 9 shows the file structure on logging server. Test4DM is the host name of deployment manager node, while Test4Mgd is the host name of the managed node.

Figure 9. File structure as a result of improved logging
A screen capture showing the file structure created by improved logging.

Conclusion

In this article, you learned about a logging centralization approach that can greatly reduce the cost of managing logging when deploying your industry solution.

With the help of this logging centralization approach, all the install and configure logs can be managed in a centralized way, for developers and customers, there is no need to worry about where to look up related deployment logs, no need to worry about when the temporary logs would be deleted, the only thing they need to do is provide a centralized logging directory. After that, all the logs can be collected from distributed target machines and put in to the centralized directory automatically.


Download

DescriptionNameSize
Sample installer and improved installersampleCode.zip59KB

Resources

Learn

Get products and technologies

  • Evaluation software: Download a trial version, work with product in an online, sandbox environment, or access it in the cloud.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=854376
ArticleTitle=Improve your industry solution installation, Part 1: Centralize the logging
publish-date=01152013