Modernized Java-based batch processing in WebSphere Application Server, Part 4: Integration with enterprise schedulers

The earlier articles in this series introduced how Java™-based batch programming can be done with IBM® WebSphere® Application Server, and discussed the available programming models and various advanced capabilities of the WebSphere Batch feature, such as parallel processing, skip record processing, and retry step processing. This installment demonstrates how the WebSphere Application Server batch infrastructure can be integrated with enterprise schedulers. This content is part of the IBM WebSphere Developer Technical Journal.

Shishir Narain (nshishir@in.ibm.com), IT Architect, IBM

Shishir Narain photoShishir Narain is Open Group certified Master IT Specialist with mature skills in IBM middleware products. He works in IBM Lab Services for WebSphere at India Software Labs, Gurgaon. He has 14 years of experience developing solutions for multiple clients. He has a Master of Technology degree from Indian Institute of Technology, Kanpur.


developerWorks Contributing author
        level

Thai La (ttla@us.ibm.com), Software Developer, IBM China

Thai La is a software developer from WebSphere Compute Grid team. She works for IBM for the last 6 years. She has done various development and support for the WebSphere Compute Grid product.



20 March 2013

Also available in Chinese

Introduction

IBM WebSphere Application Server V8.5 and later provides an execution platform for Java-based batch applications. In addition to providing a rich programming model and advanced features, such as parallel processing, skip record processing, retry step processing, and COBOL support, it also offers enterprise “qualities” — such as availability, recoverability, scalability, and performance — to batch programs. Coupled with WebSphere Application Server support, WebSphere Batch can be an attractive enterprise batch solution option.

Batch frameworks ease the development of discrete batch jobs, but they don’t provide any choreography between various jobs. Such enterprise wide job choreography is traditionally performed using enterprise schedulers like IBM Tivoli® Workload Scheduler, Control-M, or ASG-Zeke. Enterprise schedulers can provide centralized management of batch jobs running across different operating system platforms, along with advanced features like managing job dependencies, job monitoring and alert mechanisms, automated job restart, and recovery. Enterprise schedulers and batch frameworks thus provide complementary features, and together provide a complete enterprise batch solution.

WebSphere Batch is essentially an endpoint at which batch jobs can be scheduled by the enterprise schedulers. Using details and examples, this article describes how WebSphere Batch can be seamlessly integrated with enterprise schedulers. This integration enables the enterprise scheduler to be the meta-scheduler and provide inputs to the WebSphere Batch job scheduler. The job scheduler, in turn, apprises the enterprise scheduler of job state information and updates the execution log.

For the sake of simplicity, this article does not address security aspects or integration with the z/OS® environment. See Resources for information on the System z integration scenario.


Integrating WebSphere Batch with enterprise schedulers

In order to make WebSphere Batch and an enterprise scheduler work together seamlessly, WebSphere Batch provides these features:

  • Multi-platform support to enable schedulers running on different platforms to use it.
  • Ability to submit and cancel the jobs from the enterprise scheduler.
  • Continuous feedback and final return status to the enterprise scheduler.

As mentioned in the previous articles in this series, WebSphere Batch feature provides web, EJB, and web service interfaces for job management. But in order to meet the above integration requirement, WebSphere batch (on distributed platforms) provides WSGrid, a JMS-based integration solution.

The enterprise scheduler interacts with WebSphere Batch through a WSGrid client, a platform-specific batch program; WSGrid.bat on Windows® and WSGrid.sh on UNIX® platforms. The WSGrid client is basically a JMS program that places requests on the WebSphere Application Server job scheduler. The job scheduler server, enhanced with an MDB application, listens for job requests on a pre-configured SIBus queue. The MDB application then passes the request to the WebSphere Application Server job scheduler, which then processes the request normally. The job scheduler keeps emitting the logs on the pre-determined queue. The WSGrid client goes between the enterprise scheduler and the queues on the WebSphere Application Server job scheduler machine and keeps updating the enterprise scheduler. Using WSGrid enables this integration to be as simple as calling a shell program from the enterprise scheduler. Figure 1 illustrates this integration.

Figure 1. Integration of Enterprise Scheduler with WebSphere Batch
Figure 1. Integration of Enterprise Scheduler with WebSphere Batch

The sequence of activities in a typical WSGrid integration is:

  1. The enterprise scheduler calls the WSGrid client batch file.
  2. The WSGrid client puts a JMS Request message on the pre-defined queue on the WebSphere job scheduler.
  3. Listening to the request queue, the MDB application takes up this request message and uses it to submit the job.
  4. The job is then executed on the server end points, as it normally would had it been submitted from the Job Management Console.
  5. The output from the server is given back to the enterprise scheduler via the response queue.
  6. At the completion of the job, the end status is sent to WSGrid, and then the connection to the WebSphere job scheduler server is closed. While the job is in progress, WSGrid keeps getting the output which is in turn consumed by the enterprise scheduler

Setting up WSGrid

Using the infrastructure setup and project developed in Part 3 of this series, you would follow these steps to set up WSGrid:

  1. Enhance the WebSphere Scheduler server with the MDB application, and create the JMS infrastructure. This can be done using the wsgridConfig.py script:

    wsadmin.sh -f wsgridConfig.py -install -node <nodeName> -server <serverName> -providers <providerList>

    where <providerList> means “hostname,portnumber,” and portnumber being the SIB_ENDPOINT_ADDRESS port of the scheduler server. (See Resources)

    Listing1 shows the output from a sample run.

    Listing 1
    wsadmin -f wsgridConfig.py -install -providers "Anish-PC,7276" -node Anish-PCNode01 
    -server server1
    WASX7209I: Connected to process "dmgr" on node Anish-PCCellManager01 using SOAP 
    connector;  The type of process is: DeploymentManager
    . . . . .	
    wsgridConfig.py is performing install using
      target cluster or server : /Node:Anish-PCNode01/Server:server1/
      MDI application name     : JobSchedulerMDI
      JMS connection factory   : com.ibm.ws.grid.ConnectionFactory
      JMS activation spec      : com.ibm.ws.grid.ActivationSpec
      JMS input queue name     : com.ibm.ws.grid.InputQueue
      JMS output queue name    : com.ibm.ws.grid.OutputQueue
      Installing WSGrid
      JMS file store root      : /tmp\JobSchedulerBus
      SIB identifier           : JobSchedulerBus
      endpoint provider list   : Anish-PC:7276

    As you can see, this script installed the MDB application and created the JMS infrastructure on the scheduler server. Let’s look at the various components that were installed by this script:
    • The MDB application JobSchedulerMDI acts as the interface to the job scheduler. The WSGrid client uses this MDB application to exchange messages with job scheduler.
    • The bus JobSchedulerBus, built on WebSphere Application Server SIBus, is created on the job scheduler server and it hosts:
      • Request queue com.ibm.ws.grid.InputQueue: The MDB listens on this queue and takes job submission requests.
      • Reply queue com.ibm.ws.grid.OutputQueue: The MDB responds on this queue with job progress logs and status.
      • com.ibm.ws.grid.ActivationSpec: Used to bind JobSchedulerMDI MDB to the request queues.
  2. Because WebSphere Application Server is already installed, the enterprise scheduler can call the WSGrid client. WSGrid client takes in a control file as parameter; this is a simple properties file that contains information about the scheduler location. Listing 2 shows sample entries for the control file wsgrid.cntl.
    Listing 2
    scheduler-host=192.168.1.107
    scheduler-port=9080
    debug=false

Using WSGrid

Once WSGrid is configured, you can launch it via the WSGrid.sh/bat shell script. Listing 3 shows a sample of how to invoke the WSGrid client using the script WSGrid.bat and its output.

Listing 3
WSGrid.bat wsgrid.cntl SkipAndParallelJob.xml

WSGrid Version WAS85.WBAT [gm1217.01] 2012-04-24 16:15:25
CONTROL: scheduler-port=9080
CONTROL: debug=true
CONTROL: scheduler-host=192.168.1.107
DEBUG: Obtain JES jobid in checkEnvironment
DEBUG: No JES job id - running in shell.
DEBUG: creating message lisenter with host=192.168.1.107 and port=9080
DEBUG: JobMessenger.decode: pw not encoded
DEBUG: JobMessenger host=192.168.1.107
DEBUG: JobMessenger port=9080
DEBUG: Using correlator value= a479ea48-65a5-4fd8-852c-f2bdaf258ad3-b10ac453-bcb7-4467
-b49a-963c52380c50
DEBUG: JobMessenger: opening communications
DEBUG: JobSchedulerServiceProxy: user=null
DEBUG: JobSchedulerServiceProxy: protocol=http
. . . . . .
System.out: [10/01/12 19:30:52:040 EDT] **********  End SkipAndParallelJob:00257:00259 
log  **********
CWLRB5630I: [10/01/12 19:30:52:047 EDT] Step SkipAndParallelStep completes normally: 
ended normally
CWLRB3800I: [10/01/12 19:30:52:053 EDT] Job [SkipAndParallelJob:00257] ended normally.
CWLRB3880I: Job [SkipAndParallelJob:00257] ending status: RC=0

The logs in Listing 3 show how WSGrid provides continuous feedback to the enterprise scheduler. Aside from job submission, WSGrid also enables the canceling and restarting of WebSphere Batch jobs. This ensures that if the WSGrid job is cancelled, the WebSphere Batch job is cancelled also, thus ensuring synchronization of the two jobs.

On distributed operating systems, cancelling the WSGrid client also triggers the cancelling of the job running by WebSphere Batch. The canceled jobs end in restartable mode (Listing 4).

Listing 4
CWLRB5630I: [10/21/12 06:22:38:250 EDT] Step SkipAndParallelStep completes [cancelled 
on iteration 5507]: cancelled
...........................
...........................
CWLRB2580I: [10/21/12 06:22:38:360 EDT]  [10/21/12 06:22:38:360 EDT]  Job 
[SkipAndParallelJob:00272:00273] Step [SkipAndParallelStep] completed [cancelled].
CWLRB3820I: [10/21/12 06:22:38:360 EDT] Job [SkipAndParallelJob:00272:00273] ended 
abnormally [cancelled].
CWLRB5594I: [10/21/12 06:22:38:360 EDT] Step SkipAndParallelStep execution is complete: 
Cancelled or Stopped
CWLRB3860W: [10/21/12 06:22:38:375 EDT] Job [SkipAndParallelJob:00272:00273] ended 
abnormally [and is restartable]

The WSGrid program can be submitted with an additional parameter that specifies the name of the restart file. On execution, the WSGrid program creates the restart file containing the job ID. This restart file can be used by WSGrid and its calling enterprise scheduler to identify the instance of the job that needs to be restarted.

For example:

WSGrid.bat wsgrid.cntl SkipAndParallelJob.xml restart.properties

Notice the additional parameter restart.properties, which specifies the name of the restart file. When the job ends, a file named restart.properties is created, containing this line:

restart-job=SkipAndParallelJob:00278

To restart the job, use this command:

WSGrid.bat wsgrid.cntl restart.properties

The job specified in restart.properties will be restarted (Listing 5).

Listing 5
WLRB5618I: [10/21/12 09:01:53:908 EDT] Initializing step SkipAndParallelStep batch data 
stream outputStream
CWLRB5620I: [10/21/12 09:01:53:908 EDT] Opening step SkipAndParallelStep batch data 
stream outputStream
CWLRB5612I: [10/21/12 09:01:53:908 EDT] Positioning SkipAndParallelStep batch data stream 
inputStream using restart token: 0;0
CWLRB5612I: [10/21/12 09:01:53:908 EDT] Positioning SkipAndParallelStep batch data stream 
outputStream using restart token: 0;NOTSET

Conclusion

This article discussed the difference between the WebSphere Batch job scheduler and the enterprise scheduler, and showed how WebSphere Batch can be easily integrated with enterprise schedulers using the WSGrid client. Canceling and restarting jobs using WSGrid client was also explained.


Acknowledgements

The authors thank Sajan Sankaran and Abhilash Usha for reviewing this article and providing their invaluable input.


Download

DescriptionNameSize
Code sampleSkipAndParallelJob.xml5KB

Resources

Learn

Get products and technologies

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=861808
ArticleTitle=Modernized Java-based batch processing in WebSphere Application Server, Part 4: Integration with enterprise schedulers
publish-date=03202013