Contents


Modernized Java-based batch processing in WebSphere Application Server, Part 4

Integration with enterprise schedulers

Comments

Content series:

This content is part # of # in the series: Modernized Java-based batch processing in WebSphere Application Server, Part 4

Stay tuned for additional content in this series.

This content is part of the series:Modernized Java-based batch processing in WebSphere Application Server, Part 4

Stay tuned for additional content in this series.

IBM WebSphere Application Server V8.5 and later provides an execution platform for Java-based batch applications. In addition to providing a rich programming model and advanced features, such as parallel processing, skip record processing, retry step processing, and COBOL support, it also offers enterprise “qualities” — such as availability, recoverability, scalability, and performance — to batch programs. Coupled with WebSphere Application Server support, WebSphere Batch can be an attractive enterprise batch solution option.

Batch frameworks ease the development of discrete batch jobs, but they don’t provide any choreography between various jobs. Such enterprise wide job choreography is traditionally performed using enterprise schedulers like IBM Tivoli® Workload Scheduler, Control-M, or ASG-Zeke. Enterprise schedulers can provide centralized management of batch jobs running across different operating system platforms, along with advanced features like managing job dependencies, job monitoring and alert mechanisms, automated job restart, and recovery. Enterprise schedulers and batch frameworks thus provide complementary features, and together provide a complete enterprise batch solution.

WebSphere Batch is essentially an endpoint at which batch jobs can be scheduled by the enterprise schedulers. Using details and examples, this article describes how WebSphere Batch can be seamlessly integrated with enterprise schedulers. This integration enables the enterprise scheduler to be the meta-scheduler and provide inputs to the WebSphere Batch job scheduler. The job scheduler, in turn, apprises the enterprise scheduler of job state information and updates the execution log.

For the sake of simplicity, this article does not address security aspects or integration with the z/OS® environment. See Related topics for information on the System z integration scenario.

Integrating WebSphere Batch with enterprise schedulers

In order to make WebSphere Batch and an enterprise scheduler work together seamlessly, WebSphere Batch provides these features:

  • Multi-platform support to enable schedulers running on different platforms to use it.
  • Ability to submit and cancel the jobs from the enterprise scheduler.
  • Continuous feedback and final return status to the enterprise scheduler.

As mentioned in the previous articles in this series, WebSphere Batch feature provides web, EJB, and web service interfaces for job management. But in order to meet the above integration requirement, WebSphere batch (on distributed platforms) provides WSGrid, a JMS-based integration solution.

The enterprise scheduler interacts with WebSphere Batch through a WSGrid client, a platform-specific batch program; WSGrid.bat on Windows® and WSGrid.sh on UNIX® platforms. The WSGrid client is basically a JMS program that places requests on the WebSphere Application Server job scheduler. The job scheduler server, enhanced with an MDB application, listens for job requests on a pre-configured SIBus queue. The MDB application then passes the request to the WebSphere Application Server job scheduler, which then processes the request normally. The job scheduler keeps emitting the logs on the pre-determined queue. The WSGrid client goes between the enterprise scheduler and the queues on the WebSphere Application Server job scheduler machine and keeps updating the enterprise scheduler. Using WSGrid enables this integration to be as simple as calling a shell program from the enterprise scheduler. Figure 1 illustrates this integration.

Figure 1. Integration of Enterprise Scheduler with WebSphere Batch
Figure 1. Integration of Enterprise Scheduler with WebSphere Batch
Figure 1. Integration of Enterprise Scheduler with WebSphere Batch

The sequence of activities in a typical WSGrid integration is:

  1. The enterprise scheduler calls the WSGrid client batch file.
  2. The WSGrid client puts a JMS Request message on the pre-defined queue on the WebSphere job scheduler.
  3. Listening to the request queue, the MDB application takes up this request message and uses it to submit the job.
  4. The job is then executed on the server end points, as it normally would had it been submitted from the Job Management Console.
  5. The output from the server is given back to the enterprise scheduler via the response queue.
  6. At the completion of the job, the end status is sent to WSGrid, and then the connection to the WebSphere job scheduler server is closed. While the job is in progress, WSGrid keeps getting the output which is in turn consumed by the enterprise scheduler

Setting up WSGrid

Using the infrastructure setup and project developed in Part 3 of this series, you would follow these steps to set up WSGrid:

  1. Enhance the WebSphere Scheduler server with the MDB application, and create the JMS infrastructure. This can be done using the wsgridConfig.py script:

    wsadmin.sh -f wsgridConfig.py -install -node <nodeName> -server <serverName> -providers <providerList>

    where <providerList> means “hostname,portnumber,” and portnumber being the SIB_ENDPOINT_ADDRESS port of the scheduler server. (See Related topics)

    Listing1 shows the output from a sample run.

    Listing 1
    wsadmin -f wsgridConfig.py -install -providers "Anish-PC,7276" -node Anish-PCNode01 
    -server server1
    WASX7209I: Connected to process "dmgr" on node Anish-PCCellManager01 using SOAP 
    connector;  The type of process is: DeploymentManager
    . . . . .	
    wsgridConfig.py is performing install using
      target cluster or server : /Node:Anish-PCNode01/Server:server1/
      MDI application name     : JobSchedulerMDI
      JMS connection factory   : com.ibm.ws.grid.ConnectionFactory
      JMS activation spec      : com.ibm.ws.grid.ActivationSpec
      JMS input queue name     : com.ibm.ws.grid.InputQueue
      JMS output queue name    : com.ibm.ws.grid.OutputQueue
      Installing WSGrid
      JMS file store root      : /tmp\JobSchedulerBus
      SIB identifier           : JobSchedulerBus
      endpoint provider list   : Anish-PC:7276

    As you can see, this script installed the MDB application and created the JMS infrastructure on the scheduler server. Let’s look at the various components that were installed by this script:
    • The MDB application JobSchedulerMDI acts as the interface to the job scheduler. The WSGrid client uses this MDB application to exchange messages with job scheduler.
    • The bus JobSchedulerBus, built on WebSphere Application Server SIBus, is created on the job scheduler server and it hosts:
      • Request queue com.ibm.ws.grid.InputQueue: The MDB listens on this queue and takes job submission requests.
      • Reply queue com.ibm.ws.grid.OutputQueue: The MDB responds on this queue with job progress logs and status.
      • com.ibm.ws.grid.ActivationSpec: Used to bind JobSchedulerMDI MDB to the request queues.
  2. Because WebSphere Application Server is already installed, the enterprise scheduler can call the WSGrid client. WSGrid client takes in a control file as parameter; this is a simple properties file that contains information about the scheduler location. Listing 2 shows sample entries for the control file wsgrid.cntl.
    Listing 2
    scheduler-host=192.168.1.107
    scheduler-port=9080
    debug=false

Using WSGrid

Once WSGrid is configured, you can launch it via the WSGrid.sh/bat shell script. Listing 3 shows a sample of how to invoke the WSGrid client using the script WSGrid.bat and its output.

Listing 3
WSGrid.bat wsgrid.cntl SkipAndParallelJob.xml

WSGrid Version WAS85.WBAT [gm1217.01] 2012-04-24 16:15:25
CONTROL: scheduler-port=9080
CONTROL: debug=true
CONTROL: scheduler-host=192.168.1.107
DEBUG: Obtain JES jobid in checkEnvironment
DEBUG: No JES job id - running in shell.
DEBUG: creating message lisenter with host=192.168.1.107 and port=9080
DEBUG: JobMessenger.decode: pw not encoded
DEBUG: JobMessenger host=192.168.1.107
DEBUG: JobMessenger port=9080
DEBUG: Using correlator value= a479ea48-65a5-4fd8-852c-f2bdaf258ad3-b10ac453-bcb7-4467
-b49a-963c52380c50
DEBUG: JobMessenger: opening communications
DEBUG: JobSchedulerServiceProxy: user=null
DEBUG: JobSchedulerServiceProxy: protocol=http
. . . . . .
System.out: [10/01/12 19:30:52:040 EDT] **********  End SkipAndParallelJob:00257:00259 
log  **********
CWLRB5630I: [10/01/12 19:30:52:047 EDT] Step SkipAndParallelStep completes normally: 
ended normally
CWLRB3800I: [10/01/12 19:30:52:053 EDT] Job [SkipAndParallelJob:00257] ended normally.
CWLRB3880I: Job [SkipAndParallelJob:00257] ending status: RC=0

The logs in Listing 3 show how WSGrid provides continuous feedback to the enterprise scheduler. Aside from job submission, WSGrid also enables the canceling and restarting of WebSphere Batch jobs. This ensures that if the WSGrid job is cancelled, the WebSphere Batch job is cancelled also, thus ensuring synchronization of the two jobs.

On distributed operating systems, cancelling the WSGrid client also triggers the cancelling of the job running by WebSphere Batch. The canceled jobs end in restartable mode (Listing 4).

Listing 4
CWLRB5630I: [10/21/12 06:22:38:250 EDT] Step SkipAndParallelStep completes [cancelled 
on iteration 5507]: cancelled
...........................
...........................
CWLRB2580I: [10/21/12 06:22:38:360 EDT]  [10/21/12 06:22:38:360 EDT]  Job 
[SkipAndParallelJob:00272:00273] Step [SkipAndParallelStep] completed [cancelled].
CWLRB3820I: [10/21/12 06:22:38:360 EDT] Job [SkipAndParallelJob:00272:00273] ended 
abnormally [cancelled].
CWLRB5594I: [10/21/12 06:22:38:360 EDT] Step SkipAndParallelStep execution is complete: 
Cancelled or Stopped
CWLRB3860W: [10/21/12 06:22:38:375 EDT] Job [SkipAndParallelJob:00272:00273] ended 
abnormally [and is restartable]

The WSGrid program can be submitted with an additional parameter that specifies the name of the restart file. On execution, the WSGrid program creates the restart file containing the job ID. This restart file can be used by WSGrid and its calling enterprise scheduler to identify the instance of the job that needs to be restarted.

For example:

WSGrid.bat wsgrid.cntl SkipAndParallelJob.xml restart.properties

Notice the additional parameter restart.properties, which specifies the name of the restart file. When the job ends, a file named restart.properties is created, containing this line:

restart-job=SkipAndParallelJob:00278

To restart the job, use this command:

WSGrid.bat wsgrid.cntl restart.properties

The job specified in restart.properties will be restarted (Listing 5).

Listing 5
WLRB5618I: [10/21/12 09:01:53:908 EDT] Initializing step SkipAndParallelStep batch data 
stream outputStream
CWLRB5620I: [10/21/12 09:01:53:908 EDT] Opening step SkipAndParallelStep batch data 
stream outputStream
CWLRB5612I: [10/21/12 09:01:53:908 EDT] Positioning SkipAndParallelStep batch data stream 
inputStream using restart token: 0;0
CWLRB5612I: [10/21/12 09:01:53:908 EDT] Positioning SkipAndParallelStep batch data stream 
outputStream using restart token: 0;NOTSET

Conclusion

This article discussed the difference between the WebSphere Batch job scheduler and the enterprise scheduler, and showed how WebSphere Batch can be easily integrated with enterprise schedulers using the WSGrid client. Canceling and restarting jobs using WSGrid client was also explained.

Acknowledgements

The authors thank Sajan Sankaran and Abhilash Usha for reviewing this article and providing their invaluable input.


Downloadable resources


Related topics


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=861808
ArticleTitle=Modernized Java-based batch processing in WebSphere Application Server, Part 4: Integration with enterprise schedulers
publish-date=03202013