Modernized Java-based batch processing in WebSphere Application Server, Part 4
Integration with enterprise schedulers
This content is part # of # in the series: Modernized Java-based batch processing in WebSphere Application Server, Part 4
This content is part of the series:Modernized Java-based batch processing in WebSphere Application Server, Part 4
Stay tuned for additional content in this series.
IBM WebSphere Application Server V8.5 and later provides an execution platform for Java-based batch applications. In addition to providing a rich programming model and advanced features, such as parallel processing, skip record processing, retry step processing, and COBOL support, it also offers enterprise “qualities” — such as availability, recoverability, scalability, and performance — to batch programs. Coupled with WebSphere Application Server support, WebSphere Batch can be an attractive enterprise batch solution option.
Batch frameworks ease the development of discrete batch jobs, but they don’t provide any choreography between various jobs. Such enterprise wide job choreography is traditionally performed using enterprise schedulers like IBM Tivoli® Workload Scheduler, Control-M, or ASG-Zeke. Enterprise schedulers can provide centralized management of batch jobs running across different operating system platforms, along with advanced features like managing job dependencies, job monitoring and alert mechanisms, automated job restart, and recovery. Enterprise schedulers and batch frameworks thus provide complementary features, and together provide a complete enterprise batch solution.
WebSphere Batch is essentially an endpoint at which batch jobs can be scheduled by the enterprise schedulers. Using details and examples, this article describes how WebSphere Batch can be seamlessly integrated with enterprise schedulers. This integration enables the enterprise scheduler to be the meta-scheduler and provide inputs to the WebSphere Batch job scheduler. The job scheduler, in turn, apprises the enterprise scheduler of job state information and updates the execution log.
For the sake of simplicity, this article does not address security aspects or integration with the z/OS® environment. See Related topics for information on the System z integration scenario.
Integrating WebSphere Batch with enterprise schedulers
In order to make WebSphere Batch and an enterprise scheduler work together seamlessly, WebSphere Batch provides these features:
- Multi-platform support to enable schedulers running on different platforms to use it.
- Ability to submit and cancel the jobs from the enterprise scheduler.
- Continuous feedback and final return status to the enterprise scheduler.
As mentioned in the previous articles in this series, WebSphere Batch feature provides web, EJB, and web service interfaces for job management. But in order to meet the above integration requirement, WebSphere batch (on distributed platforms) provides WSGrid, a JMS-based integration solution.
The enterprise scheduler interacts with WebSphere Batch through a WSGrid client, a platform-specific batch program; WSGrid.bat on Windows® and WSGrid.sh on UNIX® platforms. The WSGrid client is basically a JMS program that places requests on the WebSphere Application Server job scheduler. The job scheduler server, enhanced with an MDB application, listens for job requests on a pre-configured SIBus queue. The MDB application then passes the request to the WebSphere Application Server job scheduler, which then processes the request normally. The job scheduler keeps emitting the logs on the pre-determined queue. The WSGrid client goes between the enterprise scheduler and the queues on the WebSphere Application Server job scheduler machine and keeps updating the enterprise scheduler. Using WSGrid enables this integration to be as simple as calling a shell program from the enterprise scheduler. Figure 1 illustrates this integration.
Figure 1. Integration of Enterprise Scheduler with WebSphere Batch
The sequence of activities in a typical WSGrid integration is:
- The enterprise scheduler calls the WSGrid client batch file.
- The WSGrid client puts a JMS Request message on the pre-defined queue on the WebSphere job scheduler.
- Listening to the request queue, the MDB application takes up this request message and uses it to submit the job.
- The job is then executed on the server end points, as it normally would had it been submitted from the Job Management Console.
- The output from the server is given back to the enterprise scheduler via the response queue.
- At the completion of the job, the end status is sent to WSGrid, and then the connection to the WebSphere job scheduler server is closed. While the job is in progress, WSGrid keeps getting the output which is in turn consumed by the enterprise scheduler
Setting up WSGrid
Using the infrastructure setup and project developed in Part 3 of this series, you would follow these steps to set up WSGrid:
- Enhance the WebSphere Scheduler server with the MDB application, and create the JMS infrastructure. This can be done using the wsgridConfig.py script:
where <providerList> means “hostname,portnumber,” and portnumber being the SIB_ENDPOINT_ADDRESS port of the scheduler server. (See Related topics)
wsadmin.sh -f wsgridConfig.py -install -node <nodeName> -server <serverName> -providers <providerList>
Listing1 shows the output from a sample run.
wsadmin -f wsgridConfig.py -install -providers "Anish-PC,7276" -node Anish-PCNode01 -server server1 WASX7209I: Connected to process "dmgr" on node Anish-PCCellManager01 using SOAP connector; The type of process is: DeploymentManager . . . . . wsgridConfig.py is performing install using target cluster or server : /Node:Anish-PCNode01/Server:server1/ MDI application name : JobSchedulerMDI JMS connection factory : com.ibm.ws.grid.ConnectionFactory JMS activation spec : com.ibm.ws.grid.ActivationSpec JMS input queue name : com.ibm.ws.grid.InputQueue JMS output queue name : com.ibm.ws.grid.OutputQueue Installing WSGrid JMS file store root : /tmp\JobSchedulerBus SIB identifier : JobSchedulerBus endpoint provider list : Anish-PC:7276
As you can see, this script installed the MDB application and created the JMS infrastructure on the scheduler server. Let’s look at the various components that were installed by this script:
- The MDB application JobSchedulerMDI acts as the interface to the job scheduler. The WSGrid client uses this MDB application to exchange messages with job scheduler.
- The bus JobSchedulerBus, built on WebSphere Application Server SIBus, is created on the job scheduler server and it hosts:
- Request queue com.ibm.ws.grid.InputQueue: The MDB listens on this queue and takes job submission requests.
- Reply queue com.ibm.ws.grid.OutputQueue: The MDB responds on this queue with job progress logs and status.
- com.ibm.ws.grid.ActivationSpec: Used to bind JobSchedulerMDI MDB to the request queues.
- Because WebSphere Application Server is already installed, the enterprise
scheduler can call the WSGrid client. WSGrid client takes in a control
file as parameter; this is a simple properties file that contains information about the scheduler location. Listing 2 shows sample entries for the control file wsgrid.cntl.
scheduler-host=192.168.1.107 scheduler-port=9080 debug=false
Once WSGrid is configured, you can launch it via the WSGrid.sh/bat shell script. Listing 3 shows a sample of how to invoke the WSGrid client using the script WSGrid.bat and its output.
WSGrid.bat wsgrid.cntl SkipAndParallelJob.xml WSGrid Version WAS85.WBAT [gm1217.01] 2012-04-24 16:15:25 CONTROL: scheduler-port=9080 CONTROL: debug=true CONTROL: scheduler-host=192.168.1.107 DEBUG: Obtain JES jobid in checkEnvironment DEBUG: No JES job id - running in shell. DEBUG: creating message lisenter with host=192.168.1.107 and port=9080 DEBUG: JobMessenger.decode: pw not encoded DEBUG: JobMessenger host=192.168.1.107 DEBUG: JobMessenger port=9080 DEBUG: Using correlator value= a479ea48-65a5-4fd8-852c-f2bdaf258ad3-b10ac453-bcb7-4467 -b49a-963c52380c50 DEBUG: JobMessenger: opening communications DEBUG: JobSchedulerServiceProxy: user=null DEBUG: JobSchedulerServiceProxy: protocol=http . . . . . . System.out: [10/01/12 19:30:52:040 EDT] ********** End SkipAndParallelJob:00257:00259 log ********** CWLRB5630I: [10/01/12 19:30:52:047 EDT] Step SkipAndParallelStep completes normally: ended normally CWLRB3800I: [10/01/12 19:30:52:053 EDT] Job [SkipAndParallelJob:00257] ended normally. CWLRB3880I: Job [SkipAndParallelJob:00257] ending status: RC=0
The logs in Listing 3 show how WSGrid provides continuous feedback to the enterprise scheduler. Aside from job submission, WSGrid also enables the canceling and restarting of WebSphere Batch jobs. This ensures that if the WSGrid job is cancelled, the WebSphere Batch job is cancelled also, thus ensuring synchronization of the two jobs.
On distributed operating systems, cancelling the WSGrid client also triggers the cancelling of the job running by WebSphere Batch. The canceled jobs end in restartable mode (Listing 4).
CWLRB5630I: [10/21/12 06:22:38:250 EDT] Step SkipAndParallelStep completes [cancelled on iteration 5507]: cancelled ........................... ........................... CWLRB2580I: [10/21/12 06:22:38:360 EDT] [10/21/12 06:22:38:360 EDT] Job [SkipAndParallelJob:00272:00273] Step [SkipAndParallelStep] completed [cancelled]. CWLRB3820I: [10/21/12 06:22:38:360 EDT] Job [SkipAndParallelJob:00272:00273] ended abnormally [cancelled]. CWLRB5594I: [10/21/12 06:22:38:360 EDT] Step SkipAndParallelStep execution is complete: Cancelled or Stopped CWLRB3860W: [10/21/12 06:22:38:375 EDT] Job [SkipAndParallelJob:00272:00273] ended abnormally [and is restartable]
The WSGrid program can be submitted with an additional parameter that specifies the name of the restart file. On execution, the WSGrid program creates the restart file containing the job ID. This restart file can be used by WSGrid and its calling enterprise scheduler to identify the instance of the job that needs to be restarted.
WSGrid.bat wsgrid.cntl SkipAndParallelJob.xml restart.properties
Notice the additional parameter restart.properties, which specifies the name of the restart file. When the job ends, a file named restart.properties is created, containing this line:
To restart the job, use this command:
WSGrid.bat wsgrid.cntl restart.properties
The job specified in restart.properties will be restarted (Listing 5).
WLRB5618I: [10/21/12 09:01:53:908 EDT] Initializing step SkipAndParallelStep batch data stream outputStream CWLRB5620I: [10/21/12 09:01:53:908 EDT] Opening step SkipAndParallelStep batch data stream outputStream CWLRB5612I: [10/21/12 09:01:53:908 EDT] Positioning SkipAndParallelStep batch data stream inputStream using restart token: 0;0 CWLRB5612I: [10/21/12 09:01:53:908 EDT] Positioning SkipAndParallelStep batch data stream outputStream using restart token: 0;NOTSET
This article discussed the difference between the WebSphere Batch job scheduler and the enterprise scheduler, and showed how WebSphere Batch can be easily integrated with enterprise schedulers using the WSGrid client. Canceling and restarting jobs using WSGrid client was also explained.
The authors thank Sajan Sankaran and Abhilash Usha for reviewing this article and providing their invaluable input.
- WebSphere Application Server V8.5 Information Center
- Developing batch applications for WebSphere Application Server 8.5
- IBM Redbook: WebSphere Application Server V8.5 Concepts, Planning, and Design Guide
- Part 1: Introducing Modern Batch and the compute-intensive programming model
- Part 2: Transaction batch programming model
- Part 3: Enterprise batch processing
- Job scheduler integration with external schedulers
- Submitting jobs from an external job scheduler
- Search Techdocs for WP101783 - Integrating zOS Scheduler with Distributed WCG using WSGRID
- Download WebSphere Application Server for Developers V8.5
- Download WebSphere Application Server V8.5 trial
- Presentation: IBM WebSphere Application Server Batch Applications