WebSphere Message Broker on z/OS : Reading MVS datasets using Java for z/OS

Traditional MVS datasets are not directly accessible from WebSphere Message Broker. This article shows you how to read them using Java for z/OS (JZOS) libraries with WebSphere Message Broker. Using a Java class to read records from a dataset provides much more flexibility in post-processing, enabling you to modify the records in many different ways.

Share:

Nagesh Subrahmanyam (nsubrahm@in.ibm.com), Advisory Systems Analyst, Global Business Services, IBM

Photo of Nagesh SubrahmanyamNagesh Subrahmanyam Is an Advisory Systems Analyst with Global Business Services in India. He has 10 years of experience with System z technologies and has a keen interest in using open-source and XML technologies with System z. You can contact Nagesh at nsubrahm@in.ibm.com.



28 March 2012

Introduction

IBM® WebSphere® Message Broker (hereafter called Message Broker) is an enterprise service bus solution where message flows route and transform incoming messages. Message flows take an input message, apply some transformation to it, and deliver it to an endpoint using the rich set of built-in Message Broker nodes.

For WebSphere on z/OS, the FileInput node and FileOutput node are a pair of built-in nodes to read and write files, respectively, using the Unix System Services (USS) file system. A third node, the FileRead node, reads USS files based on incoming messages. However, there is no direct support for traditional Multiple Virtual Storage (MVS) datasets, and therefore MVS datasets such as flat files, Partitioned Dataset (PDS) members, and Generation Data Group (GDG) members are not directly accessible from Message Broker.

Since many legacy systems have the bulk of their data stored in MVS datasets, this limitation presents a major problem. Two WebSphere Message Broker SupportPacs enable reading from and writing to MVS datasets:

A Java® solution

This article describes a Java-based solution for reading MVS datasets that uses Java for z/OS (JZOS) libraries. The advantage of using a Java class to read a dataset is the great flexibility in post-processing, because after a record has been read from the dataset in a Java class, you can modify the record in many different ways.

JZOS Toolkit

The JZOS Toolkit is a Java Native Interface (JNI) library that consists of a Java archive (JAR) file and a native dynamic library. The Toolkit includes:

  • Low-level wrappers for the z/OS C library functions.
  • A factory class to create portable Readers and Writers to text files, including MVS datasets.

For more details, see JZOS Java Launcher and Toolkit overview and Chapter 4 in JZOS Batch Launcher and Toolkit function in IBM SDK for z/OS -- Installation and User's Guide.

The JAR file (ibmjzos.jar) is available in the USS file system at <JAVA_HOME>lib/ext/ibmjzos.jar.

JZOS javadoc

An instance of the ZFile class is a thin wrapper around a C library file handle and can be used to access MVS datasets using a dataset or DD name. This class enables files to be opened if they are MVS dataset names ( //X.Y.Z ), member names ( //X.Y.Z(M) ), DD names ( //DD:INPUT01 ), or DD member names ( //DD:INPDS01(MEM01) ). All dataset access methods supported by the C library fopen() routine (for MVS datasets) are supported. Posix (HFS/zFS) files are not supported.

Use this class and its methods to access and read MVS datasets. For samples, see Resources at the bottom of the article.

Using the JavaCompute node

This section shows you how JZOS is wrapped in a Message Broker JavaCompute node.

Adding code dependencies

More information on the JavaCompute node is available in the Message Broker information center. For the sample implementation described in the previous section to work, ibmjzos.jar should be available on the CLASSPATH. Similarly, with a JavaCompute node, you need to add this dependency of an external JAR file by copying the JAR file into the work path. For more information, see Adding Java code dependencies. Also, ibmjzos.jar should be available in the build path of the Java project associated with the JavaCompute node during development of the class.

Types of Java classes for the JavaCompute node

The Message Broker Toolkit provides templates of classes based on the expected functionality of the JavaCompute node. The templates include Create (to create new messages), Filter (to filter incoming messages), and Modify (to modify incoming messages). For more information, see Creating Java code for a JavaCompute node.

Which template you choose depends on the message flow -- for example, if the incoming message is to be propagated to subsequent nodes after the file transfer is done, or if no further action is to be taken after the file transfer is done and therefore no message needs to be propagated.

Java exception handling and logging

Exception handling and logging depend on the applicable standards at a given installation. For more information, see JavaCompute node exception handling and the Failure terminal.

Building the Java class

This section shows you how to create the Java class to read an MVS dataset. The basic input for this class is the MVS dataset name (DSN). DSN names that are not enclosed in single quotes are automatically prefixed with the current user ID, as prescribed by the C library fopen() routine documentation.

The evaluate() method

For the sake of simplicity, specify that the DSN is passed as a simple string that conforms to a Message Broker comma separated value (CSV) message. The message set is shown below. Similarly, the output is propagated as BLOB. The evaluate() method is implemented via the following steps:

  1. Get the MVS DSN from the incoming message.
  2. Open and read the MVS DSN using the ZFile class.
  3. Propagate to the out terminal until there are no more records.

Below is the try block of the evaluate method to access and read the MVS dataset:

Listing 1. Reading MVS dataset in JavaCompute node
try 
{   //Some code removed for brevity 
    //Get reader of the MVS dataset 
    mvsDSN = "//'" + getMVSDatasetName(inAssembly) + "'"; 
    BufferedReader in = FileFactory.newBufferedReader(mvsDSN); 
    String recordData = in.readLine() ; 
    while (recordData != null) { 
        outBody.setValue((recordData).getBytes()); 
        outMessage.finalizeMessage(MbMessage.FINALIZE_NONE); 
        out.propagate(outAssembly) ; 
        recordData = in.readLine(); 
        } 
        in.close(); 
        alt.propagate(outAssembly); 
} catch ( ... )

Handling large datasets

The previous section showed you how to develop a Java class to read an MVS dataset. This section shows you how to handle large datasets with tens of thousand of records. The flowchart below shows the steps to read and propagate the records in a large MVS dataset:

Figure 1. Reading a large MVS dataset
Reading a large MVS dataset

Issue with MQOutput node

As shown in the flowchart above, you could have a message flow in which a message arrives at the JavaCompute node to read the MVS dataset, and then Message Broker schedules a transaction for this incoming message. But this transaction will persist until all records in the dataset are written to the out terminal, and therefore the one incoming message with the name of an MVS dataset will generate as many output messages as there are records in the dataset.

If these generated messages are wired to a MQOutput node, then there will be an equal number of MQPUTs, and at some point, the number of MQPUTs could reach the SYNCPOINT limit with more records still remaining in the dataset to be read. But since the Message Broker transaction is not yet over, a MQ commit would not have been issued, and therefore the message flow would fail with a MQRC 2024 or MQRC_SYNCPOINT_LIMIT_REACHED message.

Workaround

It is clear that a single transaction scheduled by Message Broker can generate multiple messages and cause an MQRC 2024 error when connected to an MQOutput node. To avoid this problem, you can collect multiple records as a single message to be propagated, using an XML document that has batches of a certain number of records, as shown below:

Sample XML document for a collection of records
<mvsDSNRecords batch=”1”><mvsDSNRecord>This is the actual dataset content</mvsDSNRecord>
<mvsDSNRecord>This is the actual dataset content</mvsDSNRecord>
<mvsDSNRecord>This is the actual dataset content</mvsDSNRecord>
<mvsDSNRecord>This is the actual dataset content</mvsDSNRecord>
</mvsDSNRecords>

This XML document can be put into the message queue as one message. Therefore, in a single MQPUT, the XML document enables more than one record from the MVS dataset to be put. With this approach, the number of messages propagated is reduced and so are the MQPUTs, and therefore the dataset will be read completely by the time the SYNCPOINT limit is reached.

Conclusion

This article showed you how to read an MVS dataset using Java and Message Broker. With this approach, existing batch jobs on z/OS can create datasets, and Message Broker can read them without any conversion in between. Also, using Java lets you apply any amount of programming logic to alter the records before writing them out.


Download

DescriptionNameSize
Code sampleReadMVS.zip440 KB

Resources

  • WebSphere Message Broker resources
  • WebSphere resources
    • developerWorks WebSphere developer resources
      Technical information and resources for developers who use WebSphere products. developerWorks WebSphere provides product downloads, how-to information, support resources, and a free technical library of more than 2000 technical articles, tutorials, best practices, IBM Redbooks, and online product manuals.
    • developerWorks WebSphere application integration developer resources
      How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application integration and business integration solutions.
    • Most popular WebSphere trial downloads
      No-charge trial downloads for key WebSphere products.
    • WebSphere forums
      Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users.
    • WebSphere on-demand demos
      Download and watch these self-running demos, and learn how WebSphere products and technologies can help your company respond to the rapidly changing and increasingly complex business environment.
    • developerWorks WebSphere weekly newsletter
      The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing.
    • WebSphere-related books from IBM Press
      Convenient online ordering through Barnes & Noble.
    • WebSphere-related events
      Conferences, trade shows, Webcasts, and other events around the world of interest to WebSphere developers.
  • developerWorks resources
    • Trial downloads for IBM software products
      No-charge trial downloads for selected IBM® DB2®, Lotus®, Rational®, Tivoli®, and WebSphere® products.
    • developerWorks blogs
      Join a conversation with developerWorks users and authors, and IBM editors and developers.
    • developerWorks cloud computing resources
      Access the IBM or Amazon EC2 cloud, test an IBM cloud computing product in a sandbox, see demos of cloud computing products and services, read cloud articles, and access other cloud resources.
    • developerWorks tech briefings
      Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most challenging software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide.
    • developerWorks podcasts
      Listen to interesting and offbeat interviews and discussions with software innovators.
    • developerWorks on Twitter
      Check out recent Twitter messages and URLs.
    • IBM Education Assistant
      A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere, Java technology
ArticleID=806465
ArticleTitle=WebSphere Message Broker on z/OS : Reading MVS datasets using Java for z/OS
publish-date=03282012