![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
IBM® WebSphere® Message Broker (hereafter called Message Broker) is an enterprise service bus solution where message flows route and transform incoming messages. Message flows take an input message, apply some transformation to it, and deliver it to an endpoint using the rich set of built-in Message Broker nodes.
For WebSphere on z/OS, the FileInput node and FileOutput node are a pair of built-in nodes to read and write files, respectively, using the Unix System Services (USS) file system. A third node, the FileRead node, reads USS files based on incoming messages. However, there is no direct support for traditional Multiple Virtual Storage (MVS) datasets, and therefore MVS datasets such as flat files, Partitioned Dataset (PDS) members, and Generation Data Group (GDG) members are not directly accessible from Message Broker.
Since many legacy systems have the bulk of their data stored in MVS datasets, this limitation presents a major problem. Two WebSphere Message Broker SupportPacs enable reading from and writing to MVS datasets:
This article describes a Java-based solution for reading MVS datasets that uses Java for z/OS (JZOS) libraries. The advantage of using a Java class to read a dataset is the great flexibility in post-processing, because after a record has been read from the dataset in a Java class, you can modify the record in many different ways.
The JZOS Toolkit is a Java Native Interface (JNI) library that consists of a Java archive (JAR) file and a native dynamic library. The Toolkit includes:
- Low-level wrappers for the z/OS C library functions.
- A factory class to create portable Readers and Writers to text files, including MVS datasets.
For more details, see JZOS Java Launcher and Toolkit overview and Chapter 4 in JZOS Batch Launcher and Toolkit function in IBM SDK for z/OS -- Installation and User's Guide.
The JAR file (ibmjzos.jar) is available in the USS file system at <JAVA_HOME>lib/ext/ibmjzos.jar.
An instance of the ZFile class is a thin wrapper around a C library file handle and can be used to access MVS datasets using a dataset or DD name. This class enables files to be opened
if they are MVS dataset names ( //X.Y.Z ), member names ( //X.Y.Z(M) ),
DD names ( //DD:INPUT01 ), or DD member names ( //DD:INPDS01(MEM01) ).
All dataset access methods supported by the C library fopen() routine (for MVS datasets) are supported. Posix (HFS/zFS) files are not supported.
Use this class and its methods to access and read MVS datasets. For samples, see Resources at the bottom of the article.
This section shows you how JZOS is wrapped in a Message Broker JavaCompute node.
More information on the JavaCompute node is available in the Message Broker
information center. For the sample implementation described in the previous section to work, ibmjzos.jar should be available on the CLASSPATH.
Similarly, with a JavaCompute node, you need to add this dependency of an external JAR file by copying the JAR file into the work path. For more information, see
Adding Java code dependencies.
Also, ibmjzos.jar should be available in the build path of the Java project associated with the JavaCompute node during development of the class.
Types of Java classes for the JavaCompute node
The Message Broker Toolkit provides templates of classes based on the expected functionality of the JavaCompute node. The templates include Create (to create new messages), Filter (to filter incoming messages), and Modify (to modify incoming messages). For more information, see Creating Java code for a JavaCompute node.
Which template you choose depends on the message flow -- for example, if the incoming message is to be propagated to subsequent nodes after the file transfer is done, or if no further action is to be taken after the file transfer is done and therefore no message needs to be propagated.
Java exception handling and logging
Exception handling and logging depend on the applicable standards at a given installation. For more information, see JavaCompute node exception handling and the Failure terminal.
This section shows you how to create the Java class to read an MVS dataset. The basic input for this class is the MVS dataset name (DSN). DSN names that are not enclosed in single quotes are automatically prefixed with the current user ID, as prescribed by the C library fopen() routine documentation.
For the sake of simplicity, specify that the DSN is passed as a simple string that conforms to a Message Broker comma separated value (CSV) message.
The message set is shown below. Similarly, the output is propagated as BLOB. The evaluate() method is implemented via the following steps:
- Get the MVS DSN from the incoming message.
- Open and read the MVS DSN using the ZFile class.
- Propagate to the out terminal until there are no more records.
Below is the try block of the evaluate method to access and read the MVS dataset:
Listing 1. Reading MVS dataset in JavaCompute node
try
{ //Some code removed for brevity
//Get reader of the MVS dataset
mvsDSN = "//'" + getMVSDatasetName(inAssembly) + "'";
BufferedReader in = FileFactory.newBufferedReader(mvsDSN);
String recordData = in.readLine() ;
while (recordData != null) {
outBody.setValue((recordData).getBytes());
outMessage.finalizeMessage(MbMessage.FINALIZE_NONE);
out.propagate(outAssembly) ;
recordData = in.readLine();
}
in.close();
alt.propagate(outAssembly);
} catch ( ... )
|
The previous section showed you how to develop a Java class to read an MVS dataset. This section shows you how to handle large datasets with tens of thousand of records. The flowchart below shows the steps to read and propagate the records in a large MVS dataset:
Figure 1. Reading a large MVS dataset
As shown in the flowchart above, you could have a message flow in which a message arrives at the JavaCompute node to read the MVS dataset, and then Message Broker schedules a transaction for this incoming message. But this transaction will persist until all records in the dataset are written to the out terminal, and therefore the one incoming message with the name of an MVS dataset will generate as many output messages as there are records in the dataset.
If these generated messages are wired to a MQOutput node, then there will be an equal number of MQPUTs, and at some point, the number of MQPUTs could reach the SYNCPOINT limit with more records still remaining in the dataset to be read. But since the Message Broker transaction is not yet over, a MQ commit would not have been issued, and therefore the message flow would fail with a
MQRC 2024 or MQRC_SYNCPOINT_LIMIT_REACHED message.
It is clear that a single transaction scheduled by Message Broker can generate multiple messages and cause an MQRC 2024 error when connected to an MQOutput node.
To avoid this problem, you can collect multiple records as a single message to be propagated, using an XML document that has batches of a certain number of records, as shown below:
Sample XML document for a collection of records
<mvsDSNRecords batch=”1”><mvsDSNRecord>This is the actual dataset content</mvsDSNRecord> <mvsDSNRecord>This is the actual dataset content</mvsDSNRecord> <mvsDSNRecord>This is the actual dataset content</mvsDSNRecord> <mvsDSNRecord>This is the actual dataset content</mvsDSNRecord> </mvsDSNRecords> |
This XML document can be put into the message queue as one message. Therefore, in a single MQPUT, the XML document enables more than one record from the MVS dataset to be put. With this approach, the number of messages propagated is reduced and so are the MQPUTs, and therefore the dataset will be read completely by the time the SYNCPOINT limit is reached.
This article showed you how to read an MVS dataset using Java and Message Broker. With this approach, existing batch jobs on z/OS can create datasets, and Message Broker can read them without any conversion in between. Also, using Java lets you apply any amount of programming logic to alter the records before writing them out.
| Description | Name | Size | Download method |
|---|---|---|---|
| Code sample | ReadMVS.zip | 440 KB | HTTP |
Information about download methods
- WebSphere Message Broker resources
- Youtube tutorial: Integrating Microsoft .NET code in a WebSphere Message Broker V8 message flow
This five-minute youtube tutorial shows you how simple it is to use WebSphere Message Broker V8 to build a message flow that includes Microsoft .NET code. Microsoft Visual Studio is used to build .NET code in C#, which is then integrated into a message flow using Message Broker and an HTTP RESTful interface. -
WebSphere Message Broker V8 Development Training
In this IBM Training course, you will learn about the components of the WebSphere Message Broker development and runtime environments. The course also explores message flow problem determination, and shows you how to construct message flows that use ESQL, Java, and PHP to transform messages. - WebSphere Message Broker V8 information center
A single Web portal to all WebSphere Message Broker V8 documentation, with conceptual, task, and reference information on installing, configuring, and using your WebSphere Message Broker environment. - WebSphere Message Broker developer resources page
Technical resources to help you use WebSphere Message Broker for connectivity, universal data transformation, and enterprise-level integration of disparate services, applications, and platforms to power your SOA. - WebSphere Message Broker product page
Product descriptions, product news, training information, support information, and more. - Download free trial version of WebSphere Message Broker
WebSphere Message Broker is an ESB built for universal connectivity and transformation in heterogeneous IT environments. It distributes information and data generated by business events in real time to people, applications, and devices throughout your extended enterprise and beyond. - WebSphere Message Broker documentation library
WebSphere Message Broker specifications and manuals. - WebSphere Message Broker forum
Get answers to your technical questions and share your expertise with other WebSphere Message Broker users. - WebSphere Message Broker support page
A searchable database of support problems and their solutions, plus downloads, fixes, and problem tracking.
- Youtube tutorial: Integrating Microsoft .NET code in a WebSphere Message Broker V8 message flow
- WebSphere resources
- developerWorks WebSphere developer resources
Technical information and resources for developers who use WebSphere products. developerWorks WebSphere provides product downloads, how-to information, support resources, and a free technical library of more than 2000 technical articles, tutorials, best practices, IBM Redbooks, and online product manuals. - developerWorks WebSphere application integration developer resources
How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application integration and business integration solutions. - Most popular WebSphere trial downloads
No-charge trial downloads for key WebSphere products. - WebSphere forums
Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users. - WebSphere on-demand demos
Download and watch these self-running demos, and learn how WebSphere products and technologies can help your company respond to the rapidly changing and increasingly complex business environment. - developerWorks WebSphere weekly newsletter
The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing. - WebSphere-related books from IBM Press
Convenient online ordering through Barnes & Noble. - WebSphere-related events
Conferences, trade shows, Webcasts, and other events around the world of interest to WebSphere developers.
- developerWorks WebSphere developer resources
- developerWorks resources
- Trial downloads for IBM software products
No-charge trial downloads for selected IBM® DB2®, Lotus®, Rational®, Tivoli®, and WebSphere® products. - developerWorks blogs
Join a conversation with developerWorks users and authors, and IBM editors and developers. - developerWorks cloud computing resources
Access the IBM or Amazon EC2 cloud, test an IBM cloud computing product in a sandbox, see demos of cloud computing products and services, read cloud articles, and access other cloud resources. - developerWorks tech briefings
Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most challenging software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide. - developerWorks podcasts
Listen to interesting and offbeat interviews and discussions with software innovators. - developerWorks on Twitter
Check out recent Twitter messages and URLs. - IBM Education Assistant
A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.
- Trial downloads for IBM software products
Nagesh Subrahmanyam Is an Advisory Systems Analyst with Global Business Services in India. He has 10 years of experience with System z technologies and has a keen interest in using open-source and XML technologies with System z. You can contact Nagesh at nsubrahm@in.ibm.com.
