News
Abstract
By Narushi Onaya, IMS, IIDR for IMS, Spark on z/OS Consultant, IBM Systems Lab Services
In this blog post, I’d like to introduce various reference cases where IMS data is leveraged by SQL or another method. There are two ways to directly access the IMS database, and you can also access it indirectly by converting it to another format.
Content
Suppose an enormous volume of information is stored in your Information Management System (IMS) database. If you’re asked to extract all records after a certain date from that database, what kind of API would you think of? Wouldn't you be inclined to manipulate data using your favorite platform and programming language?
Tools that make hierarchical IMS data accessible by SQL have existed since the 1980s. However, there are issues with these: (1) they can be implemented only on the mainframe, (2) the data latency is not fresh and (3) the configuration tends to become complicated.
With more recent solutions, it’s possible to use the data on the IMS database in near real time in many platforms and programing languages. Implementation can also be simplified using a GUI, so you can overcome the conventional challenges.
In this blog post, I’d like to introduce various reference cases where IMS data is leveraged by SQL or another method. There are two ways to directly access the IMS database, and you can also access it indirectly by converting it to another format.
Directly accessing IMS database
We can already directly manipulate the IMS database using SQL. It’s possible to do with the IMS standard function called open database manager (ODBM) or using Rocket Data Virtualization (DVS). Direct access is especially effective in the application development phase, when you can create test data in Excel and insert it into the IMS database with SQL, select the IMS data and validate the outcome of programs. This approach has resulted in more productive development and testing. As another example, verification to access the IMS database from Spark on the mainframe has also been successful and is especially useful for machine learning.
Indirectly accessing after converting to another format
To indirectly access the IMS database, use the data change log of the IMS, called the X’99’ log. With this approach, the IMS database is not directly referred; instead, updated information written by IMS is used. There are two methods of data convey: one is to publish data as MQ message, and another is to replicate processed data to various database management systems.
The product used for indirectly accessing solutions is selected from IBM InfoSphere Data Replication (IIDR). Multiple products belong to IIDR, and there are three types: Change Data Capture (CDC) Replication, Q Replication and SQL Replication by architecture differences. When IMS becomes a data source, only CDC Replication is available.
IIDR transfers changed data by reading the log written by IMS and allows you to process it at the replicated destination. There are several patterns:
- Event publishing: IMS change data is published in raw or CSV format MQ message, and then you can process freely by using the MQ API in applications in various environments. It is published immediately after the IMS log is written, so you can construct a mechanism equivalent to the trigger function of a relational database (RDB). This solution is implemented InfoSphere Classic CDC for z/OS (CCDC), which belongs to IIDR.
- Replication to an RDB (heterogeneous): In Japan, IBM Db2 for z/OS and Db2 for LUW are used for replication target. In this case, the IMS side uses CCDC, and the target uses InfoSphere CDC for z/OS or CDC Replication Engine for Db2 for LUW. The CCDC provides metadata information that makes the hierarchical DB logically look like RDB. You can also replicate from IMS to Db2 for i, Oracle, Microsoft SQL Server, Sybase, Teradata or Informix by selecting suitable CDC products.
- Replication to IMS (homogeneous): By accessing the IMS database of the target side, it’s possible to resolve database contention, which is an issue in the method reading the IMS database directly. Otherwise, this replication is used for creating an acting system or disaster recovery system. This solution uses InfoSphere IMS Replication for z/OS on both the source and target IMS side. This product is not included in IIDR, but it is a superset of CCDC. Therefore, from InfoSphere IMS Replication for z/OS, you can replicate to an IMS database and Db2 table at the same time.
- Others: We have a proof of concept experience of replication from IMS to Db2 for z/OS and further replication with CDC Replication Engine for Netezza Technology from Db2 to IDAA. With the latest version of CDC, replication from IMS to Kafka is also possible using CCDC and CDC Replication Engine for Kafka.
In Japan, using these solutions in combination has been on the rise with many clients, especially in the banking industry, with financial core systems — for example, replication and publishing from one IMS system to Db2, MQ and IMS simultaneously.
Figure 1: Systems multiple replication and publishing are running
Services available from IBM Systems Lab Services
IBM Systems Lab Services provides technical support from design to implementation. My team has worked with various banks in Japan that use IMS data and needed to link this data to their information analysis systems and acting systems. IBM Systems Lab Services has developed automation and verification tools for testing software replication and built an environment to measure performance.
The system shown in Figure 1 is already in operation in a production environment, and there are several projects that are scheduled to be in operation soon in which we are supporting the effective use of their IMS data.
Since IMS is often used in mission-critical systems, the number of transactions is large, and strict requirements are desired for data consistency and throughput. And IBM Systems Lab Services has the knowledge and experience to address these needs.
Contact Lab Services today if you’re looking for more ways to take advantage of your IMS database.
Related Information
Was this topic helpful?
Document Information
Modified date:
05 December 2019
UID
ibm11125579