IBM Support

Access Methods

White Papers


Abstract

DVM for z/OS technology provides multiple methods for client applications to access data. These access methods allow external applications to seamlessly make requests for read or write operations of underlying data, regardless of location or format.

Content

Interface methods for client access
DVM for z/OS technology provides multiple methods for client applications to access data. These access methods allow external applications to seamlessly make requests for READ or WRITE operations of underlying data, regardless of location or format, shown in Figure 1.Screen Shot 2021-05-23 at 7.36.49 PM
 
Figure 1 - DVM for z/OS endpoint architecture
Organizations typically have a range of applications in use depending on the business objective. Traditional mainframe applications, such as Cobol are prevalent and commonly associated with any digital transformation or modernization initiative where DVM for z/OS exists.
Modernization commonly involves transforming traditional programs into a more portable or updated interface driven by Java by using the DVM JDBC driver. Additionally, native and commercial applications leverage both DVM ODBC and JDBC drivers.
Mobile or Cloud applications can interact with data that use the web browsers and mobile applications over http/https to service Cloud environments. In each instance, for any application, DVM for z/OS provides the needed translation and routing required for access through specific interfaces; DVM Server, JDBC Gateway, DVM WOLA Service Provider, DVM DSCLIENT, Interactive System Productivity Facility (ISPF). Common Client access methods are shown in Table 1.
Client Types Access Method
C, C++ based applications ODBC driver to DVM Server
Java applications JDBC driver to DVM Server
JDBC driver to DVM Studio to DVM Server
Mainframe applications
Peer DVM instance Servers
IDF using DRDA
Mainframe applications  IDF using DRDA
Web browser (HTTP and HTTPS) z/OS Connect EE to DVM WOLA service provider
Web browser (HTTPP JDBC Gateway to DRDA
ISPF DVM Server
RDBMS (DRDA Supported)
Db2 distributed family
Oracle (DRDA)
SQL Server
IBM Informix
...
DRDA to DrDA AR to DVM Server
RDBMS (non-DRDA)
Postgres
Oracle
Apache Hive
SQL Server
MySQL
...
JDBC driver to JDBC Gateway Server 
JDBC Gateway Server to DVM DS Client
DS Client to mainframe applications or Db2 UDTF
Table 1 - DVM for z/OS access methods
Standard access
DVM for z/OS provides an SQL API, Java API, and both ODBC and JDBC drivers, to access data using the DVM Server. The JDBC and ODBC drivers are on IBM's Fix Central repository and easily downloadable for use. Once the drivers are downloaded to a Workstation, they can be uncompressed and installed.
JDBC/ODBC (including security or Kerberos)
Client applications that use the JDBC driver can connect to a Db2 subsystem or other data sources either on or off the mainframe and access a remote system catalog or defined tables.
A minimum JDBC connection string contains the hostname, port, and DBTY.
jdbc:rs:dv://,hostname>:<port nmber>;DBTY=DVS
The JDBC driver dv-jdbc-3.1.201912091012.jar requires log4j-api-<version#>.jar and log4j-core-<version#>.jar files, and is available to use with any application by using JDBC.
An IBM Db2 QMF for Workstation contains common JDBC connection string elements.
jdbc:rs:dv://<hostname>:<port number>;DBTY=DVS;Subsystem=NONE
The DVM Server can also be used as an ODBC Type 4 connection for IBM Db2 Connect. In this case, the DBTY and Subsystem parameters reference 'DB2' - jdbc:rs:dv://<hostname>:<port number>;DBTY=DB2;Subsystem=<DB2 CSSID>
ODBC (including security or Kerberos)
The ODBC driver is available for both 32-bit and 64-bit environments across Red Hat Linux, and SUSE Linux distributions. The DVM Studio is the quickest approach for accessing, connecting, and discovering virtualized data assets, however, other SQL-based client programs offering support for ODBC or JDBC connectivity are valid, such as MS-Excel.
Using MS-Excel
Screen Shot 2021-05-23 at 7.44.21 PM
Figure 2 - MS-Excel querying DVM
Using the DVM Studio to enhance programmer productivity
Once the underlying data is formally mapped and made available using a virtual source library, data can be easily discovered and previewed using default connectivity to the DVM Server. All virtual data assets provisioned through the DVM Server can be used as a reference for generating customized code snippets across available modern programming languages and leveraged for accelerating application development. Figure 3 shows the menu option in the DVM Studio to generate programming code from SQL.
Screen Shot 2021-05-23 at 7.44.53 PM
Figure 3 - Generating Sample Java code
Generated SQL code is saved in the current Workspace in your eclipse environment, shown in Figure 4.  By default, this stored in the user-profile: C:\Users<your-id>\dvm_workspace\Data Virtualization Manager\src.

Screen Shot 2021-05-23 at 7.46.22 PM
Figure 4 - Generated Java sample code
Java API
DVM for z/OS provides a Java application programming interface (API) that contains a list of classes, interfaces with methods, fields, and constructors. These prewritten classes provide significant functionality for application developers.
Metadata
The Java API allows external tools and applications to discover DVM Server objects like tables, views, as well as other object descriptions, such as column names, column data type, etc. Use the standard DatabaseMetaData API to access the DVM Server metadata, such as database product name, version, driver name, name of the total number of tables and views. If you need more information about the DatabaseMetaData Interface and the methods it offers, search on the web for the Java official documentation. Example 1 displays some standard configuration parameters for generating Java code snippets.
DatabaseMetaData databaseMetadata = conn.getMetaData();
String catalog = conn.getCatalog(); String schemaPattern = null;
String tableNamePattern = "%";
String[] types = null;
ResultSet tables = databaseMetadata.getTables(catalog, schemaPattern, tableNamePattern, types;
Example 1 - Java code snippet to retrieve all from a DVM Server
Where:
  • tableNamePattern is used to filter objects based on their name
  • types can be used to filter objects based on their type, such as TABLE objects String[] types = {"TABLE"};
The DatabaseMetaData object is obtained using the getMetaData() method of a Connection class. The getTables() method of the DatabaseMetaData interface is used to list in Example 2.
DatabaseMetaData databaseMetadata = conn.getMetaData();
String catalog = conn.getCatalog();
String schemaPattern = null;
String tableNamePattern = "%";
String colNamePattern = "%";
ResultSet tables = databaseMetadata.getColumns(catalog, schemaPattern, tableNamePattern, colNamePattern;
Example 2 - List all columns for all objects in the DVM Server
Where:
  • tableNamePattern can be used to filter objects based on their name
  • colNamePattern can be used to filter columns based on their name
The DatabaseMetaData object is obtained invoking the getMetaData() method of the Connection class. The list of columns is obtained from the getColumns() method of the DatabaseMetaData interface passing all required filters.
The DVM Server metadata using the Java API can generate reusable code snippets for all available virtualized objects. Replace XXX and YYY to match your environment for hostname/IP and Port number respectively. Replace AAA and BBB with valid credentials needed for the getConnection() method.
Include these DVM for z/OS JDBC jar files in the Java application classpath:
  • dv-jdbc-[version #].jar: DVM JDBC driver core implementation file
  • log4j-api-[version #].jar: The logging framework API file
  • log4j-core-[version #].jar: The logging framework implementation file.
  • log4j2.xml: A sample logging configuration file.
Sample reusable code is available in the Appendix of this Redbook.
DS Client
The DS Client high-level API can be called from within more traditional mainframe languages such as COBOL, Natural, or PL/I. This is a high-level API that allows an application running on z/OS to use a call-level interface to communicate with the DVM Server to process SQL requests and retrieve results buffered in a 64-bit shared memory object.
CICS and other TXN or workload balancers
To use the DsClient API with CICS, the CICS started task JCL, the program list table (PLT), and the DFHCSD file needs modification.
The DVM.AVZ.SAVZCLOD library needs to be added to the DFHRPL concatenation in each CICS region connecting to the DVM Server for the SYSP.PROCLIB(CICSBS54) data set. Figure 4 shows this by using the DVM ISPF panel.
Screen Shot 2021-05-23 at 7.49.14 PM
Figure 4 - Add the DVM.AVZ.SABZCLOD library when using CICS
Update and assemble the CICS program list table/program initialized (PLTPI) list for the DS Client task-related user exit. The entry for the AVZXMTRI program must follow the first DFHDELIM entry in the PLTPI list to ensure that the AVZXMTRI program is executed during the second stage of the CICS PLTI process.
 
  1. Locate the first DFHDELIM entry in the PLTP1 list
    DFHPLT TYPE=ENTRY, PROGRAM=DFHDELIM
  2. Insert the AVZXMTRI list as the second entry for the DS Client task-related user exit
    DFHPLT TYPE=ENTRY, PROGRAM=AVZXMTRI
  3. Run your CICS assembly job
Update the DFHCSD file:
 
  1. For each CICS region, modify and submit the AVZCICSD job that is in hlq.SAVZCNTL data set
  2. Update LIST(YOURLIST) to match the start-up group list for the CICS region
Restart CICS and check for the following message in the CICS job log:
AVZ4459I CICSE DS Client exit program AVZCTRUE is enabledAdd body text.
Using Data Virtualization Manager in a COBOL program
Assume now that you have a VSAM data set that is virtualized as a virtual table STAFFVS, and you need to access it from a program that is written in a high-level language, such as COBOL from CICS. DVM for z/OS offers a DS Client high-level language API through the program AVZCLIEN through this sequence:
  1. Open a connection to the DVM Server
  2. Send the SQL command to the DVM Server
  3. Receive the complete query results from the issues SQL statement
  4. Close the connection to the DVM Server
The program’s data structures
The COBOL program requires a DS Client Control Block (DVCB) to be defined in its working storage using a copybook. See “Example: DS Client control block (DVCB)” on page 70 of Chapter 5 in this Redbook.
AVZCLIEN Program preparation
The program is compiled without any specific options and linked as follows:
//SYSLIB DD DSN=DVM.SAVZLOAD,DISP=SHR
//SYSLMOD DD DSN=APPL.LOADLIB,DISP=SHR
//SYSIN DD * 
INCLUDE SYSLIB(AVZCLIEN)
NAME module (R)
AVZCLIEN can be found in the SAVZLOAD library.
SQL Command Writing
Instrument the SQL command in the COBOL program using COBOL character functions.
STRING
"SELECT STAFFVS_KEY_ID,
"DELIMITED BY SIZE" STAFFVS_DATA_NAME,
"DELIMITED BY SIZE" STAFFVS_DATA_DEPT,
"DELIMITED BY SIZE" STAFFVS_DATA_JOB,
"DELIMITED BY SIZE" STAFFVS_DATA_YRS,
"DELIMITED BY SIZE" FROM STAFFVS,
"DELIMITED BY SIZE" INTO SQL-TEXT.
The example statement results in dynamic SQL executed where STAFFVS is the virtual table requested.
SELECT STAFFVS_KEY_ID, STAFFVS_DATA_NAME, STAFFVS_DATA_DEPT, STAFFVS_DATA_JOB, STAFFVS_DATA_YRS 
FROM STAFFVS;
Refer to the Developer's Guide to get more information about developing your Cobol application using the DSclient API.
REST and SOAP Web Service interfaces
IBM z/OS Connect Enterprise Edition
Cloud and mobile applications have changed the way enterprises and systems interact. RESTful APIs that use JSON messages are the predominant standards for new application development. IBM z/OS Connect Enterprise Edition (zCEE) provides a framework that enables z/OS based programs and data to participate fully in the new API economy for mobile and cloud applications.
z/OS Connect EE provides RESTful API access to z/OS applications and data hosted in subsystems such as CICS, IMS, IBM MQ, and Db2 for z/OS. Moreover, the combination of z/OS Connect and DVM enables direct RESTful API access to perform Select, Insert, Update and Delete operations to traditional mainframe data such as VSAM, Sequential, SMF, Adabas, and even to non z/OS data sources. Any data that is virtualized using DVM for z/OS is available to z/OS Connect.
Interaction between the DVM Server and z/OS Connect is enabled by the DVM Service Provider, a USS component that can integrate with zCEE to enable DVM Web services invocation. In turn, the DVM Service Provider establishes the communication channel with DVM Server using WebSphere Optimized Local Adapter (WOLA)Service Provider, which is natively supplied by z/OS Connect EE.
WOLA is a function that allows fast, efficient, and low-latency cross-memory exchanges between z/OS Connect and external address spaces, such as DVM for z/OS. WOLA requires the DVM Server and zCEE server to reside on the same LPAR. WOLA uses a three-part name to uniquely identify the WOLA server (or communication channel). This name is derived from the wolaGroup, wolaName2, and wolaName3 attribute values.
Configure the DVM Server for use with z/OS Connect
Allocate a WebSphere Optimizer Local Adapter (WOLA) PDSE data set with the following characteristics to configure the DVM Server to work with the zCEE server. A best practice is to use the DVM Server naming convention for the WOLA PDSE definition.
 
  • Space units: TRACK
  • Primary Quantity:  30
  • Secondary Quantity: 2
  • Directory blocks: 15
  • Record format: U
  • Record length: 0
  • Block size: 32760
  • Data set name type: LIBRARY
Invoke the z/OS Unix shell and get to the following working directory:
cd //wlp/clients/zos
cd /usr/lpp/zosconnect/v330/wlp/clients/zos << example
Issue the following command to copy modules from USS into WOLA PDSE. These modules enable WOLA communication between a DVM Server and a zCEE server. You could have multiple zCEE and DVM servers using a single WOLA PDSE data set.
cp -Xv ./* "//'<WOLA PDSE>'"
Add your WOLA PDSE data set to the AVZRPCLB ddname in the server started task JCL.
//AVZRPCLB DD DISP=SHR,DSN=&HLQ..SAVZRPC
// DD DISP=SHR,DSN=
APF authorizes the <WOLA PDSE> data set and finds "Enable z/OS Connect interface facility" in xxxIN00 configuration member where 'xxx' is the DVM Server subsystem name.
/*------------------------------------------*/
/* Enable z/OS Connect interface facility */
/*------------------------------------------*/
if DontDoThis then do
Change "DontDoThis" in "DoThis" to enable the WOLA parameters and confirm that the ZCONNECT parameter is enabled.
Optionally, add the ZCONNECTPWNAMEX parameter and set the value to the concatenation of WolaName2 and WolaName3 separated by a dot (<WolaName2>.<WolaName3>). Default values for WolaName2 and WolaName3 are respectively NAME2 and NAME3. WolaName2 and WolaName3 can be arbitrary 1 - 8 characters strings. WolaName2 and WolaName3 are also used during zCEE WOLA configuration so, if specified, it is recommended to take a note of them.
"MODIFY PARM NAME(ZCONNECTPWNAMEX) VALUE(NAME2.NAME3)"
Customize the DEFINE ZCPATH command, which is used to define a connection to a specific z/OS Connect region (server).
"DEFINE ZCPATH",
" NAME(ZCNALL)",
" RNAME(DVJR1)",
" WNAME(DVJG1)"
NAME is an arbitrary name and RNAME is up to 12 characters in length. The WNAME is up to 8 characters in length for the WolaGroup name. RNAME and WNAME are also used during zCEE WOLA configuration so it is recommended to take note of them.
By default, DVM for z/OS retries failed connections with the zCEE server. Sometimes failed connections are caused by an inactive zCEE server, but the communication error is always logged in the DVM Server traces file. If you are using RACF, a profile definition of CBIND resource class is required to allow the zCEE server and the DVM Server to connect and function properly.
  • A generic or discrete CBIND resource definition is required. A generic definition uses a CBIND class profile of BBG.WOLA.<WolaGroup>.** with UACC(READ), whereas a discrete definition uses a CBIND class profile of BBG.WOLA.<WolaGroup>.<WolaName2>.<WolaName3> with UACC(READ).
  • The WolaGroup is the WNAME specified in the DEFINED ZCPATH command, WolaName2 and WolaName3 are the values defined in the ZCONNECTPWNAMEX parameter. If ZCONNECTPWNAMEX has not been defined, the WolaName2 value must be NAME2 and the WolaName3 value must be NAME3.
Install the DVM Service Provider
Once the zCEE server is defined with security authentication enabled, it's time to install the DVM Service Provider shipped in hlq.SAVZBIN(AVZBIN4).
  • Transfer the member hlq.SAVZBIN(AVZBIN4) to your workstation in binary mode
  • Rename the file to com.rs.dv.zosconnect.provider.feature_1.0.0.esa
  • Copy the file to a UNIX System Services (USS) directory  Connect to UNIX System Services (USS)
  • Change directory to //wlp/bin, where is the path directory for your z/OS Connect EE installation. For example, change to the following directory:
    cd /usr/lpp/zosconnect/v330/wlp/bin
  • Set JAVA_HOME = environment variable to the path of your 64-bit IBM Java SDK:
    export JAVA_HOME=/usr/lpp/java/IBM/J8.0_64
  • Set the WLP_USER_DIR environment variable to the location where your server instances and user features are stored.
    export WLP_USER_DIR=/var/zosconnect
  • ./installUtility install /dvmServiceProvider/com.rs.dv.zosconnect.provider.feature_1.0.0.esa
  • The zCEE server.xml can be edited using IBM Explorer for z/OS
    • Under xml element, add the following lines:

       
      <feature>usr:dvsProvider</features>
      <feature>zosLocalAdapters-1.0</feature>
      Where usr:dvsProvider is DVM Service Provider on zCEE. zosLocalAdapters-1.0 is the WOLA provider used to communicate between the DVM Service Provider on zCEE and DVM Server.
  • Under the <server> xml element, add entries for the required DVM Service Provider on zCEE:

     
    <zosconnect_zosConnectService
      id="zosConnectDvsService"
      serviceName="DvsService"
      serviceRef="dvsService"
      serviceDescription="IBM DV Service provider"
      invokeURI="/dvs" />
    
    <usr_dvsService
      id="dvsService"
      connectionFactoryRef="wolaCF"
      registerName="DVJR1"
      serviceName="DVJS1"
      invokeURI="/dvs" />
    Where:
    • serviceRef in <zosconnect_zosConnectService> must be the same as id in
    • invokeURI in <zosconnect_zosConnectService> must be the same as invokeURI in <usr_dvsService>. It represents the "root" directory of DVM Web services: it means that all DVM Web services are reachable with an URL, which starts with http(s)://<zos_connect_ip>:<zos_connect_port>/dvs/…
    • connectionFactoryRef in <usr_dvsService> must have the same value as connectionFactoryRef in <zosconnect_localAdaptersConnectService> and id in <connectionFactory> both defined in the next step
    • registerName in <usr_dvsService> is the same as RNAME defined in DEFINE ZCPATH command in the DVM IN00 configuration member.
    • serviceName in <usr_dvsService> must have the same value as serviceName in <zosconnect_localAdaptersConnectService> defined in the next step.
  • Under <server> xml element, add the following entries required for the WOLA Service Providers

     
    <zosLocalAdapters
        	wolaGroup="DVJG1"
        	wolaName2="NAME2"
        	wolaName3="NAME3" />
        
    <connectionFactory id="wolaCF" jndiName="eis/ola">
        	<properties.ola />
     </connectionFactory>
        
     <zosconnect_zosConnectService
        	id="sdef1"
        	serviceName="dvs1"
        	serviceAsyncRequestTimeout="600s"
        	serviceRef="svc1" />
        	
     <zosconnect_localAdaptersConnectService
        	id="svc1"
            registerName="DVJR1"
        	serviceName="DVJS1"
        	      connectionFactoryRef="wolaCF"
        	      connectionWaitTimeout="7200" />
    
    Where:

    a. wolaGroup in <zosLocalAdapters> has the same value as WNAME defined in:
        DEFINE ZCPATH command in the DVM IN00 configuration member
    b. wolaName2 and wolaName3 have the same value as WNAME defined in the ZCONNECTPWNAMEX parameter in the DVM IN00 configuration       
        member.  If ZCONNECTPWNAMEX has not been defined in DVM IN00 configuration member, WolaName2 value must be NAME2 and WolaName3
        value must be NAME3.
    c. id in <connectionFactory> is the same as connectionFactoryRef in <zosconnect_localAdaptersConnectService> and <usr_dvsService>
    d. serviceRef in <zosconnect_zosConnectService> must have the same value as id in <zosconnect_localAdaptersConnectService>
    e. registerName in <zosconnect_localAdaptersConnectService> is the same as RNAME defined in DEFINE ZCPATH command in the DVM IN00
        configuration member
    f. serviceName in <zosconnect_localAdaptersConnectService> must have the same value as serviceName in <usr_dvsService> defined in the
       previous step.
Create zCEE RESTful APIs for access to the DVM Server
When a RESTful API requests mainframe data, z/OS Connect communicates the request to the DVM Server using the WebSphere Optimized Local Adapter (WOLA). The DVM Server executes the requested Web Service to 'get' data.
Once configuration steps are completed, recycle the DVM and z/OS Connect Started tasks to establish a connection between the two servers. A successful pairing is written to the DVM Server log.
19.10.58 STC05152 AVZ4502H ZCPRHUPR subtask is active
19.10.58 STC05152 AVZ4502H ZCPRHLWR subtask is active
Setting REST z/OS Connect Web Services preferences
REST z/OS Connect Web Services preferences are required when invoking and executing a Web Service request for z/OS Connect using the DVM studio. Set Web Services preferences to 'REST using z/OS Connect' and provide a Service Provider URL using the following nomenclature. Figure 5 shows the settings for a RESTful configuration.
https://<zos_connect_ip>:<zos_connect_port>/<invokeURI>
Note: <invokeURI> is as represented in the server.xml
Screen Shot 2021-05-23 at 8.11.30 PM
Figure 5 - Preferences setting for REST-based Web Services with z/OS Connect Service Provider
Use the Max Records Parameter to set a limit for the number of records retrieved when executed and use the Prompt user before executing the generated query to prompt users before query execution.
Assign DVM Servers where Web Services are executed
Use the 'Target Systems Wizard' in the DVM Studio to identify a DVM Server where the service provider invokes web service requests. More than one Target System can be defined to suit your specific configuration, as shown in Figure 6.
Screen Shot 2021-05-23 at 8.12.57 PM
Figure 6 - Target System creation
Create the Web Services metadata repository
Use the Web Services Directory Wizard to define Partitioned Data Sets (PDS) on the mainframe where Web Services metadata is stored. You provide a 'Name' for your web services directory, a 'High-Level Qualifier' to use as a data set prefix, as you create, verify or edit metadata associated with web services for your mainframe system. The library for the web services metadata is automatically created if it doesn't exist. Figure 7 shows the Web services directory definition.
  Screen Shot 2021-05-23 at 8.14.12 PM
Figure 7 - Web Services Directory definition
Use the Microflow Library Dataset dialog to create a Microflow metadata library on your mainframe, as shown in Figure 8.
Screen Shot 2021-05-23 at 8.15.10 PM
Figure 8 - Microflow Library Dataset generation
Create Web Services using the DVM Studio
Use the Web Services Wizard to create a Web Service for your particular DVM Server's web directory. The newly created web service metadata is written to your DVM Server web directory. This wizard steps through assigning a Name, Web Service Operation Type (REST using z/OS Connect, as well as Business and Screen level SOAP option).
You can select a virtual table or stored procedure to associate with the web service you are creating and then have the option to update the name, description, or SQL statement used to put or get data from a web-based client request. Figure 9 shows the SQL editor element for defining a web service.
Screen Shot 2021-05-23 at 8.15.52 PM
Figure 9 - Web Service definition: SQL editor
The Web Services workflow accommodates dynamic inputs for the SQL statement for your Web Service with the z/OS Connect REST interface by refreshing communication between the DVM Server and the zCEE Server, as shown in Figure 10 and Figure 11.
Screen Shot 2021-05-23 at 8.16.27 PM
Figure 10 - Web Services sample list output
Screen Shot 2021-05-23 at 8.16.56 PM
Figure 11 - Web Services sample list output
Deploy DVM web services to z/OS Connect RESTful API
To define and deploy a z/OS Connect RESTful API, DVM Studio, and an Eclipse IDE with z/OS Connect EE API Toolkit are used to build and expose SAR files. The IBM Explorer for z/OS can use of Eclipse-based plug-ins from the DVM Studio and API Toolkit from zCEE to product modern applications.
 You can promote an existing web service by generating SAR files. Taking an assumed Web Service for the DVM Server AVZ1 that accesses an underlying VSAM source using http. The hypothetical DvsqlStaffvs Web Service uses the zCEE SAR file Generator to convert from stand-alone use to general consumption using the zCEE REST Interface. Figure 12 shows SAR file generation.
Screen Shot 2021-05-23 at 8.17.28 PM
Figure 12 - SAR File generation
Using the z/OS Connect EE API Toolkit, create a new project from the Project Explorer dialog. The Editor API dialog defines the name of the service, its path, and version, as shown in Figure 13.
Screen Shot 2021-05-23 at 8.18.30 PM
Figure 13 - REST API path and input parameters
For each method, define the associated SAR file using the Service button and locating the SAR file previously generated in your File System. Figure 14 shows how to associate a file to a RESTful API.
Screen Shot 2021-05-23 at 8.18.53 PM
Figure 14 - SAR file association to a REST API
Use dynamic user input parameters
If there is any user input for the Web Service, use Request Mapping to define and link any input parameters corresponding with the Web Service parameters defined on the DVM Studio. Finally, deploy your RESTful API by clicking the button indicated in Figure 15.
Screen Shot 2021-05-23 at 8.19.25 PM
Figure 15 - RESTful API deployment
Db2 Query Management Facility (QMF) API
DVM for z/OS has a unique synergy with ZCEE in that RESTful web services can be created to access and submit requests against Virtual Tables that map to underlying unstructured data residing on the mainframe. Db2 QMF provides a RESTful API that can leverage DVM for z/OS by using the distributed DVM JDBC driver. Sample Output from a JavaScript application in Example 3, created from Db2 SMF that calls the QMF REST API to run a query against SMF records.
SELECT SMF30JBN AS JOB_NAME, SMF30STM AS STEP_NAME, SMF30PGM AS PROGRAM, SMF_SSI AS TYPE, SMF30CSU AS CPU
FROM ( SELECT SMF_STY, SMF30HPT, SMF30IIP, SMF30RCT, SMF30HPT, SMF30JQT, SMF30RQT,SMF30HQT, SMF30SQT, SMF30STI, SMF_TIME,
SMF30SIT, SMF30IO, SMF30PSN,SMF30MSO, SMF30SRB, SMF30CSU, SMF30SRV, SMF30ISB, SMF30ICU, SMF30PGM,SMF30CL8, SMF30SSN,
SMF30JBN, SMF30JNM, SMF30STM, SMF30STN, SMF_SSI,SMF_SID, SMF30CPT, SMF30CPS, SMF30CPT, SMF30CPS,SMF30_TIME_ON_IFA,
SMF30_TIME_ON_SUP, SMF30TCN, SMF30TEP, SMF30SCC, LS_TIMESTAMP_LOCAL FROM SMF_03000_SMF30IDA INNER JOIN
SMF_03000_SMF30CAS B ON A.BASE_KEY = B.BASE_KEY INNER JOIN SMF_03000 C ON A.BASE_KEY = C.BASE_KEY INNER JOIN
SMF_03000_SMF30CMP D ON A.BASE_KEY = D.BASE_KEY INNER JOIN SMF_03000_SMF30URA E ON A.BASE_KEY = E.BASE_KEY INNER JOIN SMF_03000_SMF30PRF F ON A.BASE_KEY = F.BASE_KEY ) WHERE SMF_STY = 4;
Example 3 - SQL sub-select used to access an SMF record
Figure 16 previews results from the defined web service generated from the Db2 QMF RESTful API are now available to help drive operational analytics and opportunities for optimization for the application and use of resources.
Screen Shot 2021-05-23 at 8.51.34 PM
Figure 16 - Results of SMF query in a spreadsheet.
Integrated Data Facility (IDF) for mainframe applications
DVM for z/OS introduced the Integrated Data Facility (IDF), to introduce support for a DRDA server interface so that Db2 for z/OS can read or JOIN any virtual data sources provisioned on the DVM Server that has a registration in SYSIBM.LOCATIONS and SYSIBM.IPNAMES.
The DVM server subsystem in the Db2 communications database
In this case, the DVM subsystem is AVZW. The DVM Server has a valid virtual table named SASAPPLICANT with a schema having a default three-part name of RS01AVZW.DVSQL.SASAPPLICANT. Figure 17 shows a select statement using a three-part name of schema.database.table.
SELECT * FROM RS01AVZW.DVSQL.SASAPPLICANT
Screen Shot 2021-05-23 at 8.53.42 PM
Figure 17 - Selecting DVM data with three-part names using Db2
The three-part name can be simplified by creating a View or an Alias. However, the above figure uses Db2 QMF for TSO to generate output, however, the use of IDF makes DVM Server virtual tables available using three-part names or by way of a View or Alias to other mainframe applications like Cobol, Db2 Stored Procedures, Db2 for z/OS Restful Services and SPUFI.
Use cases
Mainframe applications are the primary use case with limited testing against non-Z sources. Typical non-Z applications like MS-Excel, QMF for Workstation, Cognos, SPSS, and other client applications typically use the DVM JDBC or ODBC driver through the DVM Server directly.
Db2 as an entry point
By using Db2 as the entry point, you can take advantage of the existing skill sets and leverage the applications already in place as templates for new development.
  • Use Case 1: TSO or SPUFI
    QMF for TSO or SPUFI can query the DVM data with three-part names.
  • Use Case 2: Cobol applications
    Custom Cobol programs may be used to pull data using embedded SQL with three-part names (or View or Alias). However, when the packages for the SQL are bound, they must also be bound in the DVM Server instance associated with IDF. The same situation faced with three-part names accessing another remote Db2 for the z/OS subsystem. The job that compiles the program has a step to either perform a remote bind or to copy the packages to the instance of DVM associated with IDF.
  • Use Case 3: Db2 Stored Procedures
    Db2 Stored Procedures can be created that run queries aimed at DVM data sources using IDF. SQL Call statements to these Db2 Stored Procedures can then be incorporated into DB2 Restful Services.
When to choose Db2 UDTF or IDF
DVM’s UDTFs and IDF appear similar, so how do you decide which to use? Table 3 shows a comparison between the two access interfaces.
Db2 UDTF Integrated Data Facility (IDF)
Can be created by any valid user with access to the DVM Studio and Db2 for z/OS. No setup required. IDF must be configured by the administrators for both the DVM Server and Dvb2 for z/OS using the Db2 communications database. The DVM server needs to be registered, to populate metadata with appropriate user privileges.
SELECT SELECT, SELECT INTO, INSERT, UPDATE, DELETE
By default, passes only the SELECT portion of the query to the DVM Server. No SQL pushdown of WHERe predicates IDF passes the SQL through the DVM Server with ANSI-SQL 92 support and supports WHERE predicates and an SQL pushdown
Supported by all Db2 for z/OS clients on the mainframe and over the distributed environment. For example, QMF, Cognos, MS-Excel. IDF currently is limited to mainframe client applications
Table 3 - Db2 UDTF versus Integrated Data Facility (IDF)
Db2 for z/OS UDTF
Db2 for z/OS has a database function called a user-defined table function or UDTF. A UDTF depends on a custom program that is launched by Db2 when a UDTF is called as part of a SELECT statement. This custom program can access Db2 data or it can directly access the underlying file system backing the Db2 database and return the results in the normalized relational format as if seamlessly part of the local Db2 database.
You can use the Db2 for z/OS database as an Information or Federation hub for your enterprise. This capability is read-only and does not support writeback to remote sources, however, the Db2 database as a central store connected to DVM for z/OS as a connected Information architecture to disparate data residing on the mainframe or off the mainframe as relational, Big Data, Kafka or Flat Files.
DVM for z/OS can create Db2 UDTFs that point to DVM virtual tables using the DVM Studio, which is then referenced as a UDTF or View in the local Db2 database, shown in Figure 18.
Screen Shot 2021-05-23 at 9.03.04 PM
Figure 18 - Creating a UDTF
There are currently five optimized UDTF modules available for different hardware environments.
  • AVZUDT9N for the z196 system
  • AVZUDTAN for the zEC12 system
  • AVZUDTBN for the z13 system
  • AVZUDTCN for the z14 system
  • AVZUDTDN for the z15 system
Then, Db2 for z/OS Subsystem uses the Db2 Workload Manager environment for both User Defined Functions (UDF) and User Defined Routines (UDR). The Db2 Wizard can be used to generate new UDTF definitions or Views for the Db2 for the z/OS database referencing Z provisioned data on the DVM Server, as shown in Figure 19.
Figure 19
Figure 19 - Creating a UDTF using the Db2 Wizard
The SQL results profile the Db2 UDTF execution trace. The UDTF is now present in SYSIBM.SYSROUTINES on the designated Db2 Subsystem, as shown in Figure 20.
Figure 20
Figure 20 - SQL Window report on the process of UDTF creation
Without a View, a DVM UDTF is addressed in SQL with a less familiar SQL syntax. UDTFs don't appear in SYSIBM.SYSTABLES and not exposed to 3rd party commercial software. Creating a View addresses SQL standards and the ability to discover, however, Views don't perform SQL Pushdown at the source and return all rows for processing back to the Db2 for z/OS Subsystem.
The backend DVM Server is not technically known to the Db2 database and does not know the DVM SQL Engine backing a View on a UDTF. Therefore, the WHERE clause is NOT passed through to the remote data source. Db2 receives ALL of the data and then apply the WHERE clause. This is not a challenge for Db2 in many cases, but what if the Virtualized table contains billions of records?
Here is a query that SELECTs from the VIEW on the UDTF:
SELECT TEMPID, NAME, ADDRESS, EDLEVEL,COMMENTS
FROM TWSHAWN.AVZW_VSASAPPLICANT
WHERE TEMPID >450
The following SQL statement is more efficient. The first parameter in the syntax for the DVM UDTF is a pair of single quotation marks with nothing in them ''. This spot is reserved for a WHERE clause predicate.
SELECT TEMPID,NAME,ADDRESS,EDLEVEL,COMMENTS
FROM TABLE (TWSHAWN.AVZW_SASAPPLICANT ('WHERE TEMPID >450', 'AVZW...SASAPPLICANT', '5,TEMPID,NAME,ADDRESS,EDLEVEL,COMMENTS', ''))
Some query tools, like IBM QMF, allow you to parameterize the SQL when using Host variables. The following example prompts for the WHERE clause during runtime, shown in Figure 21.
Screen Shot 2021-05-23 at 9.09.14 PM
Figure 21 - Prompting for a WHERE clause at runtime
If the WHERE clause is operating on TEXT or DATE\TIME values that require single quotation marks, the single quotation marks need to be doubled to two single quotes, not a double quote, as shown in Figure 22.
Screen Shot 2021-05-23 at 9.10.46 PM
Figure 22 - Character data in the WHERE clause
The final parameter is also a set of empty single quotation marks. This is to contain DVM runtime options like Map Reduce. This is discussed in the User Guide.
Db2 federation
DVM for z/OS has a level of integration with IBM's distributed Db2 family portfolio by distributing the DVM JDBC and ODBC drivers across all on-premises and cloud offerings; Db2 AESE, Db2 Warehouse, Db2 Warehouse on Cloud, IBM Integrated Analytics Systems, Netezza, and Cloud Pak for Data running on-premises and Cloud Pak for Data as a Service.
There is no need to download or install drivers and there is no need to enable Db2 family database engines for data federation, as this is now a built-in out-of-the-box capability that doesn't require manual commands for creating wrappers, or data type mappings. The Db2 database has optimized the DVM Server as a remote database and information architecture and performs SQL Pushdown and the ability to create local Nicknames or Remote Tables that map to virtual tables provisioned on the DVM Server for underlying Z sources like VSAM, IMS, Adabas, Sequential Files, Logstream, Syslog, SMF, and Tape systems.
  1. Connect to the database that requires access to DVM Server from either the command line or using the Data Studio Manager (DSM), Data Management Console (DSM), Unified Consoles as part of Db2 Warehouse on-prem and Cloud offerings, and the IIAS system.
  2. Create a connection server pointing to the DVM Server along with all other options specified in the following command, and as well as in Figure 23.
db2 "CREATE SERVER <server_name> TYPE JDBC WRAPPER JDBC OPTIONS
(DRIVER_PACKAGE '<path to DVM jdbc driver>',
DRIVER_CLASS 'com.rs.jdbc.dv.DvDriver',
URL 'jdbc:rs:dv://<dvm_ip_address>:<dvm_port>;DBTY=DVS;SUBSYS=NONE;
PMDSQL=true',
pushdown 'Y',
DB2_MAXIMAL_PUSHDOWN 'Y',
db2_varchar_blankpadded_comparison 'Y',
db2_char_blankpadded_comparison 'Y',
collating_sequence 'Y' )"
Screen Shot 2021-05-23 at 9.11.40 PM
Figure 22 - Server creation command
Description of all options specified in server creation statement follows:
  • pushdown ‘Y’: the federated server considers letting DVM evaluate operations. If you set PUSHDOWN to N, the federated server retrieves select columns from the remote data source and doesn't let the DVM server evaluate other operations, such as joins
  • DB2_MAXIMAL_PUSHDOWN ‘Y’: the federated server pushes as many parts of the query as possible to DVM for processing
  • db2_varchar_blankpadded_comparison 'Y’ and db2_char_blankpadded_comparison ‘Y’: the federated server pushes filtering based on character columns to DVM Server
  • collating_sequence 'Y’: character/numeric data predicates comparison, character range predicates comparison, and sort operations might be pushed down if collating sequences are the same
Create a user mapping for all users that require access to the federated DVM Server, as shown in the statement and Figure 24.
db2 "CREATE USER MAPPING FOR <user> SERVER <server_name> OPTIONS (REMOTE_AUTHID 'XXX', REMOTE_PASSWORD 'xxx')"
Screen Shot 2021-05-23 at 9.13.04 PM
Figure 24 - User mapping command
To validate the access to DVM Server, create a nickname for a remote DVM Virtual Table or Virtual View
db2 "CREATE NICKNAME <nickname> FOR <server_name>.<virtual table/view>"
IBM Cloud Pak for Data
IBM Cloud Pak® for Data is a fully integrated data and AI platform that helps modernize the way data can be collected, organized, analyzed, and infused with AI. The platform is cloud native and is designed to add powerful new capabilities for data management, DataOps, governance, business analytics, and AI. IBM Cloud Pak for Data delivers the modern information architecture to turn AI aspirations into tangible business outcomes, while improving governance and protecting your data.
IBM Cloud Pak for Data can also access z/OS data using Data Virtualization Manager for z/OS. Using a built-in JDBC driver (more details about the DVM-JDBC driver are in Chapter 5.x), any z/OS data that is virtualized with DVM is available to a wide variety of Cloud Pak for Data applications and can be cataloged, analyzed, and infused with AI on the platform.
In this chapter, we shows how to access DVM virtualized data sources using the Cloud Pak for Data DVM connector component.
Cloud Pak for Data interface and adding a DVM Connection
Log in to an available IBM Cloud Pak for Data instance shown in Figure 25.
Screen Shot 2021-05-23 at 9.14.16 PM
Figure 25 - IBM Cloud Pak for Data login
Access Platform connection from the main drop-down menu on the Cloud Pak for Data environment to create a new connection and choose Data Virtualization Manager for z/OS connection service, shown in Figure 26 and Figure 27.
Screen Shot 2021-05-23 at 9.14.52 PM
Figure 26 - Available connection service for DVM for z/OS
Screen Shot 2021-05-23 at 9.15.40 PM
Figure 27 - Connection service name, hostname, Port, and access credentials
Previewing data from your newly connected DVM server
Once a connection is established, you can easily discover provisioned data on the DVM Server. This connection provides access to persisted data on the mainframe for single queries or JOINs across sources like VSAM, IMS, Adabas, Sequential Files, Syslogs, Logstream, SMF, Tape, and so on. Using the SQL editor on the Cloud Pak for Data user console, you can preview, refine, and create virtual assets that map to the DVM for z/OS information architecture.
Therefore, you can create public or private projects typically used by Data Engineers or Data Scientists and work to requesting the publishing of virtual assets to the Watson Knowledge Catalog by the Data Curator responsible for Data Governance. Once remote Z data assets are published to the Watson Knowledge Catalog, they are accessible for Machine Learning, Analytics, and AI-related activities.
Create a new Project or Open an existing project where you want to work with the data available from the DVM Connection. The data assets, which include individual tables, metadata, or connections is available from within a CPD project. Figure 28 shows the Projects for an active user.
Figure 28
Figure 28 - List and create new projects in IBM Cloud Pak for Data

[{"Type":"SW","Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS4NKG","label":"IBM Data Virtualization Manager for z\/OS"},"ARM Category":[{"code":"a8m0z000000cxAYAAY","label":"Java Gateway"},{"code":"a8m0z000000cxAOAAY","label":"SQL Engine"},{"code":"a8m0z000000cxATAAY","label":"Studio"},{"code":"a8m0z000000cxAZAAY","label":"z\/OS Connect"}],"Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"All Version(s)"}]

Product Synonym

DVM

Document Information

Modified date:
23 June 2021

UID

ibm16453441