About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Question & Answer
Question
DFDL for optional variable length trailing fields
Answer
On Demand Consulting
Author: Phil Bareham
Introduction
This article describes how to create an IIB DFDL model using a COBOL copybook that has optional trailing fields and a variable length final field. The DFDL model also needs to be able to handle record separators that are either Line Feed or Carriage Return & Line Feed.
Continue though the wizard selecting 'HDRDET01' to be the imported structure and 'Recognize null values for all fields' everything else can be left to default. This will create a DFDL model that looks like this:
We implement this by using a 'Length Kind' of 'pattern' on the HEADER and DETAILS fields. The pattern is a regular expression (regex) in this case this will be '.*':
The default behaviour of the regex: '.*' is to match any character except line terminators where a line terminator is defined as:
COBOL copybook HDRDET01.cpy used to build the initial DFDL model
The COBOL copybook used to describe the records to be parsed by the DFDL model was a follows:
000010 01 HDRDET01.
000020 02 REPEATING_GROUP.
000050 03 HEADER.
000060 05 HEADER_RECORD_ID PIC X(3).
000070 05 HEADER_FIELD01 PIC X(8).
000080 05 HEADER_FIELD02 PIC X(6).
000090 05 HEADER_FIELD03 PIC X(25).
000100 05 HEADER_FIELD04 PIC X(14).
000110 05 HEADER_FIELD05 PIC X(4).
000120 05 HEADER_FIELD06 PIC X(20).
000130 05 HEADER_FIELD07 PIC X(18).
000140 05 HEADER_FIELD08 PIC X(4).
000150 05 HEADER_FIELD09 PIC X(4).
000160 05 HEADER_FIELD10 PIC X(4).
000170 05 HEADER_FIELD11 PIC X(4).
000180 05 HEADER_FIELD12 PIC X(4).
000190 05 HEADER_FIELD13 PIC X(8).
000200 05 HEADER_FIELD14 PIC X(8).
000210 05 HEADER_FIELD15 PIC X(16).
000220 05 HEADER_FIELD16 PIC X(40).
000230 03 DETAILS.
000240 05 DETAIL_FIELD01 PIC X(3).
000250 05 DETAIL_FIELD02 PIC X(8).
000260 05 DETAIL_FIELD03 PIC X(10).
000270 05 DETAIL_FIELD04 PIC X(13).
000280 05 DETAIL_FIELD05 PIC X(1).
000290 05 DETAIL_FIELD06 PIC X(3).
000300 05 DETAIL_FIELD07 PIC X(8).
000010 01 HDRDET01.
000020 02 REPEATING_GROUP.
000050 03 HEADER.
000060 05 HEADER_RECORD_ID PIC X(3).
000070 05 HEADER_FIELD01 PIC X(8).
000080 05 HEADER_FIELD02 PIC X(6).
000090 05 HEADER_FIELD03 PIC X(25).
000100 05 HEADER_FIELD04 PIC X(14).
000110 05 HEADER_FIELD05 PIC X(4).
000120 05 HEADER_FIELD06 PIC X(20).
000130 05 HEADER_FIELD07 PIC X(18).
000140 05 HEADER_FIELD08 PIC X(4).
000150 05 HEADER_FIELD09 PIC X(4).
000160 05 HEADER_FIELD10 PIC X(4).
000170 05 HEADER_FIELD11 PIC X(4).
000180 05 HEADER_FIELD12 PIC X(4).
000190 05 HEADER_FIELD13 PIC X(8).
000200 05 HEADER_FIELD14 PIC X(8).
000210 05 HEADER_FIELD15 PIC X(16).
000220 05 HEADER_FIELD16 PIC X(40).
000230 03 DETAILS.
000240 05 DETAIL_FIELD01 PIC X(3).
000250 05 DETAIL_FIELD02 PIC X(8).
000260 05 DETAIL_FIELD03 PIC X(10).
000270 05 DETAIL_FIELD04 PIC X(13).
000280 05 DETAIL_FIELD05 PIC X(1).
000290 05 DETAIL_FIELD06 PIC X(3).
000300 05 DETAIL_FIELD07 PIC X(8).
In the HEADER record the last four fields (FIELD13 ? FIELD16) are optional the last of the optional fields in any message can be variable length. In the DETAILS record the last field - DETAIL_FIELD07 is optional and variable length. COBOL does not support this the COBOL copybook is just used a start point to create the DFDL model.
Create the initial DFDL model from the HDRDET01 copy book
Create the initial DFDL model from the copybook by using New -> Message Model in the IIB Toolkit then select the COBOL option and select the HDRDET01 copybook from the workspace.Continue though the wizard selecting 'HDRDET01' to be the imported structure and 'Recognize null values for all fields' everything else can be left to default. This will create a DFDL model that looks like this:
Model the repeating group
The data to be parsed by this model can have a repeating group of HEADER and DETAILS records to implement this change the 'Max Occurs' for REPEATING_GROUP from '1' to 'unbounded' and 'Occurs Count Kind' from 'fixed' to 'implicit':Set the trailing fields in the HEADER and DETAILS record to be optional
The HEADER_FIELD13 to HEADER_FIELD16 and DETAIL_FIELD07 fields are optional to model this in DFDL change the 'Min Occurs' to zero and the 'Occurs Count Kind' to implicit. Note that for the HEADER record if HEADER_FIELD16 exists then HEADER_FIELD13 14 and 15 must also exist. HEADER_FIELD16 cannot exist if any of the preceding fields are not present. Change the 'Occurrences' settings for each of the optional fields as follows:
Model the variable record length in the HEADER and DETAILS records
We need tell the parser to stop processing data when it hits the end of a record even if the end of a record (LF) is before the end of the optional fields. If HEADER_FIELD13 and HEADER_FIELD14 are present but HEADER_FIELD15 is 10 bytes long (shorter than its 16 byte explicit length we need the parser so stop when it has processed the 10 byte of HEADER_FIELD15.We implement this by using a 'Length Kind' of 'pattern' on the HEADER and DETAILS fields. The pattern is a regular expression (regex) in this case this will be '.*':
The default behaviour of the regex: '.*' is to match any character except line terminators where a line terminator is defined as:
- A newline (line feed) character ('\n')
- A carriage-return followed immediately by a newline character ('\r\n')
- A carriage return character on its own ('\r')
- A next line character ('\u0085')
- A line-separator character ('\u2028')
- A paragraph-separator character ('\2029')
Handling different record terminators (Line Feed or Carriage Return & Line Feed)
In our scenario our test data the records (HEADER or DETAILS) were terminated by either Line Feed (\nl) or Carriage Return and Line Feed (\cr \nl). We needed to cater for both these possibilities in our DFDL model.
This is achieved by entering both possible terminators in the HEADER and DETAILS terminator field separated by a space:
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSQTW3","label":"IBM On Demand Consulting for Hybrid Cloud"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"","label":""}}]
Was this topic helpful?
Document Information
More support for:
IBM On Demand Consulting for Hybrid Cloud
Software version:
All Versions
Document number:
776267
Modified date:
16 March 2019
UID
ibm10776267
Manage My Notification Subscriptions