DFDL customizations

When you change the targetNamespace, you must update any XML namespace prefix declaration that refers to the namespace. The following sample highlights the targetNamespace and XML namespace prefix declaration.

<xs:schema targetNamespace="http://www.ibm.com/xmlns/prod/ztpf/dfdl/gen/sample"
      xmlns:xs="http://www.w3.org/2001/XMLSchema"
      xmlns:ns0="http://www.ibm.com/xmlns/prod/ztpf/dfdl/gen/sample"
      xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
      xmlns:tddt="http://www.ibm.com/xmlns/prod/ztpf/xml/lib/tpfattributes"
      xmlns:tpfbase="http://www.ibm.com/xmlns/prod/ztpf/dfdl/lib/tpfbase">

The names that are used in DFDL do not need to match the C/C++ language structure or basic assembler language (BAL) DSECT names. The names are used only for external representation in JSON, XML, BSON, CSV, and so on. Therefore, the DFDL element names should reflect how the data is known to your enterprise system. The following sample highlights the element name attribute in the DFDL element definition.

<xs:element name="Address" type="xs:string" dfdl:lengthKind="explicit"
        dfdl:length="30" dfdl:lengthUnits="bytes" nillable="true"
        dfdl:useNilForDefault="yes" dfdl:nilKind="literalCharacter"
        dfdl:useNilValue="%NUL;" />

When DFDL is generated from C/C++ language structures or BAL DSECT names, some data types might not accurately reflect how the field is used. The following sample highlights the element type attribute in the DFDL element definition.

<xs:element name="Address" type="xs:string" dfdl:lengthKind="explicit"
        dfdl:length="30" dfdl:lengthUnits="bytes" nillable="true"
        dfdl:useNilForDefault="yes" dfdl:nilKind="literalCharacter"
        dfdl:nilValue="%NUL;" />

For more information about how data types are converted, see Table 1 in z/TPF DFDL generation utility (tpfdfdlgen).

The DFDL specification permits several modifications that are specific to string data types. One modification is the ability to trim and pad string values by using a specific character. When a string value is smaller than the data area that it is stored in, the string is padded with the specified pad character. Conversely, to add the string to a document, you can trim the trailing pad characters from the string that is output. On the z/TPF system, the default character for trimming and padding is 0x00 as defined in tpfbase.lib.dfdl.xsd as textStringPadCharacter="%NUL;". You can change this on a per-schema or per-element basis by defining the DFDL textStringPadCharacter attribute. The following sample highlights the element definition for a string that uses spaces for pad and trim characters.

<xs:element name="Address" type="xs:string" dfdl:lengthKind="explicit"
        dfdl:length="30" dfdl:lengthUnits="bytes" nillable="true"
        dfdl:useNilForDefault="yes" dfdl:nilKind="literalCharacter"
        dfdl:nilValue="%NUL;" dfdl:textStringPadCharacter="%SP;" />

Another modification is that DFDL can handle several different string encodings. This refers to the encoding of the string as stored in the z/TPF system, not the encoding of the transmitted string. On the z/TPF system, the default encoding is EBCDIC 1047, which is defined in tpfbase.lib.dfdl.xsd as encoding="x-ibm-1047-s390". You can change this by defining the DFDL encoding attribute. The following sample highlights the element definition for a string that is encoded in UTF-8.

<xs:element name="Address" type="xs:string" dfdl:lengthKind="explicit"
        dfdl:length="30" dfdl:lengthUnits="bytes" nillable="true"
        dfdl:useNilForDefault="yes" dfdl:nilKind="literalCharacter"
        dfdl:nilValue="%NUL;" dfdl:encoding="utf-8" />

Numbers can be represented in different ways; for example, you can store some numbers in packed decimal or binary coded decimal formats. These formats are defined in DFDL by using the binaryNumberRep attribute. The following sample highlights the element definition for a packed decimal number.

<xs:element name="accountNumber" type="xs:unsignedLong" 
        dfdl:lengthKind="explicit" dfdl:length="12" dfdl:lengthUnits="bytes"
        dfdl:binaryNumberRep="packed" default="0" /> 

A number can also be represented as a decimal with an implicit decimal position. The following sample highlights the element definition for a decimal number that is stored with 1/100^th precision.

<xs:element name="balance" type="xs:decimal" dfdl:lengthKind="explicit"
        dfdl:length="8" dfdl:lengthUnits="bytes"
        dfdl:binaryDecimalVirtualPoint="2" default="0" />

A float or double number can be represented in a document by using scientific or decimal notation. These document formats are defined in DFDL by using the tddt:floatingPointFormat attribute. The following sample highlights the element definition for a float number by using decimal notation that is specified by the printf function.

<xs:element name="price" type="xs:float" dfdl:lengthKind="explicit"
               dfdl:length="4" dfdl:lengthUnits="bytes" tddt:floatingPointFormat="%f" 
               default="0"/>

An integer type number can be represented in a document by using various formats. These formats are defined in DFDL by using the tddt:integerFormat attribute. All signed integer types are converted to a long type before being formatted by the sprintf function. All unsigned integer types are converted to an unsigned long type. Therefore, the length specifier for the tddt:integerFormat attribute is always an l. The following sample highlights the element definition for an int type by using one of the format specifiers that are available to the printf function.

<xs:element name="price" type="xs:int" dfdl:lengthKind="explicit"
               dfdl:length="4" dfdl:lengthUnits="bytes" tddt:integerFormat="%ld" 
               default="0"/>

A 1-byte field can contain a number of bit field indicators that are defined in application code, rather than in the field definition, by referencing the bits through logical AND operations with a bit mask. DFDL can define fields at a byte level and a bit level. These fields can be set by using a Boolean type to convey whether the status of the bit indicator is set to true or false. Bits that are not used can be skipped by using either the DFDL leadingSkip or trailingSkip attribute. The following sample shows how a single byte with bits 0x40 and 0x20 can be defined as several Boolean elements composing 8 bits.

<xs:element name="mobile" type="xs:boolean" dfdl:lengthKind="explicit"
      dfdl:length="1" dfdl:lengthUnits="bits" dfdl:leadingSkip="1"
      dfdl:alighnmentUnits="bits" default="0" />
<xs:element name="home" type="xs:boolean" dfdl:lengthKind="explicit"
      dfdl:length="1" dfdl:lengthUnits="bits" dfdl:trailingSkip="5"
      dfdl:alignmentUnits="bits" default="0" />

The DFDL that is generated by tooling typically defines all elements as not required. You might want some elements to be required for input. A DFDL element is viewed as not required if there is a default value defined. If no default value is defined, the element is required. Additionally, you can define the default value to be that of the nillable value. This is controlled through the DFDL useNilForDefault attribute. The following sample highlights the element definition for a number and string that are not required. Removing the parts that are highlighted in bold would make them required.

<xs:element name="accountNumber" type="xs:unsignedLong"
      dfdl:lengthKind="explicit" dfdl:length="12" dfdl:lengthUnits="bytes"
      dfdl:binaryNumberRep="packed" default="0" />
<xs:element name="Address" type="xs:string" dfdl:lengthKind="explicit"
      dfdl:length="30" dfdl:lengthUnits="bytes" nillable="true"
      dfdl:useNilForDefault="yes" dfdl:nilKind="literalCharacter"
      dfdl:nilValue=%NUL;" />

The definition of unions in C/C++ language structures or ORGs in BAL DSECTs indicates that there are multiple layout definitions for a data area. When the layouts are not equivalent, you must provide DFDL with information about how to choose which layout is correct for the given data. This is true only when you are converting data to XML, JSON, BSON, and so on. When DFDL does conversions to serialize the data, the labels in the XML, JSON, and so on are used to indicate which layout is correct. One way the layout is determined during parsing is through the use of the DFDL discriminator annotation. A discriminator is a test to determine the existence of a statement. A previous field is typically referenced in a discriminator by using XPath notation to determine either the layout of the subsequent data or the existence of the layout. The following sample shows how discriminators are used to determine the layout of a z/TPFDF logical record.

<xs:element dfdl:lengthKind="implicit" name="lrec90" type="ns0:LREC90">
       <xs:annotation>
              <xs:appinfo source="http.ogf.org/dfdl/">
                     <dfdl:discriminator>{../lrecKey eq xs:hexBinary('90')}
                     </dfdl:discriminator>
              </xs:appinfo>
       </xs:annotation>
</xs:element>

You might want to keep some elements hidden, either because they have an internal use but no external value, or because they contain personally identifiable information (PII). Because DFDL is aware of both internal and external representation, an element can have visible internal-only representation through the use of hidden groups. Such elements do not appear in the JSON, XML, BSON, and so on. Using hidden groups is one way to have different views of the data, depending on a DFDL variable. A hidden group uses a standard group definition but uses a DFDL hiddenGroupRef attribute to indicate that it is hidden. The hiddenGroupRef attribute can be used on only an empty sequence statement. The following sample shows the definition of hidden element age, which depends on the value of DFDL variable allowPII.

<xs:annotation>
	<xs:appinfo source="http://www.ogf.org/dfdl/">
	      <dfdl:defineVariable name="allowPII" type="xs:boolean"
                   external="true"/>
       </xs:appinfo>
</xs:annotation>

<xs:group name="PII_Info">
      <xs:sequence>
            <xs:element name="age" type="xs:unsignedByte"
                  dfdl:lengthKind="explicit" dfdl:length="1"
                  dfdl:lengthUnits="bytes" default="0" />
      </xs:sequence>
</xs:group>

<xs:choice>
      <xs:group ref="PII_Info">
            <xs:annotation>
                  <xs:appinfo source="http://www.ogf.org/dfdl/">
                        <dfdl:discriminator>{$allowPII}</dfdl:discriminator>
                  </xs:appinfo>
            </xs:annotation>
      </xs:group>
      <xs:sequence dfdl:hiddenGroupRef="PII_Info" />
</xs:choice>

Any name-value pair can be accessed in a DFDL schema by using a DFDL external variable. The name-value pair variable must be defined in tpfNameValue.lib.dfdl.xsd. System generated name-value pairs such as ISrvcName are already defined in that file. The following example illustrates how to use REST provider support to set the operationId name-value pair variable, which can be stored in a 16-byte string element in the REST provider request structure.

<xs:schema
      targetNamespace="http://www.ibm.com/xmlns/prod/ztpf/dfdl/gen/sample"
      xmlns:xs="http://www.w3.org/2001/XMLSchema"
      xmlns:nv="http://www.ibm.com/xmlns/prod/ztpf/name-value"
      xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
      xmlns:tpfbase="http://www.ibm.com/xmlns/prod/ztpf/dfdl/lib/tpfbase">

<xs:import namespace="http://www.ibm.com/xmlns/prod/ztpf/name-value" 
           schemaLocation="tpfNameValue.lib.dfdl.xsd"/>

<xs:group name="srvcName">
      <xs:sequence>
            <xs:element name="operationId" type="xs:string" dfdl:length="16"
                  dfdl:outputValueCalc="{$nv:ISrvcName}"/>
      </xs:sequence>
</xs:group>

<xs:sequence dfdl:hiddenGroupRef="srvcName"/>

You might need some strings to be variable in length. You can use a DFDL expression to indicate that a string has a variable length and define the length of the string. To calculate the length of the field, the DFDL expression uses XPath notation to reference a previous field to define the length. Because string lengths are not usually passed in documents, DFDL supports calculated fields to store calculated results, such as string lengths, in an element. These elements also can be hidden so that the element that contains the length is not present in documents that are created by DFDL. The following sample shows the definition of a variable length string with a hidden string length element.

<xs:group name="hiddenLen">
      <xs:sequence>
            <xs:element name="addrLen" type="xs:unsignedShort"
                  dfdl:lengthKind="explicit" dfdl:length="2"
                  dfdl:lengthUnits="bytes"
                  dfdl:outputValueCalc="{dfdl:valueLength(../Address,'bytes')}"
            />
      </xs:sequence>
</xs:group>

<xs:sequence dfdl:hiddenGroupRef="hiddenLen" />
<xs:element name="Address" type="xs:string" dfdl:lengthKind="explicit"
       dfdl:length="{../addrLen}" dfdl:lengthUnits="bytes" />

You might need some arrays to be variable in size; that is, in the number of items in the array. You can use a DFDL expression to indicate that an array has a variable number of items and define the number of items in the array. To calculate the size of the array, the DFDL expression uses XPath notation to reference a previous field to define the size. Because array sizes are not usually passed in documents, DFDL supports calculated fields to store calculated results, such as array sizes, in an element. These elements also can be hidden so that the element that contains the array size is not present in documents that are created by DFDL. The following sample shows the definition of a variable size array with a hidden array size element.

<xs:schema
      targetNamespace="http://www.ibm.com/xmlns/prod/ztpf/dfdl/gen/sample"
      xmlns:xs="http://www.w3.org/2001/XMLSchema"
      xmlns:fn="http://www.w3.org/2005/xpath-functions"
      xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
      xmlns:tpfbase="http://www.ibm.com/xmlns/prod/ztpf/dfdl/lib/tpfbase">

<xs:group name="hiddenSize">
      <xs:sequence>
            <xs:element name="arrayCount" type="xs:unsignedShort"
                  dfdl:lengthKind="explicit" dfdl:length="2"
                  dfdl:lengthUnits="bytes"
                  dfdl:outputValueCalc="{fn:count(../Numbers)}" />
      </xs:sequence>
</xs:group>

<xs:sequence dfdl:hiddenGroupRef="hiddenSize" />
<xs:element name="Numbers" type="xs:int" dfdl:lengthKind="explicit"
       dfdl:length="4" dfdl:lengthUnits="bytes" minOccurs="0"
       maxOccurs="unbounded" dfdl:occursCountKind="expression"
       dfdl:occursCount="{../arrayCount}" />

DFDL is used to describe a contiguous area of data. However, you can use attributes that are specific to the z/TPF system in DFDL to describe non-contiguous data, such as is often used in C/C++ language structures or BAL DSECTs. The z/TPF specific attributes support the definition of pointers on element definitions, which can be useful for variable length strings. The DFDL length attributes do not change but are instead commuted to the pointed-to element. The pointers can be 4 bytes or 8 bytes in length. The following sample highlights how you can change a string element to an 8-byte pointer to the string.

<xs:schema
      targetNamespace="http://www.ibm.com/xmlns/prod/ztpf/dfdl/gen/sample"
      xmlns:xs="http://www.w3.org/2001/XMLSchema"
      xmlns:tddt="http://www.ibm.com/xmlns/prod/ztpf/xml/lib/tpfattributes"
      xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
      xmlns:tpfbase="http://www.ibm.com/xmlns/prod/ztpf/dfdl/lib/tpfbase">

<xs:element name="Address" type="xs:string" dfdl:lengthKind="explicit"
       dfdl:length="{../addrLen}" dfdl:lengthUnits="bytes"
       tddt:indirectKind="pointer" tddt:indirectLength="8" />