Transforming between COBOL copybooks and JSON events with the data transformation utility

The data transformation utility is included in Open Enterprise SDK for Apache Kafka to help you transform data between native COBOL copybooks and the event format in JSON. The utility is available as a command-line tool.

The utility generates COBOL copybooks and code snippets for a specific Kafka event schema. The generated copybooks can be used to process the Kafka event JSON object in COBOL applications. The utility-generated code snippets are used to transform COBOL event data byte arrays into a JSON object for a Kafka producer, and a JSON object into COBOL event data byte arrays for a Kafka consumer.

The key steps for using the utility are outlined as follows:

Defining attributes in the language properties file
Running the utility

Defining attributes in the language properties file

Before you run the data transformation utility, define the following attributes in the properties file.

The following attributes are mandatory in the properties file:

Table 1. Mandatory attributes
Attributes	Description
`arrayMaxItems`	Specifies the default value for the "maxItems" property of any array field that does not have an explicitly set "maxItems" property in the JSON schema.
`artifactsLocation`	Specifies the directory where the generated copybooks or code snippets are stored. This directory must exist.
`integerMax`	Specifies the default maximum value for an integer field in the JSON schema. This is the value used for the "maximum" property for any integer field that does not have an explicitly set "maximum" property in the JSON schema.
`integerMin`	Specifies the default minimum value for an integer field in the JSON schema. This is the value used for the "minimum" property for any integer field that does not have an explicitly set "minimum" property in the JSON schema.
`jsonSchemaFile`	Specifies the location and name of the JSON schema file. The file has an extension of `.json`.
`language`	Specifies the language for the artifact generation. Currently the only supported value is "COBOL".
`logFileDirectory`	Specifies the directory where the log generated from the data transformation utility is stored. This directory must exist.
`numberFractionalDigits`	Specifies the number of fractional digits to be used in COBOL for a field of type "number" in the JSON schema.
`numberNonFractionalDigits`	Specifies the number of non-fractional digits to be used in COBOL for a field of type "number" in the JSON schema.
`stringMax`	Specifies the default maximum length for a string field in the JSON schema. This is the value used for the "maxLength" property for any string field that does not have an explicitly set "maxLength" property in the JSON schema.

The following attributes are optional in the properties file:

Table 2. Optional attributes
Attributes	Description
`cobolJsonStoreVariableName`	Specifies the name of the COBOL variable to which the JSON schema file will be moved in the case of a Kafka producer. This attribute is optional and has a value of "CON-EVENT-DATA" if not specified.
`cobolRootLevelVariableName`	Specifies the name of the COBOL level-01 variable used when the copybook is being generated. This attribute is optional and has a value of "EVENT-DATA" if not specified.

Running the utility

Ensure that the KAFKA_HOME environment variable is set to <product_path>/ixy/v1r1m0/zkafka
```
export KAFKA_HOME=<product_path>/ixy/v1r1m0/zkafka
```
Ensure the PATH environment variable is set to <product_path>/ixy/v1r1m0/zkafka/util
```
export PATH=$PATH:$KAFKA_HOME/util
```

Copy the sample language properites file

cp <product_path>/ixy/v1r1m0/zkafka/util/lang.properties <filename>.properties

Check out the utility help information and understand the usage:
```
data-transformation-utility.sh --help
```
Make sure that you have updated the sample <filename>.properties file in this directory with the relevant attributes by referring to Defining attributes in the language properties file.

Enter the following command to run the utility:

./data-transformation-utility.sh -l /path/to/<filename>.properties

These steps result in four copybook snippets being created in the artifactsLocation directory that is specified in the <filename>.properties file.

The names of these copybooks are based on the name of the JSON schema file that you use. For example, if your JSON schema file name is my-json-schema.json, the files generated are my-json-schema-ConsumerSnippet.cpy, my-json-schema-Copybook.cpy, my-json-schema-ProducerSnippet.cpy, and my-json-schema-VariablesSnippet.cpy.

Table 3. Generated artifacts
File name	Purpose
`my-json-schema-ConsumerSnippet.cpy`	This file contains the COBOL snippet that is used to generate the COBOL data from the JSON schema file in the case of a Kafka consumer.
`my-json-schema-Copybook.cpy`	This file contains the copybook structure that stores the COBOL data to be converted into a JSON object in the case of a Kafka producer, or the copybook that is used to store the COBOL data converted from a JSON object in the case of a Kafka consumer.
`my-json-schema-ProducerSnippet.cpy`	This file contains the COBOL snippet that is used to generate the JSON schema file from the COBOL data in the case of a Kafka producer.
`my-json-schema-VariablesSnippet.cpy`	This file contains the COBOL variables snippet that is used in both the Kafka producer and consumer use cases.