How payload size and transformation affect performance for the API provider

When data passes through a IBM® z/OS® Connect server, the server needs to transform each byte in the payload between JSON and the language structure. Therefore, the larger the JSON payload, the more CPU processing time is needed. Although this data transformation is zIIP eligible, unnecessary CPU usage might slow down your overall performance.

When you create a service archive (.sar) file, you can use the API toolkit to reduce the size of your JSON payload by excluding unused fields or renaming fields, to improve performance. These features are not available with the build toolkit.

If your payload contains arrays, you can optimize the JSON payload by using OCCURS DEPENDING ON (ODO) in COBOL, and REFER in PL/I, or use array counters, or both. This method can reduce the number of array elements in the JSON payload.

Using ODO (COBOL) and REFER (PL/I) can also reduce the size of the language structure, which might further improve z/OS subsystem performance. For example, by reducing IMS message sizes.

The following sections describe these techniques in more detail:

Exclude unused fields

When creating IBM z/OS Connect services using the API toolkit, fields that are not needed for your service should be excluded from the JSON payload by clearing the checkbox for those fields in the Include column in the Service Interface Editor. This method reduces the size of the JSON payload and makes the schemas more accurate as they do not describe properties that are not relevant.

 Example 1 

In this example, the request and response use the same copybook, but different fields. When the same copybook is used for both the request and the response, define separate request and response service interfaces if you plan to exclude or rename different fields in the request versus the response.

By default, all fields are included for both the request and the response. So for an HTTP PUT or POST, all fields in the request and response are subject to data transformation. For an HTTP GET, all fields in the response are subject to data transformation.

Figure 1 shows an HTTP PUT request that includes a 100 element array. The copybook of 30,376 bytes generates a JSON payload of 64,496 bytes.
Figure 1. A request that includes all fields.
A screen capture showing a sample PUT request with a 100 element array

Transforming unnecessary fields in this JSON payload is inefficient and degrades performance. Therefore, clear the Include checkbox of fields that are not needed for the request, and similarly for the response, as shown in Figure 2. In this example, the request does not require all the response fields to be checked, as the COBOL application does not require these fields as input to the program. This reduces the number of bytes from 30,376 to 112, and reduces the JSON payload size from approximately 64,496 to 122.

Figure 2. A request that excludes unnecessary fields.
A screen capture showing a request with some response fields unchecked
 Example 2 
The request in Figure 3 is asking for details about a user’s car insurance but is not interested in the user’s holiday insurance or building insurance. Therefore, by clearing the checkboxes for the holiday insurance and building insurance, the size of the JSON payload is reduced considerably.
Figure 3. A request that excludes holiday and building insurance fields.
A screen capture showing a request to an insurance application that excludes unnecessary fields

Rename fields

When a copybook is imported in the API toolkit, the field names that are shown in the Interface Rename column in the Service Interface Editor default to the same name as the original field name. For example, the following JSON snippet in Figure 4 is generated from part of the COBOL copybook imported in Figure 4.
Figure 4. A JSON schema with default field names
{  
    "CAR_CLAIMS_DATA": {
        "CAR_CLAIM_DATE": "...",
        "CAR_CLAIM_DETAILS": "...",
        "CAR_CLAIM_PAID_OUT": "..."
    }
}
These field names are not good practice and an API developer fluent in JSON would expect to see something similar to Figure 5.
Figure 5. A JSON schema with renamed fields
{  
    "carClaims": {
        "date": "...",
        "details": "...",
        "paidOut": "..."
    }
}

To rename fields, use the Interface Rename feature of the API toolkit. Renaming these fields to names shorter than the default saves CPU processing cost by reducing the amount of JSON parsing or generation that is needed, and also provides field names that are more familiar to the JSON programmer.

Figure 6 is an extract from a typical data structure imported from a COBOL copybook and shows how the fields are renamed in the Interface Rename column to shorter names that are more recognizable to an API developer. These renamed fields are used in the JSON payload, so the shorter the names, the less transformation is needed, resulting in improved performance.
Figure 6. A request with renamed fields.
A screen capture showing a request to an insurance application that excludes unnecessary fields

This saving is especially noticeable when large arrays are used in the data structure, as shown in the Figure 6.

Array sizes

The size of the JSON payload can also be reduced by using array sizes determined by using the following techniques:

  • OCCURS DEPENDING ON (ODO) for COBOL copybooks, or REFER for PL/I includes.
  • Array counters.
ODO or REFER
In Figure 7, CARMODEL_INFO_DATA has a maximum of 100 elements in the array. However, a user might want to request only the first 10 cars listed in a database, rather than have the maximum set of 100 cars returned. Therefore, by specifying 10 in the request, only 10 elements of the array will be included in the JSON response payload, instead of 100 elements, considerably reducing the size of the payload to be transformed.
Figure 7. A copybook with an array and ODO.
A copybook with an array and ODO
Array counters
You can use array counters in two ways,
  • To optimize the JSON payload for fixed-length arrays.
  • To optimize the JSON payload for nested variable-length arrays (ODO or REFER).
Array counters for a fixed-length or variable-length array.
Consider the following snippet from a fixed array copybook definition.
15 CARMODEL-INFO-TABLE.
   20 CARMODEL-INFO-DATA OCCURS 100 TIMES.
      25 CARMODEL-1-REF                 PIC X(12).
      25 CARMODEL-2-MODEL               PIC X(12).
      25 CARMODEL-3-COST                PIC X(12).
      25 CARMODEL-4-LEATHERSEATS        PIC X(12).
           ::           ::          ::
After you import this copybook into the API toolkit, depending on the COBOL (or PL/I) application, it might be possible to specify an array counter to limit the number of array elements.
Figure 8. Specifying an array counter to limit the number of array elements.
Specifying an array counter to limit the number of array elements.

In this example, the number of entries in the array field CARMODEL_INFO_DATA, which is an ODO/REFER array subject, is controlled by array counter REQ_MAX_NBR_OF_REC_REQUESTED. Specifying an array counter allows you to set a minimum number of entries so that the array appears variable-length to an API client.

Array counters with nested ODO or REFER.
Nested ODO or REFER statements can generate very large JSON payloads. The use of array counters can reduce the size of these JSON payloads considerably, and improve overall performance.

The following example demonstrates the value of an array counter in the context of a nested ODO array.

The field CARMODEL-NBR-OF-COMMENTS is used to maintain the count of the ODO array CARMODEL-COMMENTS by using the arrays counter support. This allows each CARMODEL-INFO-DATA to have a different number of comments in the request or response JSON within the range of 1 to (REQ-MAX-NBR-OF-COMMENTS <= 20) times.
05 REQ-NBR-OF-REC-RETURNED PIC 9(09) COMP.
05 REQ-MAX-NBR-OF-COMMENTS PIC 9(09) COMP.

          ::       ::       ::

15 CARMODEL-INFO-TABLE.
   20 CARMODEL-INFO-DATA OCCURS 1 TO 100 TIMES
         DEPENDING ON REQ-NBR-OF-REC-RETURNED.
      25 CARMODEL-NBR-OF-COMMENTS PIC 9(09) COMP.
      25 CARMODEL-COMMENTS OCCURS 1 TO 20 TIMES
            DEPENDING ON REQ-MAX-NBR-OF-COMMENTS.
         30 CARMODEL-COMMENT-A.
         30 CARMODEL-COMMENT-B.
         30 CARMODEL-COMMENT-C.

As the arrays are nested, the size of the JSON payload can become very large. Therefore, the use of array counters can reduce the size of the JSON payload, and improve performance as less transformation is needed.

Figure 9 shows the two arrays, with the second ODO highlighted, and shows the array counter that is assigned to it.
Figure 9. An array counter used in a nested ODO array.
An array counter used in a nested ODO array.

For request service interfaces, ODO/REFER array objects are automatically set by IBM z/OS Connect so that the application receives only as many array entries in the ODO/REFER array subject as are present in the JSON. For nested ODO/REFER array subjects, respective ODO array objects are automatically set to the maximum number of entries across all occurrences of the nested array subject in the request JSON.

For response service interfaces, ODO/REFER array objects must be set by the application so that IBM z/OS Connect processes only the requested number of array entries in the respective ODO/REFER array subjects. For nested ODO/REFER array subjects, respective ODO array objects must be set by the application to the maximum number of entries across all occurrences of the array subject in the response data.

For more information about array counters, see Defining array counters.

Conclusion

You can use these techniques separately or concurrently to ensure that IBM z/OS Connect processes only the payload data that is relevant to your application and eliminate unnecessary transformation.