Which fields from the input log records to include in CSV or JSON output

You must decide which fields from the input log records you want to include in the CSV or JSON output.

By default, the CSV and JSON commands select all fields from the input log records. This is typically more than you need, unless you plan to use the CSV or JSON for the type of deep dive analysis that you can perform using the log browser in the Transaction Analysis Workbench ISPF dialog.

For a few record types, ISPF dialog option 5 Analytics provides a curated list of fields as a suggested starting point.

Tip: To list all field names for a log code, create a form: on the primary option menu of the Transaction Analysis Workbench ISPF dialog, select option 2 Forms. Enter NEW on the command line. Specify the log type and code, and then press Enter. A new form is displayed, containing all fields in that log code. To exit without saving the new form, press the Cancel function key (F12).

Selecting fields individually by name

To select fields to include in CSV or JSON output, use either of the following methods:

Immediately after the CSV or JSON command, insert a FIELDS command that lists the field names.
Create a form, and then use the FORM parameter of the CSV or JSON command to refer to the name of the form.

Selecting all numeric fields

In some log record types, most or all numeric fields might be useful for performance analysis, while only some non-numeric fields might be useful for this purpose. For these log record types, it can be useful to include all numeric fields in the output, and then select non-numeric fields individually by name as required.

Including all numeric fields without naming them individually is useful when numeric fields are frequently added to a log record type to reflect new features in the system that generate those logs. New numeric fields are automatically included in output.

To include all numeric fields without naming them individually, specify the ALLNUMBERS parameter on the CSV or JSON command.

To select additional, non-numeric, fields by name, use the FIELDS command or FORM parameter.

Writing bit flags as separate fields

Some fields contain bit flags, where each bit conveys a different meaning. Each bit flag has its own field name.

To write bit flags as separate fields in the output, use the FIELDS command to explicitly specify the bit flag field names.

To list bit flag field names, use the log browser in the Transaction Analysis Workbench ISPF dialog to zoom on the corresponding parent field.

The value of an individual bit field is output as true or false for JSON, 1 or 0 for CSV. If the value is an accumulation of multiple instances of that field from repeating sections, the value is an integer that represents the number of instances where the bit flag is on.

Excluding missing fields from JSON output

By default, the JSON command excludes fields that are selected for output but are missing from the input log record.

This default behavior reduces data volume, which is especially useful if the pricing plan of your analytics platform is based on data volume.

To always include all fields that are selected for output, even when fields are missing from an input log record, specify the MISSING(INCLUDE) parameter on the JSON command. Fields that are missing from the input log record will have the JavaScript value null.

If you specify MISSING(INCLUDE), be aware that some JSON parsers interpret the value null as the string value "null". That behavior is typically undesirable. For example: if you use numeric functions to process the field, then a string value is inappropriate, and might cause errors.