Reading and writing files environment variables in DataStage

By default, the DataStage reads ahead 500 bytes to get the next delimiter. For streaming inputs (socket, FIFO, and others) this behavior is suboptimal, since the DataStage might block (and not output any records). When DataStage reads a delimited record, it reads this many bytes (minimum legal value is 2) instead of 500. If a delimiter is NOT available within N bytes, N is incremented by a factor of 2 (when this environment variable is not set, this factor changes to 4).

The default is 128 (that is, 128 KB). It can be set to values from 8 upward, but is clamped to a minimum value of 8. That is, if you set it to a value less than 8, then 8 is used. Tune this value upward for long-latency files (typically from heavily loaded file servers).

The default is 128 (that is, 128 KB). It can be set to values from 8 upward, but is clamped to a minimum value of 8. That is, if you set it to a value less than 8, then 8 is used. Tune this upward for long-latency files (typically from heavily loaded file servers).

By default, records that do not contain all the fields in the import schema are rejected by the import operator. Set the APT_IMPORT_HANDLE_SHORT environment variable so that these records are imported successfully. The missing fields are given the default value for their type, or null if the field is nullable.

Setting the APT_IMPORT_HANDLE_SHORT environment variable disables optimization for fixed-length schemas by the copy operator, which might cause slower performance for some fixed-length schemas.

The APT_IMPORT_HANDLE_SHORT environment variable is not supported for importing fixed-length records without a record limiter, as incorrect data might be imported.

In DataStage Version 11.3 the behavior of the Sequential File stage was changed so that a file pattern is converted into a fileset before the files are imported. This change allows the files to be processed in parallel rather than sequentially.

Set the APT_IMPORT_PATTERN_USES_CAT environment variable to force the behavior of prior versions of DataStage. This prior behavior was to concatenate all the files that match a file pattern before importing the files sequentially.

If the APT_IMPORT_PATTERN_USES_FILESET_MOUNTED environment variable is set and the pattern indicates that the files are on a mounted file system, all the available nodes are used for the file set.

By default, when reading, DataStage will read ahead 500 bytes to get the next delimiter. If it is not found, DataStage looks ahead 4*500=2000 (1500 more) bytes, and so on (4X) up to 100,000 bytes. This variable controls the upper bound which is by default 100,000 bytes. Use this variable instead of APT_DELIMITED_READ_SIZE when a larger than 500 bytes read-ahead is wanted.

By default, if a fixed-length string contains all characters it is treated as an empty string. Defining this environment variable overrides the default behavior. For example, even if the fixed length string contains all characters, it is treated as a nonempty string.

It changes Sequential File export default behavior from writing a space before positive decimals to writing a plus character. Its default value is False. This variable has a Boolean type.

It creates quiet warnings during reading string or unstring fields that are longer than the defined maximum length. Its default value is False. This variable has a Boolean type.

When APT_IMPORT_PATTERN_USES_FILESET is set, sequential read (import) turns any file pattern into a fileset before processing the files. It allows the files to be processed in parallel as apposed to sequentially. By default, file pattern processing concatenates the files to be used as the input. Its default value is False. This variable has a Boolean type.

By default import turns any file pattern into a fileset before processing the files by using the ls command to get the list of files. Setting this environment variable will instead use the find command to get the list of files. It can be used to work around limitations with ls on systems with low ARG_MAX settings which limit the number of files a pattern can read. Find behavior is not identical to ls so the order of files and processing of wild cards can differ. Its default value is False. This variable has a Boolean type.

It disables the feature of importing files containing spaces in their names using file patterns. Its default value is False. This variable has a Boolean type.

It enables final_delimitor behavior as implemented prior to DataStage 7.0.1. Its default value is False. This variable has a Boolean type.

It defines the size of the data that is read from S3 in a single chunk. Its default value is “500”. This variable has a Number type.

It defines the size of the part uploaded to S3 as a single chunk. Its default value is “500”. This variable has a Number type.