Controlling how files are separated into records

Set the Record detection and other properties on node Records and Elements tabs to read files in different formats.

The following examples are based on the examples described in Using a local file as input for your message flow and Reading a file on a remote FTP, FTPS, or SFTP directory. In each case, the input file to use, the property settings, and the expected results are described.
The examples, which describe using the FileInput node, can be applied to the FTEInput node, with the following provisos:
  • The FTEInput node has no Basic tab.
  • FTP and SFTP are not used to transmit files to an FTEInput node.
The examples can also be applied to the FileRead node, with the following provisos:
  • The FileRead node propagates only one record from the file and, by default, this record is the first record in the file. The FileRead node can be configured to propagate a specific record. For more information see Routing or enriching a message based on the contents of a file.
  • FTP and SFTP are not used to transmit files to an FileRead node.

Example 1. Records are separated by a DOS or UNIX line end

This example is identical to the one described in Using a local file as input for your message flow or Reading a file on a remote FTP, FTPS, or SFTP directory. Create an input file called test_input1.xml with the following content:
<Message>test1</Message>
<Message>testtwo</Message>
<Message>testthree</Message>
Each line ends with a line terminator.
The properties to set are:
Tab Property Value
Records and Elements Record detection Delimited
Delimiter DOS or UNIX Line End
Delimiter type Postfix
The FileInput node detects records that end with a DOS or UNIX line end and creates a message for each one that it finds.
The result is the propagation of three messages, as follows:
  • Message 1:
    <Message>test1</Message>
  • Message 2:
    <Message>testtwo</Message>
  • Message 3:
    <Message>testthree</Message>
The DOS or UNIX line end is not part of any propagated message.

Example 2. Records are separated by a custom delimiter

Create an input file called test_input2.xml with the following content:
<Message>test01</Message>,<Message>test001</Message>,<Message>test0001</Message>
There must be no line terminator at the end of this file data; the XMLNSC parser ignores the line terminator if it is present.
In addition to the property settings described in Using a local file as input for your message flow or Reading a file on a remote FTP, FTPS, or SFTP directory, set these properties:
Tab Property Value
Basic File name or pattern test_input2.xml
Records and Elements Record detection Delimited
Delimiter Custom Delimiter
Custom delimiter 2C
Delimiter type Infix
The hexadecimal X'2C' represents a comma in ASCII. On other systems, you must use a different hexadecimal code.

The FileInput node detects the comma character and uses it to separate records. Because the value of the Delimiter type property is Infix, a comma is not required at the end of the file.

The result is the propagation of three messages, as follows:
  • Message 1:
    <Message>test01</Message>
  • Message 2:
    <Message>test001</Message>
  • Message 3:
    <Message>test0001</Message>
The comma character is not part of any propagated message. There are no commas in the bodies of the message in this example; if the message bodies did contain commas, the records would be split at those points resulting in incorrectly formed messages being propagated to the rest of the flow.

Example 3. Records are separated by a fixed number of bytes

Create an input file called test_input3.xml with the following content:
<Message>123456789</Message><Message>abcdefghi</Message><Message>rstuvwxyz</Message>
There must be no line terminator at the end of this file.
In addition to the property settings described in Using a local file as input for your message flow or Reading a file on a remote FTP, FTPS, or SFTP directory, set these properties:
Tab Property Value
Basic File name or pattern test_input3.xml
Records and Elements Record detection Fixed Length
Length 28
The FileInput node splits the input file into records each 28 bytes long.
The result is the propagation of three messages, as follows:
  • Message 1:
    <Message>123456789</Message>
  • Message 2:
    <Message>abcdefghi</Message>
  • Message 3:
    <Message>rstuvwxyz</Message>
Each message is 28 bytes long. If the file contains trailing bytes, for example a carriage return-line feed pair, a further message containing these bytes is propagated; the trailing bytes might or might not be recognized by the message domain, message model, and message type assigned to parse the message.

Example 4. Records are whole files

Create an input file called test_input4.xml with the following content:
<Message>Text string of a length decided by you, even including line 
terminators, as long as it only contains this tag at the end.</Message>
There must be no line terminator at the end of this file; if there is one, it has no effect.
In addition to the property settings described in Using a local file as input for your message flow or Reading a file on a remote FTP, FTPS, or SFTP directory, set these properties:
Tab Property Value
Basic File name or pattern test_input4.xml
Records and Elements Record detection Whole File
The FileInput node does not split the file; it propagates all of the content of the file as a single record to be parsed by the message domain, message model, and message type as specified on the node. In this example, you are using the XMLNSC parser and message set xml1, so the message is recognized.
The result is the propagation of one message, as follows:
  • Message 1:
    <Message>Text string of a length decided by you, even including line terminators, as long as
it only contains this tag at the end.</Message>
Trailing bytes (for example, line terminators) are included.

Example 5. Records are recognized as separate messages by the parser specified in the Message domain property

Create an input file called test_input5.xml with the following content:
<Message>Text string of a length decided by you  </Message><Message>and another</Message>
<Message>and another on a new line</Message>
Line terminators at the end of this file, or at the end of lines, are acceptable.
In addition to the property settings described in Using a local file as input for your message flow or Reading a file on a remote FTP, FTPS, or SFTP directory, set these properties:
Table 1.
Tab Property Value
Basic File name or pattern test_input5.xml
Records and Elements Record detection Parsed Record Sequence
The FileInput node defers to the parser to determine the record boundaries. In this example, message set xml1 in domain XMLNSC must recognize the complete <Message> XML format. XMLNSC absorbs trailing white space (for example, line terminators).
The result is the propagation of three messages, as follows:
  • Message 1:
    <Message>Text string of a length decided by you  </Message>
  • Message 2:
    <Message>and another</Message>
  • Message 3:
    <Message>and another on a new line</Message>
Trailing white space (for example, line terminators) are included in the messages.