How the integration node processes files
The integration node reads files with the FileInput, FTEInput, CDInput, and FileRead nodes, and writes files with the FileOutput, CDOutput, and FTEOutput nodes.
- FileInput node: read and write permissions on the input directory and input files.
- FileOutput node: read and write permissions on the output directory.
- FileRead node: read permissions on the input directory and input files. Note: if a Finish File action is used (to move, rename or delete the file), write permissions on the input directory and input files is also required.
The FileInput, FileOutput, and FileRead nodes inherit the permissions assigned to the integration node's user ID and group. See your operating system documentation for details on how to set permissions and the impact on file access.
You can use the FTEInput and FTEOutput nodes to receive or send files to a destination on a IBM MQ File Transfer Edition network.
You can use the CDInput and CDOutput nodes to receive or send files to a destination on an IBM Sterling Connect:Direct® network.
A file, or a record within a file, is analogous to a message in a queue. The directory that contains the file is analogous to a message queue.
How the integration node reads a file at the start of a flow
The FileInput node processes messages that are read from files. The FileInput node searches a specified input directory (in the file system attached to the integration node) for files that match specified criteria. The node can also recursively search the input directory's subdirectories. Files that meet the criteria are processed in age order, that is the oldest files are processed first regardless of where they appear in the directory structure. Optionally, files from a remote FTP or SFTP server can be moved to the local directory whenever the directory is to be scanned. You can find the file that you require by specifying an explicit file name or a file name pattern that includes wildcard characters. If the file is locked, it is ignored during the directory scan. You can also exclude files from being read by specifying a file exclusion pattern. Whether files are either processed or ignored is determined first according to the file name or pattern and second according to the file exclusion pattern.
The FileInput node creates an mqsitransitin subdirectory in the input directory. The mqsitransitin subdirectory holds and locks the input files while they are being processed. The integration node reads the file and propagates a message, or messages, by using the contents of the file.
If an integration server that processes files in this input directory is removed, check the mqsitransitin subdirectory for partially processed or unprocessed files. Move any such files back into the input directory and remove the integration server UUID prefix from the file names, so that they can be processed by a different integration server. For more information about the mqsitransitin subdirectory, see How multiple file nodes share access to files in the same directory.
- A single record (whole file)
- Separate records, each of a fixed length (fixed-length records)
- Separate records each delimited by a specified delimiter (delimited records)
- Separate records that are recognized by a parser that you specify (parser record sequence)
After the file has been successfully processed, it is either deleted from the file system or moved to an archive subdirectory of the specified (local) directory.
When the last record of the file has been processed successfully, if the End of Data terminal of the FileInput node is connected in the message flow, an End of Data message is propagated to the End of Data terminal. The End of Data message consists of an empty BLOB message and a LocalEnvironment.File structure, and it allows explicit end-of-flow processing to be performed in another part of the message flow.
The message (or messages) propagated from the file can be used as input to any message flow and output node. You can create a message flow that receives messages from files and generates messages for clients that use any of the supported transports to connect to the integration node.
Whenever a message is propagated from a file, the FileInput node stores information about the file in the LocalEnvironment.File message tree. This information includes the offset of the record of the message in the file that is being processed, and the record number in that file. In addition, when a wildcard is used in a file name pattern, the characters matched in the file name are placed in the WildcardMatch element of the local environment tree.
If a file is backed out, or a file is left in the mqsitransitin subdirectory because processing failed, remove the integration server UUID prefix from the file name and move it back into the input directory. Processing is automatically retried, beginning at the first record in the file.
The FTEInput node receives files from the IBM MQ File Transfer Edition network. The integration node reads the file and propagates a message, or messages, by using the contents of the file. After an FTEInput node has processed a file, the file is deleted. For details of information that the FTEInput node provides in the local environment, see Using local environment variables with file nodes.
The CDInput node receives files from the IBM Sterling Connect:Direct network. The integration node reads the file and propagates a message, or messages, by using the contents of the file. After a CDInput node has processed a file, the file is deleted. For details of information that the CDInput node provides in the local environment, see Using local environment variables with file nodes.
The FileRead node can read a whole file, or a single, nominated record. A common use of the FileRead node is to route or enrich messages based on the contents of the file. When developing the message flow you can specify the name and location of the file to be read. You can override these values at run time based on the contents of a message. TheFileRead node reads a file in the middle of a message flow.
The FileRead node complements the existing FileInput and FileOutput nodes.
How the integration node reads a file from the middle of a flow
The FileRead node locks files when they are being read and does not move files to a subdirectory to read them. Files can be read completely or separated into records, in the same way as the FileInput node. After the file is processed by the FileRead node, it can be deleted, archived, or left unchanged.
When a FileRead Node has the Remote Transfer option selected, and a local file is not available, one is transferred from a remote server. Only one thread on a single integration server is permitted to transfer a particular file at any time. Any message flow invocations that require access to this same file wait until an in-flight transfer is completed and then use the local copy of the file. The FileRead node creates a lock in the locks subdirectory of the local input directory to prevent multiple integration servers from transferring the same file at the same time. This locking only functions correctly if the file system correctly implements Posix locking semantics, in particular NFS file systems must be at least NFS v4.
If the Delete remote file after successful transfer property is selected, then after successfully transferring a file it is deleted from the remote server, otherwise it is left in place.
If the FileRead Node has the Finish File Action property set to No Action, then it ensures that the input file referenced by the File Name pattern is unique in the input directory. This behavior means that if more than one file matches in the local directory due to a wildcard match, then an exception is thrown. This behavior is preserved for remote files, so if the Finish File Action is set to No Action and multiple files match the file pattern on the remote server, an exception is thrown. Under these conditions, the broker is unable to determine which of the multiple files present should be processed.
How the integration node writes a file
The FileOutput, CDOutput, and FTEOutput nodes write messages to files in the file system of the integration node. When a message is received on the In terminal of the node, it creates and writes a file as a series of one or more records. One record is written to a file for every message received. The name of the file is specified either by a file name pattern in the node or an explicit file name that is either specified in the node or derived from the message.
- A single record (whole file). The file that is created consists of one record.
- Concatenated records (unmodified records). The records are not padded to any required length, and they are not separated by a delimiter.
- Uniform length records (fixed-length records). Records that are shorter than the specified length are padded to the required length.
- Separate records (delimited records). Records are either terminated or separated by a specified delimiter.
- FileOutput node: The file is then moved to the specified
output directory. If this transfer fails, you can specify if the file is deleted,
moved to the failure directory (mqsifailure), renamed before being moved to the
failure directory, or no action is taken on the file.
Optionally, the file can be moved to a directory on a remote FTP or SFTP server. The server is identified by the Remote server and port property on the node. Alternatively, you can override the node property by setting a value in the local environment (see Local environment overrides for the remote server on the FileOutput node).
- FTEOutput node: The file is sent to the destination agent on the IBM MQ File Transfer Edition network.
- CDOutput node: The file is sent to the IBM Sterling Connect:Direct network.
If the output node produces a single record from the file (whole file), the file is moved immediately to the output directory without requiring a message to be propagated to the Finish File terminal. In this case, any message sent to the Finish File terminal of the node has no effect on any file, but the message is still propagated to a message flow attached to the End of Data terminal.
How data is appended to the end of a local file
- The first method is to write directly to the output directory. Each record that is written is appended immediately to the specified file. The contents of the file can be read while data is being written to the file. The Finish File action, and subsequently the Action on Finial File failure, are not effective unless FTP options are used. If FTP options are used, the Finish File action transfers the file to the remote machine.
- The second method is to stage the file in the mqsitransit directory. The records are collected in the file. If a file exists in the output directory when the Finish File action occurs, the file is moved into the transit directory and the data is appended to the end of this file. The file is then moved back to the output directory and the new data is now visible to an application.
On the FTP tab if Retain local file and Stage in the transit directory are both selected then Action if file exists occurs on the Finish File action.
How the FileOutput node deals with failures
Action on final file processing failure | Write Mode | Remote Transfer | Outcome |
---|---|---|---|
No Action | Stage in mqsitransit directory | Either | File is left in mqsitransit subdirectory unless Record Definition is Whole File then File is not written to output directory |
No Action | Write directly to the output file | No | File is not written to output directory |
No Action | Write directly to the output file | Yes | File is left in output directory unless there is a failure writing to the local output directory |
Delete | Stage in mqsitransit directory | Either | File is deleted from mqsitransit subdirectory |
Delete | Write directly to the output file | No | File is not written to output directory |
Delete | Write directly to the output file | Yes | File is deleted from the local output directory |
Move To mqsifailure, or move to mqsifailure and add timestampStage in mqsitransit directory | Stage in mqsitransit directory | Either | File is moved from mqsitransit subdirectory to mqsifailure subdirectory |
Move To mqsifailure, or move to mqsifailure and add timestampStage in mqsitransit directory | Write directly to the output file | No | File is not written to output directory, and mqsifailure subdirectory is not created |
Move To mqsifailure, or move to mqsifailure and add timestampStage in mqsitransit directory | Write directly to the output file | Yes | File is moved from the local output directory to mqsifailure subdirectory unless there is a failure that prevents the file being written to the local output directory. In this case, the file can not be moved to mqsifailure as it did not exist in the local output directory. |
How data is appended to the end of a remote file
The FileOutput node builds a file either in the output directory or in the mqsitransit directory. On the FTP tab, if Action if remote file exists is set to Append, when the Finish File action occurs the file is transferred to the remote machine by using the FTP Append verb. The data is appended to the end of the remote file or, if the file does not exist, the file is created.