Local FS
Use the Local FS destination to write records to files in a local file system.
When you configure a Local FS destination, you can define a directory template and time basis to determine the output directories that the destination creates and the files where records are written.
You can define a file prefix and suffix, the data time zone, and properties that define when the destination closes a file. You can specify the amount of time that a record can be written to its associated directory and what happens to late records.
You can write records to the specified directory, use the defined Avro schema, and roll files based on record header attributes. For more information, see Record Header Attributes for Record-Based Writes.
The destination can generate events for an event stream. For more information about the event framework, see Dataflow Triggers Overview.
You can use Gzip, Bzip2, Snappy, LZ4, and other compression formats to write output files.
Directory Templates
The Local FS destination uses directory templates to create output and late record directories. Local FS writes records to the directories based on the configured time basis.
You can alternatively write records to directories based
on the targetDirectory
record header attribute. Using the
targetDirectory
attribute disables the ability to define
directory templates.
When you define a directory template, you can use a mix
of constants, field values, and datetime variables. You can use the
every
function to create new directories at regular
intervals based on hours, minutes, or seconds, starting on the hour. You can also
use the record:valueOrDefault
function to create new directories
from field values or a default in the directory template.
/outputfiles/${record:valueOrDefault("/State", "unknown")}/${YY()}-${MM()}-${DD()}-${hh()}
- Constants
- You can use any constant, such as
output
. - Datetime Variables
- You can use datetime variables, such as
${YYYY()}
or${DD()}
. The destination creates directories as needed, based on the smallest datetime variable that you use. For example, if the smallest variable is hours, then the directories are created for every hour of the day that receives output records. every
function- You can use the
every
function in a directory template to create directories at regular intervals based on hours, minutes, or seconds, beginning on the hour. The intervals should be a submultiple or integer factor of 60. For example, you can create directories every 15 minutes or 30 seconds. record:valueOrDefault
function- You can use the
record:valueOrDefault
function in a directory template to create directories with the value of a field or the specified default value if the field does not exist or if the field is null:${record:valueOrDefault(<field path>, <default value>)}
Time Basis
When using directory templates, the time basis helps determine when directories are created. It also determines the directory Local FS uses when writing a record, and whether a record is late.
When using the targetDirectory
record
header attribute to write records, the time basis determines only whether a
record is late.
You can use the following times as the time basis:
- Processing Time
- When you use processing time as the time basis, the destination creates directories based on the processing time and the directory template, and writes records to the directories based on when they are processed.
- Record Time
- When you use the time associated with a record as the time basis, you specify a Date field in the record. The destination creates directories based on the datetimes associated with the records and writes the records to the appropriate directories.
Late Records and Late Record Handling
When you use a record time as the time basis, you can define a time limit for records to be written to their associated output file. When the destination creates a new output file in a new directory, the previous output file is kept open for the specified late record time limit. When records that belong in that file arrive within the time limit, the destination writes the records to the open output file. When the late record time limit is reached, the output file is closed and any record that arrives past this limit is considered late.
You can send late records to a late records file or to the stage for error handling. When you send records to a late records file, you define a late records directory template.
/tmp/out/${YYYY()}-${MM()}-${DD()}-${hh()}
The first records that arrive have a datetime between the hours of 02:00 and 02:59, and so are written to an output file in the 02 directory. When records with a datetime between the hours of 03:00 and 03:59 arrive, the destination creates a new file in an 03 directory. The destination keeps the file in the 02 directory open for another hour.
If a record with a datetime between the hours of 02:00 and 02:59 arrives before the hour time limit, the destination writes the record to the open file in the 02 directory. After one hour, the destination closes the output file in the 02 directory. Any records with a datetime between the hours of 02:00 and 02:59 that arrive after the one-hour time limit are considered late. The late records are sent to the stage for error handling.
Timeout to Close Idle Files
You can configure the maximum time that an open output file can remain idle. After no records are written to an output file for the specified amount of time, the Local FS destination closes the file.
You might want to configure an idle timeout when output files remain open and idle for too long, thus delaying another system from processing the files.
- You configured the maximum number of records to be written to output files or the maximum size of output files, but records have stopped arriving. An output file that has not reached the maximum number of records or the maximum file size stays open until more records arrive.
- You configured a date field in the record as the time basis and have
configured a late record time limit, but records arrive in
chronological order. When a new directory is created, the output file
in the previous directory remains open for the configured late record
time limit. However, no records are ever written to the open file in
the previous directory.
For example, when a record with a datetime of 03:00 arrives, the destination creates a new file in a new 03 directory. The previous file in the 02 directory is kept open for the late record time limit, which is an hour by default. However, when records arrive in chronological order, no records that belong in the 02 directory arrive after the 03 directory is created.
In either situation, configure an idle timeout so that other systems can process the files sooner, instead of waiting for the configured maximum records, maximum file size, or late records conditions to occur.
Recovery
The Local FS destination supports recovery after an unexpected stop of the pipeline by renaming temporary files when the pipeline restarts.
_tmp_<prefix>_<runnerId>
Where
<prefix>
is the file prefix defined for the destination
and <runnerId>
is the ID of the pipeline runner performing the
pipeline processing. For example, when the destination prefix is defined as
sdc
and the destination runs from a single-threaded
pipeline, the temporary file is named like so: _tmp_sdc_0
. _tmp_
string and to
replace the pipeline runner ID with a random unique identifier like
so:<prefix>_e7ce67c5-013d-47a7-9496-8c882ddb28a0
However, when the pipeline stops unexpectedly, the temporary files remain. When the pipeline restarts, the destination scans all subdirectories of the defined directory template to rename any temporary files that match the defined prefix for the destination. After the destination renames the temporary files, it continues writing to new output files.
_tmp_<prefix>
-
the destination renames that file also.- The directory template includes an expression with the record:value or record:valueOrDefault function.
- If the record:value or record:valueOrDefault function evaluates to an empty string or to a subdirectory, the destination cannot determine those locations when the pipeline restarts. As a result, the destination cannot rename any temporary files written to those locations.
- The directory is defined in the targetDirectory record header attribute.
- When the directory is defined in the targetDirectory record header attribute, the destination cannot determine where to look for temporary files when the pipeline restarts. As a result, it cannot rename the temporary files.
In either of these situations, you must manually rename the temporary files.
File recovery can slow down the pipeline as it restarts. If needed, you can configure the destination to skip file recovery.
Event Generation
The Local FS destination can generate events that you can use in an event stream. When you enable event generation, the destination generates event records each time the destination closes a file or completes streaming a whole file.
- With the HDFS File Metadata executor to move
or change permissions on closed files.
For an example, see Managing Output Files.
- With the Email executor to send a custom email
after receiving an event.
For an example, see Sending Email During Pipeline Processing.
- With a destination to store event information.
For an example, see Preserving an Audit Trail of Events.
For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.
Event Records
Local FS event records include the following event-related record header attributes. Record header attributes are stored as String values:
Record Header Attribute | Description |
---|---|
sdc.event.type | Event type. Uses one of the following types:
|
sdc.event.version | Integer that indicates the version of the event record type. |
sdc.event.creation_timestamp | Epoch timestamp when the stage created the event. |
- File closure
- The destination generates a file closure event record when it closes an output file.
- Whole file processed
- The
destination generates an event record when it completes
streaming a whole file. Whole file event records have the
sdc.event.type
record header attribute set towholeFileProcessed
and have the following fields:Field Description sourceFileInfo A map of attributes about the original whole file that was processed. The attributes include: - size - Size of the whole file in bytes.
Additional attributes depend on the information provided by the origin system.
targetFileInfo A map of attributes about the whole file written to the destination. The attributes include: - path - An absolute path to the processed whole file.
checksum Checksum generated for the written file. Included only when you configure the destination to include checksums in the event record.
checksumAlgorithm Algorithm used to generate the checksum. Included only when you configure the destination to include checksums in the event record.
Data Formats
- Avro
- The destination writes records based on the Avro schema. You can use one of the following methods to specify the location of the Avro schema definition:
- Binary
- The stage writes binary data to a single field in the record.
- Delimited
- The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.
- JSON
- The destination writes records as JSON data. You can use one of
the following formats:
- Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
- Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
- Parquet
- The destination writes a Parquet file for each partition and includes the Parquet schema in every file.
- Protobuf
- Writes a batch of messages in each file.
- SDC Record
- The destination writes records in the SDC Record data format.
- Text
- The destination writes data from a single text field to the destination system. When you configure the stage, you select the field to use.
- Whole File
- Streams whole files to the destination system. The destination writes the data to the file and location defined in the stage. If a file of the same name already exists, you can configure the destination to overwrite the existing file or send the current file to error.
Configuring a Local FS Destination
Configure a Local FS destination to write data to a local file system.