Whole File Transformer
The Whole File Transformer processor transforms fully written Avro files to highly efficient, columnar Parquet files. Use the Whole File Transformer in a flow that reads Avro files as whole files and writes the transformed Parquet files as whole files.
Sources and targets that support whole files include cloud storage stages such as Amazon S3, Azure Data Lake Storage, and Google Cloud Storage, and as well as local and remote file systems such as Local FS and SFTP/FTP/FTPS Client. For a full list of whole file sources and targets or more information about the whole file data format, see Whole file data format.
You can use the Whole File Transformer to convert Avro files to Parquet within a flow.
When performing the conversion of Avro files to Parquet, the Whole File Transformer performs the conversion in memory, then writes a temporary Parquet file in a local directory on the Data Collector machine. Ensure that Data Collector has the necessary memory and storage to perform this processing.
When you configure the Whole File Transformer, you specify the local directory to use. You can configure a prefix and suffix for the resulting Parquet files and the buffer size and rate at which to process the Avro files. You can also configure standard Parquet properties, such as a compression codec, row group size, and page size.
Typically, when using the Whole File Transformer processor, you will use a separate flow for processing data and generating the Avro files to be converted. But if you have Avro files generated by a third party, then you can simply create the flow to convert the files to Parquet.
Implementation overview
When using the Whole File Transformer processor, you will typically configure two flows. The flow that includes the Whole File Transformer must be a whole file flow, which does not permit additional processing of file data. As a result, any processing that you require must take place in a separate flow.
This separate flow includes a source to read data of any data format, the processors that you need, and a file-based target to write Avro files. Let's call this flow the Write Avro flow. If you already have Avro files ready, you can skip this flow.
Then, a second flow reads the Avro files using the whole file data format, transforms the files to Parquet using the Whole File Transformer, then writes the Parquet files as whole files. Let's call this the Parquet Conversion flow.
Together, the implementation looks like this:
Write Avro flow
The Write Avro flow reads data from the source, performs any required processing, and uses a file-based Avro target such as Local FS to write Avro files to a file system.
The target file system acts as a staging area. As the Write Avro flow closes each Avro file, the Parquet Conversion flow can begin processing the file and convert it to Parquet.
Since each Avro file is ultimately transformed to a corresponding Parquet file, the Avro files should be of substantial size to take advantage of Parquet capabilities. 2 GB is recommended.
The Write Avro flow includes any source, any number of processors, and an Avro-enabled file-based target, as follows:
For a list of targets that support writing Avro, see Data format support.
Parquet conversion flow
The Parquet Conversion flow processes the closed Avro files as whole files, leveraging the efficiency of the whole file data format. This flow converts the Avro files to Parquet using the Whole File Transformer processor.
The Parquet Conversion flow must use a file-based source enabled for whole files, such as the Directory source, to process the large Avro files.
The Whole File Transformer processor converts each Avro file to Parquet, writing each Parquet file temporarily to a user-defined Data Collector directory. Then a whole file target, such as Google Cloud Storage, moves the Parquet file to the target system.
Whole files can only be processed by the Whole File Transformer, so this flow performs no additional processing.
As a result, the Parquet Conversion flow looks like this:
For a step-by-step review of creating these flows, see the implementation example.
For a full list of whole file sources and targets, see Data format support.
Memory and storage requirements
When performing the conversion of Avro files to Parquet, the Whole File Transformer performs the conversion in memory, then writes a temporary Parquet file in a local directory on the Data Collector machine. Ensure that Data Collector has the necessary memory and storage to perform this processing.
The available memory is determined by the Data Collector Java heap size. The available storage is the available disk space on the Data Collector machine. Flows using the Whole File Transformer processor require available memory and storage based on the size of the largest Avro file to be converted.
For example, when converting files that are up to 2 GB in size, ensure that Data Collector has at least 2 GB of available memory and storage when the flow runs.
Generated record
When processing whole files, the Whole File Transformer processor updates the file information in the whole file record and adds a record header attribute.
Whole file records include fields that contain file reference information. The record does not include any actual data from the file to be streamed.
The Whole File Transformer processor moves the existing information in the fileInfo map field,
generated by the flow
source, to a new record header attribute named sourceFileInfo. It
then updates the fileInfo field with information for the temporary Parquet file, such as the updated
path and file location.
Implementation example: Amazon S3 Parquet conversion
Let's say we have several thousand small objects in Amazon S3. We want to convert the numerous small objects to large Parquet objects so we can more quickly analyze the data.
To do this, we set up two flows. The first flow reads the Amazon S3 objects and writes large Avro files to a local file system. This is the Write Avro flow. The second Parquet Conversion flow reads the Avro files as whole files and transforms each file to a corresponding Parquet file, which is written back to S3.
The resulting workflow looks like this:
Write Avro flow example
The Write Avro flow can be pretty simple - it just needs to write large Avro files. You can make it complicated by performing additional processing, but at its most basic, all you need is a source and an Avro-enabled file-based target to write the Avro files.
The Write Avro flow uses the Amazon S3 source to read the multitude of small S3 objects. Then, it writes Avro files to a local file system using the Local FS target, as follows:
Note the following configuration details:
- Amazon S3 source
- The Amazon S3 source requires no special configuration. We simply configure it to access the objects to be processed and specify the data format of the data. Configure error handling, post-processing, and other properties as needed.
- Local FS target
- In the Local FS target, we carefully configure the following properties:
- Directory Template - This property determines where the Avro files are written. The directory
template that we use determines where the source of the second flow picks up the Avro
files.
We'll use
/avro2parquet/. - File Prefix - This is an optional prefix for the output file name.
When writing to a file, the Local FS target creates a temporary file that uses
tmp_as a file name prefix. To ensure that the second flow picks up only fully-written output files, we define a file prefix for the output files. It can be something simple, likeavro_. - Read Order - To read the files in the order they are written, we use
Last Modified Timestamp. - Max File Size - The Avro files are converted to Parquet files in the second flow,
so we want to ensure that these output files are fairly large. Let's say we want the Parquet files to be roughly 3 GB in size, so we set Max File Size to
3 GB.Important: To transform the Avro files, the Whole File Transformer requires Data Collector memory and storage equivalent to the maximum file size. Consider this requirement when setting the maximum file size. - Data Format - We select
Avroas the data format, and specify the location of the Avro schema to be used.
- Directory Template - This property determines where the Avro files are written. The directory
template that we use determines where the source of the second flow picks up the Avro
files.
Parquet conversion flow example
The Parquet Conversion flow is simple as well. It uses the whole file data format, so no processing of file data can occur except the conversion of the Avro files to Parquet.
We use the Directory source and whole file data format to read the fully written Avro files. Then comes the Whole File Transformer processor for the Parquet conversion, and the Amazon S3 target to write the Parquet files back to S3 as whole files.
The Parquet Conversion flow looks like this:
Note the following configuration details:
- Directory source
- In the Directory source, we carefully configure the following properties:
- Files Directory - To read the files written by the first flow, point this to the
directory used in the Local FS Directory Template property:
/avro2parquet/. - File Name Pattern - To pick up all output files, use a glob for file name pattern:
avro_*.We use the
avro_prefix in the pattern to avoid reading the active temporary files generated by the Local FS target. - Data Format - We use
Whole Fileto stream the data to the Whole File Transformer.
- Files Directory - To read the files written by the first flow, point this to the
directory used in the Local FS Directory Template property:
- Whole File Transformer processor
- The Whole File Transformer processor converts Avro files to Parquet in memory, then writes temporary Parquet files to a local directory. These whole files are then streamed from the local directory to Amazon S3.
- Amazon S3 target
- The Amazon S3 target streams the temporary Parquet files from the Whole File Transformer temporary file directory to Amazon S3.
Runtime
We start both flows at the same time.
The Write Avro flow processes Amazon S3 objects and writes 3 GB Avro files to the
/avro2parquet/ directory. The output files are named avro_<file
name>.
After the first file is written, the Parquet Conversion flow picks up the file. It
streams the file from the Directory source to the Whole File Transformer processor. The Whole File
Transformer converts the Avro file to Parquet in memory, and writes it to the specified temporary
directory, /tmp/out/.parquet. The temporary file then streams from the Whole File
Transformer to the Amazon S3 location defined in the target.
In the end, the multitude of Avro files are converted to large columnar Parquet files and streamed back to Amazon S3 for easy analysis. Two simple flows, one Whole File Transformer, and you're done!