Apache Iceberg for AWS

7.3 and later

The Apache Iceberg for AWS destination writes data to an Apache Iceberg table stored on Amazon S3. The destination does not write data to Iceberg on other data platforms. For information about supported versions, see Supported Systems and Versions in the Data Collector documentation.

To write data to an Iceberg table on AWS, the Apache Iceberg for AWS destination writes Avro or Parquet files as objects to an Amazon S3 bucket. Then, the destination uses an AWS Glue data catalog to register the files with Iceberg.

With the Apache Iceberg for AWS destination, you specify the region, bucket, and common prefix to define where to write objects. You select the authentication method that the destination uses to connect to Amazon S3, and you can optionally enable server-side encryption. You can also use a connection to configure the destination.

The Apache Iceberg for AWS destination can write to an existing Iceberg table or a new table. When the destination creates a table, you can define partitioning for the table.

When you configure the destination, you select the write mode and schema evolution mode to use. The schema evolution mode defines whether and how the destination updates the Iceberg table to allow for schema changes in the data.

Before you use the Apache Iceberg for AWS destination, complete the AWS prerequisite tasks.

Important: By default, the destination does not process timestamp data. For more information, see Processing timestamp data.

AWS Prerequisites

To write data to Iceberg tables on AWS, the Apache Iceberg for AWS destination writes Avro or Parquet files as objects to an Amazon S3 bucket. Then, the destination uses an AWS Glue data catalog to register the file with Iceberg.

Before you use the destination, complete the following tasks as needed:

Create an AWS Glue database for use with Iceberg.
For more information about using Iceberg with AWS Glue, see the Amazon documentation.
Assign required permissions.
The minimum required permissions are as follows:
- Read and write objects in the warehouse S3 bucket.
- List the warehouse S3 bucket and relevant prefixes.
- Create, read, and update AWS Glue database and table metadata that is used by the Iceberg table.

Creating the Iceberg table

You can configure the Apache Iceberg for AWS destination to create the Iceberg table that the destination writes to.

The Apache Iceberg for AWS destination creates Iceberg tables based on the schema of the first record that it processes, and the specified table name and location.

Use the Create table property to create the Iceberg table. Optionally use the Nullable fields property to make all columns in the table optional or required. You can also configure partitioning for the table.

Partitioning a new table

When the Apache Iceberg for AWS destination creates an Iceberg table, you can define partitioning for the table.

To partition a table, you choose the Iceberg partition transform to use in the Partition specification property. You can use a single partition transform for the table. If you do not specify a partition transform, the table includes no partitions.

You can use one of the following Iceberg partition transforms:

Identity - Creates partitions based on the specified column name. Specify the column name to use.
Bucket - Creates partitions based on a 32-bit hash of the source value. Use the following format, where N is the number of buckets:
```
bucket (N, <column>)
```
Truncate - Creates partitions based on a string or integer value that is truncated to the specified width. Use the following format:
```
truncate(<width>, <string or integer column>)
```
Year - Creates partitions based on the year that is extracted from data in the specified column. Use the following format:
```
year(timestamp, column)
```
Month - Creates partitions based on the month that is extracted from data in the specified column. Use the following format:
```
month(timestamp, column)
```
Day - Creates partitions based on the date that is extracted from data in the specified column. Use the following format:
```
day(timestamp, column)
```
Hour - Creates partitions based on the hour that is extracted from data in the specified column. Use the following format:
```
hour(timestamp, column)
```

For more information about Iceberg partitioning transforms, see the Iceberg documentation.

Write mode

The Write Mode property defines how the Apache Iceberg for AWS destination writes data to Iceberg. Choose between the following options:

Append: Adds new data to the Iceberg table. This write mode does not change existing data.; Use this mode when you want to retain all existing data.
Overwrite: Overwrites the table. This write mode replaces any existing data in the Iceberg table with new data from the pipeline.; Use this mode when you want to replace all existing data.

Schema evolution mode

7.5 and later

You can configure the schema changes that the Apache Iceberg for AWS destination applies to Iceberg tables. To specify how the destination handles schema changes, configure the Schema Evolution property.

The Schema Evolution property provides the following options:

None: The destination makes no schema changes to Iceberg tables. All data written to existing Iceberg tables must have the same schema as the tables.
Additive: The destination adds new columns to Iceberg tables as needed, but does not change existing columns.
Compatible: The destination makes changes to existing columns that are compatible with an earlier version. Compatible changes include adding new columns, compatible data type changes, and making required fields optional.

Supported type promotions

When you use the Compatible schema evolution mode, the Apache Iceberg for AWS destination changes column data types in Iceberg tables under the following conditions:

The target data type has an equal or greater precision and scale, preventing data loss.
The target data type is the top-level type for the column. For example, the destination does not change data types for a column within a map or list column.

The following table provides examples of valid data type changes that the destination can perform when it uses the Compatible schema evolution mode:

Iceberg table data type	Supported type promotion
Integer	Long
Float	Double
Decimal	Decimal with equal or greater precision and scale
Fixed	Binary
Fixed	Fixed with an equal or greater precision

Amazon server-side encryption

You can configure the stage to use Amazon Web Services server-side encryption (SSE) to protect the data that is written to Amazon S3. When configured for server-side encryption, the stage passes required server-side encryption configuration values to Amazon S3. Amazon S3 uses the values to encrypt the data as it is written to Amazon S3.

When you enable server-side encryption for the stage, you select one of the following ways that Amazon S3 manages the encryption keys:

Amazon S3-Managed Encryption Keys (SSE-S3)

When you use server-side encryption with Amazon S3-managed keys, Amazon S3 manages the encryption keys for you.

AWS KMS-Managed Encryption Keys (SSE-KMS)

When you use server-side encryption with AWS Key Management Service (KMS), you specify the Amazon resource name (ARN) of the AWS KMS encryption key that you want to use. You can also specify key-value pairs to use for the encryption context.

Customer-Provided Encryption Keys (SSE-C)

When you use server-side encryption with customer-provided keys, you specify the following information:

Base64 encoded 256-bit encryption key
Base64 encoded 128-bit MD5 digest of the encryption key using RFC 1321

For more information about using server-side encryption to protect data in Amazon S3, see the Amazon S3 documentation.

Processing timestamp data

Apache Iceberg stores timestamps with microsecond precision, but Avro and Parquet files store timestamp data with millisecond precision by default. As a result, the Apache Iceberg for AWS destination does not process timestamp data by default.

The following table describes the differences:

Data types	Timestamp precision	Example
Default Avro and Parquet timestamp data	milliseconds	1705318245123
Iceberg timestamp data	microseconds	1705318245123000

To allow the Apache Iceberg for AWS destination to process timestamp data, you can configure Data Collector to promote Avro and Parquet timestamp precision from milliseconds to microseconds. However, when configured to promote, Data Collector promotes the timestamp precision for all Avro and Parquet files that Data Collector generates.

Important: Before you enable promoting millisecond timestamp data to microseconds, make sure that all downstream systems are compatible with microsecond-precision timestamps.

To enable the Apache Iceberg for AWS destination to process timestamp data, edit the Data Collector configuration file, $SDC_CONF/sdc.properties, and set the avro.iceberg.timestamp.compatibility.enable property to true.

For more information, see Configuring Data Collector in the Data Collector documentation.

Iceberg data types

When the Apache Iceberg for AWS destination writes data to Iceberg, the destination converts Data Collector data types to Iceberg data types.

The following table describes how the destination converts data types:

Data Collector data types	Iceberg data types
Boolean	Boolean
Byte Array (variable length)	Binary
Byte Array (fixed length)	Fixed
Date	Date
Datetime	Timestamp
Decimal	Decimal
Double	Double
Float	Float
Integer	Integer
Long	Long
String	String
Zoned Datetime	Timestamp

Configuring an Apache Iceberg for AWS destination

Configure an Apache Iceberg for AWS destination to write data to an Apache Iceberg table stored on AWS. Before you use the destination, complete the AWS prerequisite tasks.

In the Properties panel, on the General tab configure the general stage properties.
For more information, see the following topics:
On the Iceberg tab, configure the main properties for the destination, including the table name and location.
For more information, see the following topics:
On the S3 tab, configure Amazon information, such as connection and authentication details.
Optionally, on the Region tab, configure the Amazon regional details.
Optionally, on the Assume role tab, configure the destination to assume a role.
For more information, see Assume another role.
Optionally, on the SSE tab, configure the destination to use server-side encryption.
For more information, see Amazon server-side encryption.
Optionally, on the Advanced tab, configure advanced stage properties.
On the Data format tab, select the data format for staged files and related properties.
For more information, see Data formats overview.