What's new in watsonx.data

• Faster Migration from Delta Lake to watsonx.data Iceberg : You can now migrate your Delta Lake tables to watsonx.data Iceberg much faster and with significantly optimized resource usage by using a Spark application. During migration, the system preserves your existing data files, table properties, and partitioning, eliminating unnecessary rewrites. For more information, see Submitting Spark jobs to migrate Delta Lake tables to Apache Iceberg.
• You can now configure IBM Cloud Object Storage (COS) or Amazon S3 as the engine home when provisioning a watsonx.data Spark engine on Software Hub. For more information, see Provisioning native Spark engine.

• Enhanced handling for reserved keywords in Salesforce connector: The Salesforce connector automatically handles reserved keywords in column names, improving your experience when querying Salesforce data. Previously, you encountered errors when querying columns with reserved keyword names such as GROUP. With the data-source.identifier-quote=" property now configured by default, the connector seamlessly escapes these keywords, enabling you to query all Salesforce columns without errors or manual setup. If you are on an earlier version prior to 2.3.1, you can manually add this property to your Salesforce connector's catalog.properties file and then restart your pod via the backend. Do not delete your pods. If you face any issues, reach out to support for assistance.
• Enhanced data consistency and configurable locking for Iceberg connector: The Iceberg connector now performs commit operations atomically as a single transaction instead of two separate operations, improving data consistency and reliability. Previously, the connector executed replaceTable and updateTableStatistics as separate operations, which could result in an inconsistent state if one operation succeeded while the other failed. The improved implementation combines these operations into a single atomic transaction for both file-based and Glue-based metastores, ensuring data integrity during commits. Additionally, a new configuration property iceberg.engine.hive.lock-enabled allows you to enable or disable Hive table locks when the Iceberg connector accesses Hive tables. This property can be overridden at the table level using the engine.hive.lock-enabled table property. Disabling locks can improve performance in environments where concurrent access is controlled externally, though it may affect atomicity guarantees. This configuration is relevant only for file-based and Thrift-based metastores, as Glue does not use metastore locks by default.
• OpenShift Data Foundation (ODF) and Fusion Data Foundation (FDF) Support for Custom S3 Storage: watsonx.data now provides seamless connectivity to both OpenShift Data Foundation (ODF) and Fusion Data Foundation (FDF) S3-compatible bucket storage, expanding your options for integrating external object stores.You can connect ODF and FDF buckets as Custom S3 Storage using either HTTP or HTTPS endpoints, enabling flexible, secure integration with your OpenShift or FDF-based environments. For information about configuring Custom S3 Storage connections, see the Custom S3 storage.
• Multi-Region Access Points (MRAP) for global data access: The Amazon S3 storage configuration now includes AWS Multi-Region Access Points (MRAP) for global data access. When adding Amazon S3 storage, select Global access point (MRAP) from the S3 access point topology option. For more information, see Amazon S3.

• OpenTelemetry is now available for the Spark engine, in addition to the existing Presto (Java), Presto (C++) engine and the Milvus service. This expanded capability provides a unified way to collect and export telemetry data (traces and metrics) from all four services, giving you a more complete picture of your system's performance. For more information, see Adding telemetry diagnostic tools through the user interface.
• The Instana and Grafana integration now has new dashboard metrics for Spark and Presto (C++) metrics. For more information, see Supporting dashboards.

Support for Spark 3.4 runtime is deprecated and the default version is changed to Spark 3.5 runtime. To ensure a seamless experience and to leverage the latest features and improvements, switch to Spark 3.5. To update the Apache Spark version, see Editing Spark details.

• Partitioned table support - Prestissimo writes to partitioned Iceberg tables and applies partition transforms efficiently using batch evaluation. It supports identity, temporal transforms (year, month, day, hour), bucket, and truncate, and generates Iceberg-compliant partition directory paths.
• Data file statistics collection - During write operations, Prestissimo collects and reports essential data file statistics to Iceberg manifest files, including record count, file size, and partition details.
• Sorted table write support - Prestissimo supports writing sorted Iceberg tables to enable optimized query performance for workloads that benefit from sorted data.

• New dashboard metrics are available for Presto (C++)

  The Instana integration now includes eight dashboards for Presto (C++) metrics. For more details, see Supporting dashboard metrics for Presto (C++).

This release of watsonx.data introduces the following enhancement to Query Optimizer.

This release of watsonx.data introduces the following enhancements to IBM Cloud Pak for Data Command Line Interface (IBM CPDCTL):

This release of watsonx.data introduces the following ingestion enhancement.

A new toggle is available in the target panel of the ingestion screen to control the delete mode for ingested Iceberg format tables with Copy-on-Write (COW) as default mode. Switching to the Merge-on-Read mode enable row-level deletion during ingestion.

The option to register external Spark engines in watsonx.data is deprecated in this release and will be removed in the near future. watsonx.data already includes native Spark engines that you can provision and use directly, including the Apache Gluten-accelerated Spark engine and the watsonx.data Spark engine.