Integrating Cloudera in watsonx.data

You can integrate Cloudera with watsonx.data to enable zero-copy querying of remote data. Cloudera provides an enterprise data platform that enables organizations to manage, process, and analyze data across hybrid and multi-cloud environments.

By integrating Cloudera with watsonx.data, you can query Hive and Iceberg tables stored in Cloudera HDFS without copying data, enabling seamless data federation across your data landscape.

How it works

Using Presto engine:

  1. Create Hive tables in Cloudera using the Hue editor.
  2. Configure HDFS storage component in watsonx.data.
  3. Associate the catalog with your Presto engine.
  4. Query the remote tables using watsonx.data Presto engine without copying data.

Using Spark engine:

  1. Configure volume mounts and object storage in watsonx.data.
  2. Upload Kerberos and HDFS configuration files to the volume.
  3. Submit PySpark scripts via the REST API to query Hive or Iceberg tables.

Architecture overview

The integration works through the following components:

  1. Cloudera HDFS - Distributed file system storing Hive and Iceberg table data
  2. Hive Metastore - Centralized metadata repository for table definitions
  3. watsonx.data Presto engine - Query engine for SQL-based federated queries
  4. watsonx.data Spark engine - Query engine for PySpark-based queries with Kerberos authentication
  5. HDFS Storage Component - Bridge between watsonx.data and Cloudera

Supported table and storage formats

  • Hive tables - Query Hive tables stored in various formats (Parquet, ORC, Avro, Text) using both Presto and Spark engines
  • Iceberg tables - Query Iceberg tables stored in Cloudera HDFS using the Spark engine
  • Storage formats - Parquet, ORC, Avro, Text files

Key features

  • Zero-copy data access
  • Support for both Kerberos and non-Kerberos authentication
  • Query federation through watsonx.data
  • Integration with watsonx.data Presto and Spark engines
  • Direct access to Hive Metastore
  • Iceberg time-travel and snapshot support via Spark engine

Important limitations

  • Tables are read-only from watsonx.data
  • INSERT, UPDATE, and DELETE operations are not supported when querying tables through watsonx.data
  • Data modifications must be performed directly in Cloudera
  • Spark engine requires Kerberos authentication and volume-mounted configuration files
  • Non-Kerberos authentication is supported with the Presto engine only

Security considerations

Authentication:

  • Non-Kerberos: Suitable for development and testing environments with basic HDFS user authentication
  • Kerberos: Recommended for production environments with enterprise-grade security

Data access:

  • All queries execute with the permissions of the authenticated user or principal
  • HDFS enforces file-level security policies
  • Storage credentials must have appropriate read permissions on HDFS locations

Network security:

  • Ensure network connectivity between watsonx.data and Cloudera cluster
  • Configure firewall rules to allow traffic on required ports (HDFS NameNode, Hive Metastore)
  • For Kerberos, ensure connectivity to KDC (Key Distribution Center)

Next steps

Related information