Overview

Why use IBM Cloud Data Engine?

IBM Cloud Data Engine is IBM Cloud’s central service for data lakes. Combining IBM Cloud Data Engine with data in IBM Cloud Object Storage enables you to create an active workspace for a range of big data analytics use cases.

Features

IBM Cloud Data Engine features

Easy data exploration

IBM Cloud Data Engine uses Apache Spark, an open source, fast, extensible, in-memory data processing engine optimized for low latency and ad hoc analysis of data.

Instant querying

No ETL or schema definition needed to enable SQL queries. Analyze data where it sits in IBM Cloud Object Storage using our query editor and REST API.

Saves time and resources

Run as many queries as you need; with pay-per-query pricing, you pay only for the data scan. Compress or partition data to drive savings and performance.

Highly available and durable

IBM Cloud Data Engine is highly available and executes queries using compute resources across multiple facilities.

Protected

Control access to your data with IBM Identity and Access Management and IBM Key Protect. Grant users granular control of your IBM Cloud Object Storage buckets.

Supports open data formats

IBM Cloud Data Engine supports a variety of data formats such as CSV, JSON and Parquet, and allows for standard ANSI SQL.

How customers use it

Big data log analysis

Diagram showing how applications move through the cloud to a requestor

Big data log analysis

Build and run data pipelines and analytics of your log message data with the full power of SQL. IBM Cloud Object Storage provides seamless scalability and elasticity for cheap and durable storage.

See the tutorial

Building a data lake

Diagram showing how different users interact with the cloud

Building a data lake

Store data in native formats and query instantly. No server configuration or ETL required.

See the tutorial

Data pipelines

Diagram how SQL queries are processed before moving to analysis and stored

Data pipelines

Run highly parallelized data pipelines from IBM Cloud Object Storage to IBM Cloud databases such as IBM Db2® Warehouse or Db2 on Cloud. Connecting your cloud database to your data lake can help you gain more precise control over data quality and lifecycle.

Watch the video (06:33)

Database consolidation

Diagram showing the separation of data, cloud databases and users

Database consolidation

Move on-premises databases to IBM Cloud Object Storage while consolidating license costs and retiring servers. Identify relevant data sets with IBM Cloud Data Engine and push to open-source databases.

Related products

IBM Cloud® Object Storage

A web-scale data storage service that is highly secure, cost-effective and resilient, ensuring continuous access to your data.

IBM Aspera® Transfer Service

Transfer large data sets to and from IBM Cloud Object Storage reliably and at maximum speed. (431 KB)

IBM Cloud® Mass Data Migration

Accelerate the secure movement of large amounts of data into IBM Cloud Object Storage, using rugged, 120 TB-usable capacity portable storage devices.

Next steps

Access the IBM Cloud Data Engine free trial or contact us for pricing details.

Start the free trial
Contact us