IBM Cloud Data Engine connection

To access your data in IBM Cloud Data Engine, create a connection asset for it.

Important: The IBM Cloud Data Engine connector is deprecated and will be discontinued in a future release.

IBM Cloud Data Engine is a service on IBM Cloud that you use to build, manage, and consume data lakes and their table assets in IBM Cloud Object Storage (COS). IBM Cloud Data Engine provides functions to load, prepare, and query big data that is stored in various formats. It also includes a metastore with table definitions. IBM Cloud Data Engine was formerly named "IBM Cloud SQL Query."

Prerequisites

  • An IBM Cloud Data Engine Standard plan instance is required in order to create tables or views.
  • Before you can run SQL queries, you need to have one or more Cloud Object Storage buckets to hold the data to be analyzed and to hold the query results. For instructions, see Provisioning storage.

Create a connection to IBM Cloud Data Engine

To create the connection asset, you need these connection details:

  • The Cloud Resource Name (CRN) of the IBM Cloud Data Engine instance. Go to the IBM Cloud Data Engine service instance in your resources list in your IBM Cloud dashboard and copy the value of the CRN from the deployment details.
  • Target Cloud Object Storage: A default location where IBM Cloud Data Engine stores query results. You can specify any Cloud Object Storage bucket that you have access to. You can also select the default Cloud Object Storage bucket that is created when you open the IBM Cloud Data Engine web console for the first time from IBM Cloud dashboard. See the Target location field in the IBM Cloud Data Engine web console.
  • IBM Cloud API key: An API key for a user or service ID that has access to your IBM Cloud Data Engine and Cloud Object Storage services (for both the Cloud Object Storage data that you want to query and the default target Cloud Object Storage location).

You can create a new API key for your own user:

  1. In the IBM Cloud console, go to Manage > Access (IAM).
  2. In the left navigation, select API keys.
  3. Select Create an IBM Cloud API Key.

Credentials

IBM Cloud Data Engine uses the SSO credentials that are specified as a single API key, which authenticates a user or service ID.
The API key must have the following properties:

  • Manage permission for the IBM Cloud Data Engine instance
  • Read access to all Cloud Object Storage locations that you want to read from
  • Write access to the default Cloud Object Storage target location
  • Write access to the IBM Cloud Data Engine instance

For Credentials, you can use secrets if a vault is configured for the platform and the service supports vaults. For information, see Using secrets from vaults in connections.

Choose the method for creating a connection based on where you are in the platform

In a project
Click Assets > New asset > Connect to a data source. See Adding a connection to a project.
In a deployment space
Click Add to space > Connection. See Adding connections to a deployment space.
In the Platform assets catalog
Click New connection. See Adding platform connections.

Next step: Add data assets from the connection

Where you can use this connection

You can use IBM Cloud Data Engine connections in the following workspaces and tools:

Projects

  • Data Refinery (Watson Studio)
  • SPSS Modeler (SPSS Modeler service)
  • Synthetic Data Generator (Synthetic Data Generator service)
  • Watson Machine Learning Accelerator (Watson Machine Learning Accelerator service)

Catalogs

  • Platform assets catalog

Restrictions

You can only use this connection for source data. You cannot write to data or export data with this connection.

Federal Information Processing Standards (FIPS) compliance

The IBM Cloud Data Engine connection cannot be created in a FIPS environment.

IBM Cloud Data Engine setup

To set up IBM Cloud Data Engine on IBM Cloud Object Storage, see Getting started with IBM Cloud Data Engine.

Supported encryption

By default, all objects that are stored in IBM Cloud Object Storage are encrypted by using randomly generated keys and an all-or-nothing-transform (AONT). For details, see Encrypting your data. Additionally, you can use managed keys to encrypt the SQL query texts and error messages that are stored in the job information. See Encrypting SQL queries with Key Protect.

Running SQL statements

Video to learn how you can get started to run a basic query

Learn more

Parent topic: Supported connections