Refining data stored in tables in a Hive warehouse in Data Refinery

Prerequisite

Create the connection to the Hadoop cluster. See Hive via Execution Engine for Hadoop connection.

Restrictions

The Data Refinery flow source and target and the Hadoop environment must reference the same Hadoop system.

You must use a Hadoop environment to run Data Refinery jobs on a Hadoop cluster.

Procedure

1. Create a connected data asset for the source (data that you want to refine):

   1. Go to the project page.
   2. Click Assets > Import asset > Connected data.
   3. Click Select source.
   4. Select the Hive via Execution Engine for Hadoop connection. Navigate to the data you want and click Select.
   5. Type a name and description.
   6. Click Create. The asset appears on the project Assets page.

2. Repeat step 1 to create a connected data asset for the target file for the output of the Data Refinery flow.

3. Create a Data Refinery flow:

   1. Click the connected data asset for the source that you created in step 1.
   2. Click Prepare data to open Data Refinery.
   3. Apply operations to refine the data.

4. Change the target location for the output file:

   1. Click the Flow settings icon on the toolbar. Go to the Target data set tab, click Select target.
   2. Click Data asset, and then select the connected data asset for the target output file and click Next.
   3. In the Select target and format properties window, select a Write mode and a Table action.
   4. Click Save and then Apply.

5. Create a job that runs the Data Refinery flow in the Hadoop environment:

   1. From the Data Refinery toolbar, click the Jobs icon , and then select Save and create a job.
   2. Enter a name and description. Select the Hadoop environment.
   3. Optional: Add a one-time or repeating schedule.
   4. Create the job and run it immediately, or create the job and run it later.

Known issues

Troubleshooting Hadoop environments

Learn more

• Adding data from a connection to a project
• Refining data
• Jobs in a project