Looking up data in a Hive data source

You can configure the connector to complete a normal lookup or a sparse lookup on an Hive data source.

In the following figure, a Lookup stage extracts data from a Hive data source based on the input parameter values that the Lookup stage provides. Although the reference link appears to go from the Hive Connector stage to the Lookup stage, the link transfers data both to and from the Hive Connector stage. Input parameters are transferred from the input link on the Lookup stage to the reference link, and output values that the Hive Connector stage provides are transferred from the Hive Connector stage to the Lookup stage. The output values are routed to the columns on the output link of the Lookup stage according to the column mappings that are defined for the Lookup stage.

Figure 1. Example of looking up data on a Hive data source
Example of looking up data on a Hive data source
Note: It is recommended to use the lookup functionality judiciously as the processing of lookup jobs might be slow. This is because the lookup operations might be fetching single record and in order to fetch the record on the Hive data source, a Map Reduce or a Tez job could be running.