Optim High Performance Unload configuration for Big Data or external Db2 destinations

The db2hpu.dest file must be organized by sections, each one corresponding to a given destination. Each section must be organized like the following:

• a section name, specified between brackets, identifying the associated destination. Note that the section name value is case sensitive. The possible values for the section name are:
  • Cloudant: this is the value to be used for a section relative to a NoSQL environment corresponding to a Cloudant database
  • CouchDB: this is the value to be used for a section relative to a NoSQL environment corresponding to a CouchDB database
  • MongoDB: this is the value to be used for a section relative to a NoSQL environment corresponding to a MongoDB database
  • HDFS: this is the value to be used for a section relative to an Hadoop environment corresponding to an HDFS file system
  • Hive: this is the value to be used for a section relative to an Hadoop environment corresponding to an Hive database
  • HBase: this is the value to be used for a section relative to an Hadoop environment corresponding to an HBase database
  • MapReduce: this is the value to be used for a section relative to a customized MapReduce utility for an Hadoop environment
  • RemoteDB2: this is the value to be used for a section relative to a remote Db2 database cataloged locally
  • Warehouse: this is the value to be used for a section relative to a Db2 Warehouse environment
  • AWS_S3: this is the value to be used for a section relative to an Amazon S3 environment
  • Swift: this is the value to be used for a section relative to a Swift environment
• a list of parameters, among which some are mandatory ones, each specified with a pattern of the kind name=value.

url

It is a mandatory one. It allows to specify the url of the NoSQL destination concerned. If necessary, it is also possible to add a port number after the url, with the following pattern: <url>:<port>.

dbname

It is a mandatory one for the Cloudant and MongoDB destinations, for other destinations it is not required. It allows to specify the database name to be used for the JSON documents uploaded.

collectionname

It can only be used for the MongoDB destination, for which it is a mandatory one. It allows to specify the collection name to be used for the JSON documents uploaded.

hdfspath

It can be used along with whatever destination keyword, and it is a mandatory one. It must be used to specify a location on the associated HDFS file system. For the HDFS destination keyword, this location corresponds to the final one where the file containing the data extracted has to be copied. For the other keywords, it corresponds to the temporary location where the file containing the data extracted is copied for a subsequent upload command against this copy. The location specified must exist before executing the copying step, otherwise an error is returned by the HDFS file system when this step is performed. Also this location must correspond to an absolute HDFS path, an HDFS path being compulsorily a UNIX-like one.

pig

It can only be used along with the HBASE destination keyword, and it is an optional one. Its value can be yes or no. By default, if this parameter is not set, the value is no. In order to upload data to an HBase destination, there are two tools which can be used (hadoop or pig). The tool used, when the pig parameter is set to no or when it is not specified, is hadoop, along with the usage of a MapReduce program provided and installed with Optim High Performance Unload, and called “HBaseMapReduce.jar”. Otherwise, when the pig parameter is set to yes, the tool used is pig.

url

It can only be used along with the HIVE destination keyword, and it is a mandatory one in this case. It must be used to indicate a JDBC url corresponding to an Hive database. The url to be specified here must strictly correspond to an already functioning one. The pattern expected for the specification of this url varies, depending if an associated authentication method has been specified or not:

• without authentication:

  jdbc:hive2://<hivehost>:<port>/<db>

• with a Kerberos authentication:

  jdbc:hive2://<hivehost>:<port>/<db>;principal=<Hive_Principal>

In order to get more details about these JDBC url specifications, one can have a look to the Hive documentation.

When a Kerberos authentication is involved, there are two distinct Kerberos principals to be considered and specified:

• the one to be used for the preliminary authentication step before the upload command, which must be specified in the db2hpu.dest configuration file, through its user parameter,
• the one at the end of the JDBC url specification, corresponding to the Hive administrator.

command

It can only be used along with the MAPREDUCE keyword, and it is a mandatory one in this case. It allows to specify a partial command line based on the usage of a MapReduce program of one's own, along with optional command line arguments to be passed to it. This string specification is then used for the generation of the effective upload command.

alias

It is an optional one. Its value is case sensitive. It must be set if an alias is specified into a LOADDEST clause within a control file. In this case, this parameter is the criteria allowing to determine that its associated section is the one to be taken into account. For a migration task towards a NoSQL or a Db2 Warehouse destination, if a standard authentication method is specified into a LOADDEST clause, it must be set too. In this case, its value must match with the name of a section created in the credentials file for the destination type considered.

binpath

It is an optional parameter. When considering a Db2 Warehouse destination, it is useless if migrating data calling the Db2 Load API. It allows to specify a directory which is used when building the upload command, by prepending it to the name of the upload tool, in order to avoid any problem with finding it when trying to execute it for the effective upload.

host

It is a mandatory parameter when migrating data only, otherwise it is useless. When considering a Db2 Warehouse destination, it is useless if migrating data calling the Db2 Load API. It allows to specify the name of the machine on which the upload command to be used for the data migration is going to be generated and executed.

When considering an Amazon S3 destination, the authentication relies on two parameters, which are the access key and the secret key associated to the Amazon account considered. In the case where Optim High Performance Unload is configured to generate an upload command based on the cURL tool, these keys must be set within the db2hpu.dest configuration file through the usage of the ‘accesskey’ and ‘secretkey’ parameters. If the tool to be considered is AWS CLI, these keys must be stored into it before executing any upload command generated with Optim High Performance Unload, by the user aimed to execute the upload command. This step can be performed by executing the ‘aws configure’ command, as follows:

aws configure
AWS Access Key ID [None]: USER_ACCESS_KEY
AWS Secret Access Key [None]: USER_SECRET_KEY
Default region name [None]: BUCKET_REGION_NAME
Default output format [None]:

In order to rely on such an authentication method, the WITH STANDARD AUTH option must be specified into the LOADDEST clause. This standard method must be used when the authentication to the destination considered is based on the usage of a user and its associated password. This is the unique method which can be specified for a Db2 destination, for the Cloudant and CouchDB NoSQL destinations and for the Swift destination. It can be specified too for the MongoDB destination.

user

It allows to specify the user to be considered. This user must have the appropriate permissions to perform the expected data transfer command, and it is taken into account when generating it. The way its associated password is handled depends on the task performed:

• if the task performed is a standard unload one towards a remote Db2 destination, accompanied by a Db2 Load command generation, this parameter is an optional one. If the associated LOADDEST clause contains the specification of its WITH STANDARD AUTH option, it is used into the preliminary connection step generated before the Db2 Load command itself.
• if the task performed is a standard unload one towards a Big Data destination, accompanied by an upload command generation, this command is generated with a preliminary step created for an obfuscated prompting of the password. When executing the command in question, the password must be typed first, and then the effective upload command is executed, taking into account the password typed.
• if the task performed is an automatic migration one, this parameter is a mandatory one. The underlying data transfer command generated must internally contain a reference to a password. This password is read from the credentials file associated to the user executing the automatic migration. It must be stored beforehand into this file, creating a credentials section of the appropriate type: the remote, cloudant, couchdb, mongodb, warehouse and swift types must be respectively considered, depending if the destination chosen is a remote Db2 one, a Cloudant one, a CouchDB one, a MongoDB one, a Db2 Warehouse one or a Swift one. If the destination is a remote Db2 one, the user specified must be the same as the one set to the user parameter into the credentials section considered.
• when considering a Swift destination, there are several rules applying to this parameter. It is an optional one. In most cases, this parameter must be set with a user name. But, if the ‘id’ parameter is set to yes, and the authentication version chosen is 3, this parameter must be set with a user ID. If this parameter is not set, Optim High Performance Unload will try to get the value of an environment variable depending on the authentication version chosen. If the authentication version chosen is 1, the environment variable considered is the “ST_USER” one. If the authentication version chosen is 2, or if the authentication version chosen is 3 and the ‘id’ parameter is not set to yes, the environment variable considered is the “OS_USERNAME” one. If the authentication version chosen is 3 and the ‘id’ parameter is set to yes, the environment variable considered is the “OS_USER_ID” one.
  Note: in a Swift environment, one can set a password associated to a user through an environment variable, the “ST_KEY” one for the authentication version 1, the “OS_PASSWORD” one for the authentication version 2 or 3. But, these environments variables are never taken into account by Optim High Performance Unload when performing an automatic migration. Such a password must be stored into the Optim High Performance Unload credentials file instead.

In order to use such an authentication method, the WITH KERBEROS AUTH option must be specified into the LOADDEST clause. This Kerberos method must be used when the authentication to the destination considered is based on the usage of a Kerberos principal. This authentication method can be specified for the MongoDB destination and is the unique one which can be specified for all the Hadoop destinations.

user

It allows to specify the Kerberos principal to be considered. This Kerberos principal must have the appropriate permissions to perform the expected data transfer command. It is taken into account when generating the data transfer command, by adding a preliminary step for the obtainment of a Kerberos ticket by this principal. This step is based on a kinit command, which is part of the Kerberos client. It must be installed before trying to obtain a ticket. Obtaining such a ticket also relies on the usage of a password. The way this password is handled depends on the task performed:

• If the task performed is a standard unload one accompanied by an data transfer command generation, the associated keytab parameter can be configured too. If it is configured, the preliminary step for the obtainment of the Kerberos ticket is generated so that it directly refers to the keytab file configured. If it is not configured, the preliminary step is generated so that it will prompt for a password when being executed.
• If the task performed is an automatic migration one, it is mandatory to configure an associated keytab parameter.

keytab

It allows to specify an absolute path to a keytab file in which the principal to be taken into account must have its password stored beforehand.