Migrating Data Virtualization objects by using Git

About this task

Export and import your Data Virtualization assets across different environments from your Git repository through Data Virtualization APIs. By using Git, Admin users can quickly synchronize assets by promoting Data Virtualization objects from a Data Virtualization instance to a Git branch, and by pulling updates from Git back into Data Virtualization. For more information, see Managing asset promotion with Git.

Known issues and limitations

You cannot re-import objects with a different Git registration

After you import your objects, you cannot re-import those objects (or any subsequent objects) to the same target instance with a different Git registration, or this error message appears:

dvapi_gitintegration_runtime.GitIntegrationAPIError: API request failed: 
{'code': 500, 'message': 'Retrieving Git import files from Platform API: failed with exception - java.lang.Exception: Exception occurred during GET request i
n getGitPullFromPlatform. Error: getDVAPIFileResponse call failed with status: 400'}

Grantable privileges might not be preserved after migration

When you export and import data source connections, grantable privileges might not be fully preserved. Privileges that are grantable on the source system might be imported on the target system without the ability to grant them to other users. If grantable access is required after importing connections, then it must be manually re-granted by the connection owner or an authorized administrator.

Before you begin

Pre-requisite steps

Export your Data Virtualization objects into Git

Export your Data Virtualization objects to your Git branch.

1. Prepare your objects for export.
   • Create new objects. For example:

     CREATE SCHEMA "DEMO";
     CREATE TABLE "DEMO"."T_BASE" (ID INT);
     CREATE TABLE "DEMO"."T_MOD"  (ID INT);
     CREATE TABLE "DEMO"."T_GONE" (ID INT);

   • Create a new Db2 nickname (virtualized table) on a connection. Run the VIRTUALIZETABLE stored procedure. See VIRTUALIZETABLE.

     call dvsys.VIRTUALIZETABLE(...);

   • Change and delete objects. For example:

     DROP TABLE "DEMO"."T_GONE";
     ALTER TABLE "DEMO"."T_MOD" ADD COLUMN C2 INT;
     CREATE TABLE "DEMO"."T_NEW" (ID INT);

2. Retrieve a list of all the objects that have changed since the last Git commit including newly created objects, deleted objects, and objects that have changed since the last export. From this list, you can copy any objects you need and paste them into the exportObjects or deleteObjects JSON arrays in the next step.

   Replace these variables in the following command:

   • <INSTANCE_ROUTE>: The hostname of your platform instance.
   • <DV_NAMESPACE>: The namespace where your Data Virtualization instance is deployed.
   • <AUTH_TOKEN>: The platform access token for authenticating with Data Virtualization.

   Request:

   curl -k -X GET \
     'https://<INSTANCE_ROUTE>/icp4data-databases/dv/<DV_NAMESPACE>/dvapiserver/v1/dv/git/export-delta' \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer <AUTH_TOKEN>' \

3. Export your objects to Git.

   Specify these fields in the following export API request:

   • <INSTANCE_ROUTE>: The hostname of your platform instance.
   • <DV_NAMESPACE>: The namespace where your Data Virtualization instance is deployed.
   • <AUTH_TOKEN>: The platform access token for authenticating with Data Virtualization.
   • exportObjects: A list of objects you want to export.
   • deleteobjects: A list of objects you want to delete.
   • <COMMIT_MESSAGE>: An optional commit message.
   • <token-id>: The token-id you obtained from the pre-requisite step.
   • includeDependecies: Export all related objects that the objects depend on.
   • includeStatistics: Run and include statistic statements for exported database objects. These statistics statements are re-created during the import process for its associated object.
     Note: You might experience increased processing times if you enable statistics collection for objects that contain a large number of columns (e.g., hundreds).
   • optionsCSV: Currently, you cannot migrate governance objects. Ensure you set SKIP_GOVERNANCE to true.
   Export API request:

   curl -k -X POST \
     'https://<INSTANCE_ROUTE>/icp4data-databases/dv/<DV_NAMESPACE>/dvapiserver/v1/dv/git/export' \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer <AUTH_TOKEN>' \
     -d '{
       "identifier":"demo",
       "includeDependencies":true,
       "includeStatistics":false,
       "useGit":true, 
       "exportObjects":[{"schema":"DEMO","name":"T_MOD"},{"schema":"DEMO","name":"T_BASE"}],
       "deleteObjects":[{"schema":"DEMO","name":"T_GONE"}],
       "commit_message":"<COMMIT_MESSAGE>",
       "token_id":"<token-id>",
       "optionsCSV": {
         "SKIP_GOVERNANCE": true
       }
     }'

   Sample success response:

   {"config_id":"a227edd77dec476abea626afb34187e5","message":""}

4. Navigate to the Git web client, then create a pull request and merge the changes from one branch into another (e.g., development branch into your QA or production branch).

Optional task: Manage your connection mappings

Control how connection IDs are mapped during import operations in Data Virtualization by running the MANAGE_CID_MAPPING stored procedure. See Managing connection mappings during import.

Import your Git objects into Data Virtualization

Specify these fields in the following import API request:

curl -k -X POST \
  'https://<INSTANCE_ROUTE>/icp4data-databases/dv/<dv namespace>/dvapiserver/v1/dv/git/import' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <AUTH_TOKEN>' \
  -d '{
    "gitZipLoc": "<PATH_TO_DOWNLOAD_FILE (e.g. /tmp/dv_git_import/git-import.zip)>",
    "token_id": "<token-id>",
    "optionsCSV": {
      "SKIP_GOVERNANCE": true
    }
  }'

{"cid_map":[],"connections":[],"exportIdentifier":"git_mode_identifier","objects":[{"grant_status":"pending","import_status":"success","schema":"LOCAL_SKMT","type":"S"},{"grant_status":"success","import_status":"success","schema":"SKMT_SCHEMA","type":"S"},{"grant_status":"success","import_status":"success","name":"SKMT_VT","schema":"SKMT_SCHEMA","type":"N"}],"useGit":true}

Learn more

To troubleshoot Git specific migration errors, see Troubleshooting data migration errors in Data Virtualization.