Importing upgrade pipelines into IBM Optim Test Data Management

Use the nextgen-upgrade utility to import the generated pipeline JSON files into a running IBM® Optim Test Data Management instance. The utility resolves database connection references, creates the required connection profiles, and registers the pipelines so they are available for use.

Before you begin

Before you import upgrade pipelines, ensure that the following prerequisites are met:

  • You have generated the pipeline JSON files by running the utility in upgrade mode. The files are located under nextgen-upgrade-workspace/upgrade/. For more information, see Generating upgrade pipelines from extracted data.
  • The IBM Optim Test Data Management instance is running and accessible over the network from the machine running the utility.
  • You have a IBM Optim Test Data Management user account with permission to create connection profiles and import pipelines.
  • You have the following details available:
    • IBM Optim Test Data Management server hostname or IP address
    • Server port number (default: 7725)
    • Username and password

Procedure

  1. Run the nextgen-upgrade binary with the --import-json flag and the IBM Optim Test Data Management API connection parameters:
    ./nextgen-upgrade \
        --import-json \
        --api-host <nextgen-hostname> \
        --api-port 7725 \
        --api-user <username> \
        --optimdirpath ./nextgen-upgrade-workspace/upgrade

    The following tables describe the available parameters:

    Table 1. Required import parameters
    Parameter Description
    --import-json Flag that enables import mode. No value is required.
    --api-host Hostname or IP address of the IBM Optim Test Data Management API server.
    --api-user Username for the IBM Optim Test Data Management instance.
    --optimdirpath Directory containing the pipeline JSON files to import.
    Table 2. Optional import parameters
    Parameter Default Description
    --api-port 7725 Port number of the IBM Optim Test Data Management API.
  2. When prompted, enter your IBM Optim Test Data Management login password.
    Enter <username> login password:
    Tip: The password is not echoed to the screen, stored, logged, or written to disk. It is used only to authenticate with the IBM Optim Test Data Management API.
  3. If a pipeline references a database connection that does not yet exist in IBM Optim Test Data Management, respond to the prompt for the database name.
    WARNING: Connection '<DBALIAS>' not found. Creating '<DBALIAS>' connection using dummy_hostname and port <port> for <db-type> type. 
    WARNING: Connection '<DBALIAS>' needs a database name Enter actual database name for '<DBALIAS>':
    • Enter the actual database name and press Enter to use it.
    • If you do not know the database name at this time, press Enter at both prompts. The connection is created with a placeholder database name (dummy_db_<DBALIAS>).
    Note: The connection hostname is always created initially with a placeholder value (dummy_hostname). You must update it in the IBM Optim Test Data Management interface after import to point to the real database server.
  4. Review the import summary that is printed after all pipeline files are processed.
    ============================================================
    Import Summary
    ============================================================
      Total Pipelines:         5
      Pipelines imported:      5
      Pipelines skipped:       0
      Pipelines errored:       0
      Connections created:     2
    ============================================================
    
    Important Notes:
    ------------------------------------------------------------
      [1] Connection 'SOURCEDB' created with dummy hostname. Update connection properties using NextGen UI/API
      [2] Connection 'TARGETDB' created with dummy database name. Update connection properties using NextGen UI/API
    ------------------------------------------------------------
    Note:
    • When a pipeline is imported for the first time, new connection profiles are created in IBM Optim Test Data Management using the database alias name from the pipeline file.
    • When a pipeline references a connection that already exists in IBM Optim Test Data Management, the existing connection is reused and no new connection is created.
    • Update the connection properties for any connections listed under Important Notes with actual values before you run the pipelines.
    • The utility continues processing all pipeline files even if some fail. If the summary shows errors, review the error details that are printed after the summary, resolve each reported issue, and re-run the import.
  5. Update any placeholder connection properties with actual values by using the IBM Optim Test Data Management UI or API.

    For more information about updating connection properties, see Creating a connection.

Results

The pipeline JSON files are imported into IBM Optim Test Data Management and the corresponding flows are created. Update any placeholder connection properties with actual database credentials before you run the flows.

The following table describes common errors and their resolutions:
Table 3. Pipeline import errors and resolutions
Error message Likely cause Resolution
Error: --api-host and --api-user are required with --import-json One or both required flags were not provided. Add --api-host <hostname> and --api-user <username> to the command.
Error: --optimdirpath is required The --optimdirpath flag was not provided. Add --optimdirpath <path> pointing to the directory that contains the generated pipeline JSON files.
no JSON files found in <path> The specified directory contains no .json files. Verify the path is correct and that pipeline generation completed successfully before you run import.
failed to get token: <error> The IBM Optim Test Data Management host or port is incorrect, or the server is not reachable. Verify the --api-host and --api-port values and confirm network access to the server.
token request failed with status 401 Unauthorized Incorrect username or password. Re-enter the correct credentials.
failed to fetch connections: <error> Authentication succeeded but the connections API call failed. Confirm the user account has permission to read connection profiles.
failed to import pipeline: <error> The server could not be reached during pipeline import. Verify network access to the server and retry.
import pipeline failed with status <status>: <error> IBM Optim Test Data Management returned an error response during import (for example, 403 Forbidden or 500 Internal Server Error). Review the status and error details. Confirm the user account has permission to create pipelines.
import completed with <N> error(s) One or more pipeline files failed to process or import. Review the error details section that is printed after the summary, resolve each reported issue, and re-run the import.

What to do next

Validate the imported workflows and run them to confirm successful data processing. For more information, see Validating and running upgraded workflows in IBM Optim Test Data Management.