Generating upgrade pipelines from extracted data

Use the nextgen-upgrade utility to process the extracted JSON files and generate pipeline JSON files that are ready to import into IBM® Optim Test Data Management.

1. Confirm that the extracted data directory from the previous step is available and note its path. You pass this path by using --optimdirpath.
2. Run the nextgen-upgrade utility against the extracted data directory:

   
   ./nextgen-upgrade \
     --optimdirpath <extracted-data-directory>
   

   The following table describes the available parameters:

   Table 1. Processing parameters

   Parameter	Required or optional	Description
   --optimdirpath	Required	Path to the directory that contains the extracted data JSON files from the previous step.
   --platform	Optional	Platform type. Defaults to distributed. Use this parameter only if you are upgrading from a z/OS environment; specify zos.

3. Review the console output to confirm the run completed successfully.

   A successful run produces output similar to:

   Starting nextgen-upgrade 2.0.0
   
   === ReadFiles: Starting file reading process ===
   ReadFiles: PSTOBJ[1] - PSTOBJ2 - {record count}
   ReadFiles: PSTDBA[1] - PSTDBA2 - {record count}
   ReadFiles: PSTREL[1] - PSTREL2 - {record count}
   === ReadFiles: File reading completed successfully ===
   
   === ExtractAndAssemble: Starting object extraction and assembly ===
   ExtractAndAssemble: Grouped into {record count} unique PSTOBJ objects
   ExtractAndAssemble: Grouped into {record count} unique relations
   ExtractAndAssemble: Collected {record count} PSTDBA records
   === ExtractAndAssemble: Extraction completed successfully ===
   
   === ParseAll: Starting parsing ===
   ParseAll: Completed parsing ../.. objects of type {objects}
   ParseAll: Completed parsing ../.. relations of type Relationship
   ParseAll: Completed parsing .. DBAs of type Database Alias
   === ParseAll: Parsing completed successfully ===
   
   === Object Summary Starting ===
   
   Parsed Objects:
     {parsed object}                         {record count}
     ...
   
   Unparsed Objects:
     {unparsed object}                       {record count}
     ...
   
   All processed files written to: ./nextgen-upgrade-workspace/parsed
   === Object Summary completed successfully ===
   
   === Generation: Starting pipeline ===
   
     [Extract Creation Workflow]
       extract_DEMO.EXTRACT1_{timestamp}.json
       (skipped) extract_DEMO.EXTRACT2_{timestamp}
         [issue] unsupported_dbalias: references DBAlias with unsupported target type 'ODBC'
       ...
   
     [Extract Insert Workflow]
       extract_insert_DEMO.INSERT1_{timestamp}.json
       (skipped) extract_insert_DEMO.INSERT2_{timestamp}
         [issue] customer_action_required: source extract file is ambiguous
       extract_insert_DEMO.INSERT3_{timestamp}.json
         [issue] ColumnMap: masking expression is partially supported
   
   Generation: Pipeline output written to: ./nextgen-upgrade-workspace/upgrade
   === Generation: Pipeline execution completed ===

   The following list explains the output indicators:

   • A pipeline file name, printed without a prefix indicates that the pipeline was successfully generated.
   • (skipped) indicates the pipeline was not generated because a required dependency was missing.
   • [issue] indicates something that requires review. An issue can appear on a skipped pipeline (explaining why it was skipped) or on a generated pipeline where part of the definition needs manual attention, such as a partially supported data-masking configuration.
4. Verify that the output directory contains the generated pipeline files.

   nextgen-upgrade-workspace/
   ├── logs/
   │   ├── nextgen-upgrade_<YYYYMMDD_HHmmss>.log
   │   └── data_collection.json
   ├── parsed/
   │   ├── PSTOBJ_<object>.json
   │   ├── PSTDBA.json
   │   └── PSTREL.json
   └── upgrade/
       ├── extract_<name>_<YYYYMMDD_HHmmss>.json
       └── extract_insert_<name>_<YYYYMMDD_HHmmss>.json

   Troubleshooting

   The following table describes common errors and their resolutions:

   Table 2. Pipeline generation errors and resolutions

   Error message	Likely cause	Resolution
   Error: --optimdirpath is required	The extracted data path was not provided.	Add --optimdirpath <path> to the command.
   failed to read files: no PSTOBJ*.json files found in directory: <path>	The extracted data directory does not contain the expected JSON files.	Confirm that the data extraction step completed successfully and verify the path.
   failed to read files: directory does not exist: <path>	The path supplied to --optimdirpath does not exist.	Check the path for typographical errors. Confirm that the data extraction step completed and wrote output to that directory.
   failed to extract objects: <detail>	One or more extracted JSON files are malformed or contain unexpected content.	Rerun the data extraction step. If the error persists, review the extracted JSON files for corruption.
   failed to generate pipelines: <detail>	An internal error occurred while building a JSON pipeline from the extracted data.	Review the log output for details. Contact support if the error persists.
   (skipped) appears in the output for a pipeline	A required referenced object or dependency was not available in the extracted data.	Review the [issue] line below the skipped pipeline name and correct the source data before you rerun.
   [issue] appears for a generated pipeline	The pipeline was created, but part of the definition needs manual review - for example, a partially supported data-masking configuration.	Import the pipeline and review the affected configuration in IBM Optim Test Data Management.
   The upgrade/ directory is empty after the run	All candidate pipelines were skipped.	Review the console output for [issue] details under each skipped pipeline name.
   Error: --platform must be 'distributed' or 'zos'	An unsupported value was passed to --platform.	Use --platform distributed.