IBM Support

Using online Migration to migrate CCM server from 4.x to 5.x can result in corruption of streams and workspaces

Troubleshooting


Problem

Using online migration to migrate a 4.x Change and Configuration Management (CCM) server to a 5.x release may result in corruption of streams and workspaces.

Symptom

  • Online migration logs may contain exception stack traces indicating transient SQL exceptions occurred during migration.

    The addTables migration log may indicate that less than 100% of items were processed:


    2015-02-28 00:44:19,959 CRJAZ2693I Running online migration on 389,576 item states. 385,141 previously processed. 2 new type handlers.
    2015-02-28 00:44:19,960 Type "ChangeHistory", 150,816 of 153,434 item status previously processed.
    2015-02-28 00:44:19,960 Type "HistoricBasis", 234,325 of 236,142 item status previously processed.
    2015-02-28 00:44:27,873 Status: 385,192 of 389,576 processed (98% complete). Estimated completion time: 2/28/15 12:45 AM.
    2015-02-28 00:44:30,988 CRJAZ2691I No more items to process.
    2015-02-28 00:44:31,001 CRJAZ2739I No state migration failures occurred during online migration.


  • The migrated server may have active FilesystemService.compareWorkspaces service calls that are hung and running for hours or days.

  • Workspaces or Streams may contain gaps in recent history. Users may receive unexpected gap messages when trying to accept or deliver.

  • When attempting to accept or deliver changes users may receive error messages similar to

    Internal problem: change sets for /component/example/file.txt have null common ancestor.

    or

    Internal problem: change sets yielded null common ancestor.

Cause

Retryable database exceptions are not being retried and the migration does not report an appropriate error condition.


Online migration records the last migrated item state that was processed. When migration resumes, it continues processing items starting after the last migrated state. After online migration is run, if the particular item that was the last migrated state is modified, a subsequent online or offline migration will be unable to find the proper state to resume migration. This will result in new states not being migrated.

Unmigrated states can lead to corruption in the 5.x server.


These issues have been reported in the following defects:

APAR PI37709. Further details are available in defect work item 348803

APAR PI37026. Further details are available in defect work item 348914

APAR PI37708. Further details are available in defect work item 347359

Diagnosing The Problem

Online migration should only be run once, then when the server is brought down for offline migration, before running the upgrade, you should perform the following queries to determine if you are affected by this problem:


DB2, SQLServer: "select VALUE_COL from JAZZ.PROPERTIES where KEY_COL='NewTypeMigrationService.status'"
Oracle: "select VALUE_COL from CCMDBUSER.JAZZ_PROPERTIES where KEY_COL='NewTypeMigrationService.status'"


This should produce output similar to:


{"startTime":1426770690267,"totalProcessed":1918,"totalStates":1918,"modifiedOnLastStateProcessed":"2015-03-18T13:55:40.856-04:00","idOnLastStateProcessed":"_X2uBoc2gEeS9N4caXGPgAA","processedTime":357302,"lastUpdated":"2015-03-19T14:02:34.105-04:00","estimatedOfflineTimeToCompletion":0,"estimatedOnlineTimeToCompletion":0}



Determine the idOnLastStateProcessed and modifiedOnLastStateProcessed values.  In the example above they are idOnLastStateProcessed="_X2uBoc2gEeS9N4caXGPgAA" and modifiedOnLastStateProcessed="2015-03-18T13:55:40.856-04:00".


Run the following query, substituting the appropriate idOnLastStateProcessed value:




DB2, SQLServer: "select KEY_UUID, MODIFIED from REPOSITORY.ITEM_STATES where KEY_UUID='_X2uBoc2gEeS9N4caXGPgAA'"
ORACLE: "select KEY_UUID, MODIFIED from CCMDBUSER.REPOSITORY_ITEM_STATES where KEY_UUID='_X2uBoc2gEeS9N4caXGPgAA'"



This query is expected to return 1 result and the MODIFIED time from the query is expected to match the modifiedOnLastStateProcessed value. 


If no results are found, or the modified time does not match, then the database will be corrupted
.  The upgrade should not continue and the Online migration should be reverted. Refer to Repository tools command to revert an online migration for information on reverting the Online migration. To verify that the online migration revert was successful, rerun the first query from the diagnosis which should now return no results.

The time format in the migration status may be slightly different than the format used in the DB query results.  It is only important that the different formats represent the same time.  On the SQLServer database the DB query result may be a number which is the number of milliseconds since January 1, 1970, 00:00:00 GMT. 

Resolving The Problem


Important notes:

  • When upgrading to 5.0.x, the latest iFix must be applied to the 5.0.x install before performing online migration.
  • Whether or not your server shows the symptoms listed in the "Symptoms" section, if you have used online migration before applying the latest iFix, DO NOT upgrade the server without first following the instructions listed in the "Diagnosis" section below.

Corrective action



If an online migration and subsequent offline migration has completed and users are experiencing symptoms (as noted above), take the following actions:

    1. Restore the Jazz environment and its associated databases to the status prior to performing the offline migration.
    2. Revert the online migration by following the instructions in Repository tools command to revert an online migration.
    3. Either perform an offline migration or apply the latest iFix to the 5.0.x install before re-attempting an online migration followed by an offline migration.
Leverage the Jazz Community

The Jazz based products have an active community that can provide you with additional resources. Browse and contribute to the User forums, contribute to the Team Blog and review the Team wiki.
Refer to technote 1319600 for details and links.

[{"Product":{"code":"SSUC3U","label":"IBM Engineering Workflow Management"},"Business Unit":{"code":"BU055","label":"Cognitive Applications"},"Component":"Team Server","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"},{"code":"PF035","label":"z\/OS"}],"Version":"5.0.2","Edition":"","Line of Business":{"code":"LOB02","label":"AI Applications"}}]

Product Synonym

Rational Team Concert

Document Information

Modified date:
29 September 2018

UID

swg21699466