Issues with upgrading to Content Engine 5.0 or 5.1 with Image Services federated content

Answer

CE environments with IS federated content may encounter the issues described below when upgrading to CE 5.0 or 5.1.

Description of issues

1. After upgrading to CE 5.0 or 5.1, the system will asynchronously update all federated documents and annotations to make them compatible with the new CE release. However, CFS-IS updates against non-upgraded federated documents, that occur during the async upgrade will be dropped. No error will be logged to indicate that the CFS-IS updates were dropped. This will cause the document metadata on the IS and CE systems to be out of synchronization. For more information, see APAR 39406 (https://www-304.ibm.com/support/entdocview.wss?uid=swg1PJ39406).

For annotation federation, the CE error logs will show the following entry:

com.filenet.api.exception.EngineRuntimeException: FNRCI0079E: IS_PD_ANNOT_DOC_REPL_ERROR: The annotation CFS-IS 111111111 1 11111 1 could not be processed because the document is not replicated.

2. In CE 4.5.1, it is possible to have duplicate Annotations. However, CE 5.0 and 5.1 do not allow duplicate annotations. When upgrading a CE environment which has duplicate annotations, the CE 5.0 and 5.1 Asynchronous Upgrade will encounter an error and the upgrade will stop. For more information, refer to APAR 39399 (https://www-304.ibm.com/support/entdocview.wss?uid=swg1PJ39399). The CE error logs will show the following:

2011-08-23T01:28:31.757Z 6F6F6F6F ENG FNRCD0015E - ERROR ObjectStoreUpgrade(CEOBJ) Asynchronous Replication Task: encountered an exception.
com.filenet.api.exception.EngineRuntimeException: FNRCD0015E: DB_NOT_UNIQUE: The update or insert failed due to an attempt to create a duplicate value in a unique index. Statement List:
SQL: "UPDATE ExternalIdentity SET external_id=?,replication_data=null WHERE parent_id=? AND parent_prop_id=? AND ordinal=?" In-bindings: ('0466690442',{11111111-1111-1111-1111-111111111111},{11111111-1111-1111-1111-111111111111},0)
SQL: "UPDATE DocVersion SET replication_group_id = ? WHERE object_id = ?" In-bindings: ({11111111-1111-1111-1111-111111111111},{11111111-1111-1111-1111-111111111111})
SQL: "INSERT INTO ExternalIdentity (object_id, object_class_id, parent_id, parent_prop_id, ordinal, repository_id, external_id) VALUES (?, ?, ?, ?, ?, ?, ?)" In-bindings: ({11111111-1111-1111-1111-111111111111},{11111111-1111-1111-1111-111111111111},{11111111-1111-1111-1111-111111111111},{11111111-1111-1111-1111-111111111111},0,{11111111-1111-1111-1111-111111111111},'1111111111.0000.0000000000.0000')
SQL: "UPDATE Annotation SET replication_group_id = ? WHERE object_id = ?" In-bindings: ({11111111-1111-1111-1111-111111111111},{11111111-1111-1111-1111-111111111111})
SQL: "INSERT INTO ExternalIdentity (object_id, object_class_id, parent_id, parent_prop_id, ordinal, repository_id, external_id) VALUES (?, ?, ?, ?, ?, ?, ?)" In-bindings: ({11111111-1111-1111-1111-111111111111},{11111111-1111-1111-1111-111111111111},{11111111-1111-1111-1111-111111111111},{11111111-1111-1111-1111-111111111111},0,{11111111-1111-1111-1111-111111111111},'1111111111')
SQL: "UPDATE DocVersion SET replication_group_id = ? WHERE object_id = ?" In-bindings: ({11111111-1111-1111-1111-111111111111},{11111111-1111-1111-1111-111111111111})

[the above sequence may repeat several times]

at com.filenet.engine.dbpersist.DBOracleContext.throwEngineException(DBOracleContext.java:333)
at com.filenet.engine.dbpersist.DBExecutionElement.execute(DBExecutionElement.java:266)
at com.filenet.engine.dbpersist.DBExecutionContext.getNextResult(DBExecutionContext.java:131)
at com.filenet.engine.dbpersist.DBStatementList.getNextResult(DBStatementList.java:600)
at com.filenet.engine.queueitem.ReplicationAsyncUpgradeHandler.run(ReplicationAsyncUpgradeHandler.java:387)
at com.filenet.engine.queueitem.AsyncUpgradeQueueItemHandler.execute(AsyncUpgradeQueueItemHandler.java:132)
at com.filenet.engine.queueitem.QueueItemHandler$1.run(QueueItemHandler.java:37)
at com.filenet.engine.context.CallState.doAs(CallState.java:217)
at com.filenet.engine.context.CallState.doAs(CallState.java:134)
at com.filenet.engine.queueitem.QueueItemHandler.executeAs(QueueItemHandler.java:31)
at com.filenet.engine.queueitem.QueueItemExecutor.loadAndExecuteQueuedRow(QueueItemExecutor.java:173)
at com.filenet.engine.queueitem.QueueExecutor.dispatchQueuedRow(QueueExecutor.java:377)
at com.filenet.engine.queueitem.QueueExecutor.dispatchEvent(QueueExecutor.java:209)
at com.filenet.engine.queueitem.QueueExecutor.execute(QueueExecutor.java:135)
at com.filenet.engine.tasks.BackgroundTask.safeExecute(BackgroundTask.java:217)
at com.filenet.engine.tasks.BackgroundTask$BackgroundTaskPriviledgedExceptionAction.run(BackgroundTask.java:1000)
at java.security.AccessController.doPrivileged(AccessController.java:284)
at javax.security.auth.Subject.doAs(Subject.java:573)
at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:194)
at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:151)
at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at com.filenet.apiimpl.util.J2EEUtilWS.doAs(J2EEUtilWS.java:215)
at com.filenet.engine.context.CallState.doAsSystem(CallState.java:397)
at com.filenet.engine.tasks.BackgroundTask.run(BackgroundTask.java:173)
at com.filenet.engine.jca.workmgr.ThreadPool$WorkerThread.run(ThreadPool.java:97)
Caused by: com.ibm.websphere.ce.cm.DuplicateKeyException: ORA-00001: unique constraint (OBJECTSTORE1.I_EXTERNALIDENTITY76) violated
ORA-06512: at line 1

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:44)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:516)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:605)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:667)
at com.ibm.ws.rsadapter.AdapterUtil.mapException(AdapterUtil.java:2111)
at com.ibm.ws.rsadapter.jdbc.WSJdbcUtil.mapException(WSJdbcUtil.java:1034)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.execute(WSJdbcPreparedStatement.java:646)
at com.filenet.engine.dbpersist.DBExecutionElement.execute(DBExecutionElement.java:218)
... 25 more

For CE 5.0, a Test Fix is available (5.0.0.0-P8CE-ALL-TF009) and for CE 5.1, an Interim Fix is available (5.1.0.0-P8CE-ALL-IF004), to resolve these issues and should be installed prior to upgrading.

As a work-around, the CFS-IS dispatcher can be disabled (with the IS Import Agent tab in FEM) prior to the CE upgrade and kept disabled until the asynchronous part of the CE 5.0/5.1 upgrade has completed. Note that the status of the automatic upgrade can be checked at http://<server>:<port>/FileNet/AutomaticUpgradeStatus

Note: If the system is upgraded to CE 5.1 without the proper fix, and the above errors are encountered, the only way to truly fix the problem is to rollback the database, apply the proper patches, and perform the upgrade again.