The database schema version is not consistent with the BPM Product version after installing interim fix and upgrading profile in BPM 8.5.*

Body

 

I just installed the Business Process Manager (BPM) interim fix given and followed the instructions provided in the interim fix read me file for upgrading profiles etc. before starting the servers.
 
Now when I tried to start the deployment manager, it failed with this error:

runConfigActions script execution failed. Exit code: 1 
 
Exception caught while waiting for runConfigActions script to complete: {BPM_ROOT}\profiles\DmgrProfile\bin\runConfigActions.bat"
 

I looked and saw that the SystemOut.log file was present in the logs folder under the dmgr profile and the below errors were what I found in the SystemOut.log:
 
[11/7/16 14:19:16:921 IST] 00000001 ProviderTrack I com.ibm.ffdc.osgi.ProviderTracker AddingService FFDC1007I: FFDC Provider Installed: com.ibm.ffdc.util.provider.FfdcOnDirProvider@c3216c25 
[11/7/16 14:19:16:934 IST] 00000001 Ffdc          I com.ibm.ffdc.util.provider.FfdcOnDirProvider logIncident FFDC1003I: FFDC Incident emitted on {BPM_ROOT}\profiles\DmgrProfile\logs\ffdc\ffdc.245498604919908572.txt com.ibm.bpm.bootstrap.BootstrapHelper.execute 129
[11/7/16 14:19:16:940 IST] 00000001 BootstrapHelp I BootstrapProcessServerData admin command failed due to: The database [jdbc:db2://localhost:50000/BPM8572] schema version [8.5.7] is not consistent with the BPM Product version [8.5.7.201609], ensure the database is updated successfully before starting the server.. Please check the FFDC and wsadmin.traceout for the detail of the error. 
[11/7/16 14:19:16:944 IST] 00000001 Ffdc          I com.ibm.ffdc.util.provider.FfdcOnDirProvider logIncident FFDC1003I: FFDC Incident emitted on {BPM_ROOT}\profiles\DmgrProfile\logs\ffdc\ffdc.5373385286610678469.txt com.ibm.websphere.management.async.client.AsyncCommandClient.execute 187 
[11/7/16 14:19:16:950 IST] 00000001 AbstractShell E   WASX7120E: Diagnostic information from exception with text "java.lang.Exception: java.lang.Exception: The database [jdbc:db2://localhost:50000/BPM8572] schema version [8.5.7] is not consistent with the BPM Product version [8.5.7.201609], ensure the database is updated successfully before starting the server. 
" follows: 
 
 java.lang.Exception: The database [jdbc:db2://localhost:50000/BPM8572] schema version [8.5.7] is not consistent with the BPM Product version [8.5.7.201609], ensure the database is updated successfully before starting the server. 

at com.ibm.bpm.bootstrap.BootstrapProcessServerDataHelper.validateDatabaseSchemaVersion(BootstrapProcessServerDataHelper.java:223)

at com.ibm.bpm.bootstrap.BootstrapProcessServerDataHelper.bootstrapData(BootstrapProcessServerDataHelper.java:334) 

at com.ibm.bpm.bootstrap.BootstrapHelper.execute(BootstrapHelper.java:128) 

at com.ibm.websphere.management.async.client.AsyncCommandClient.execute(AsyncCommandClient.java:177) 

at com.ibm.ws.scripting.adminCommand.AdminCmdController.executeCmd(AdminCmdController.java:1877) 

at com.ibm.ws.scripting.adminCommand.AdminCmdController.processCmd(AdminCmdController.java:318) 

at com.ibm.ws.scripting.adminCommand.AdminCmdProxyHandler.invoke(AdminCmdProxyHandler.java:92) 

at com.ibm.ws.scripting.adminCommand.AdminCmdProxyBase.__method4(AdminCmdProxyBase.java:58) 

at com.ibm.ws.scripting.adminCommand.AdminTask.bootstrapProcessServerData(Unknown Source) 

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95) 

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56) 

at java.lang.reflect.Method.invoke(Method.java:620) 

at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java) 

at org.python.core.PyMethod.__call__(PyMethod.java) 

at org.python.core.PyObject.__call__(PyObject.java) 

at org.python.core.PyInstance.invoke(PyInstance.java) 

at org.python.pycode._pyx50.f$0(<input>:1) 

at org.python.pycode._pyx50.call_function(<input>) 

at org.python.core.PyTableCode.call(PyTableCode.java) 

at org.python.core.PyCode.call(PyCode.java) 

at org.python.core.Py.runCode(Py.java) 

at org.python.core.Py.exec(Py.java) 

at org.python.util.PythonInterpreter.exec(PythonInterpreter.java) 

at com.ibm.bsf.engines.jython.JythonEngine$BSFPythonInterpreter.runcode(Unknown Source) 

at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java) 

at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java) 

at com.ibm.bsf.engines.jython.JythonEngine.iexec(Unknown Source) 

at com.ibm.bsf.BSFManager$7.run(Unknown Source) 

at java.security.AccessController.doPrivileged(AccessController.java:456) 

at com.ibm.bsf.BSFManager.iexec(Unknown Source) 

at com.ibm.ws.scripting.AbstractShell.executeCommands(AbstractShell.java:1121) 

at com.ibm.ws.scripting.AbstractShell.run(AbstractShell.java:2282) 

at com.ibm.ws.scripting.WasxShell.main(WasxShell.java:1172) 

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95) 

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56) 

at java.lang.reflect.Method.invoke(Method.java:620) 

at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:234) 

at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:96) 

at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:77) 

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95) 

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56) 

at java.lang.reflect.Method.invoke(Method.java:620) 

at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587) 

at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198) 

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) 

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) 

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369) 

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179) 

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95) 

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56) 

at java.lang.reflect.Method.invoke(Method.java:620) 

at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340) 

at org.eclipse.core.launcher.Main.basicRun(Main.java:282) 

at org.eclipse.core.launcher.Main.run(Main.java:981) 

at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:402) 

at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:165) 
 
[11/7/16 14:19:16:951 IST] 00000001 AbstractShell A   WASX7093I: Issuing message: "WASX7015E: Exception running command: "AdminTask.bootstrapProcessServerData(['-default', 'true'])"; exception information: java.lang.Exception: java.lang.Exception: The database [jdbc:db2://localhost:50000/BPM8572] schema version [8.5.7] is not consistent with the BPM Product version [8.5.7.201609], ensure the database is updated successfully before starting the server. 

 

This looks to me that some database update scripts also need to be run, but the interim fix read me file "Update the database" section specifically says this:
"It is not necessary to update the database for this interim fix or any of the prerequisites." 
 
Are we missing something while trying to install this fix here?

 

Possible root cause analysis:

Actually besides stopping and starting the server appropriately, there is no need to perform DB updating because when making the interim fix, BPM development has already added in the automated config actions to do the profile upgrade. Users will still need to correctly stop/start their servers though, for the fix to take effect. If the server was not stopped and started correctly, then the changes may not have been federated to all profiles. The act of stopping/starting the server for the purpose of running the DB upgrade command may have, in effect, caused the correct federation to occur.

Based on the error below:

 java.lang.Exception: The database [jdbc:db2://localhost:50000/BPM8572] schema version [8.5.7] is not consistent with the BPM Product version [8.5.7.201609], ensure the database is updated successfully before starting the server.                     at com.ibm.bpm.bootstrap.BootstrapProcessServerDataHelper. 


it is clear that after applying the interim fix and restarting the server, the DBUpgrade command was not run successfully. Users can check BPMConfig_<date>-<timeStamp>.log and DBUpgrade_timestamp.log under  <install_root>/logs/config  directory for this.

 

One possible solution:

This is step 11 of the post install instructions, after you do (DBUpgrade), and then start server, everything works fine. See "Upgrading profiles from IBM Business Process Manager Version 8.5.x to IBM Business Process Manager Version 8.5 Refresh Pack 7" where it states:

Run one of the following commands to upgrade your database.
Notes: If you have multiple deployment environments, complete this step for each deployment environment. You must complete this step for each Advanced, Standard, and Express deployment environment. Skip this step for AdvancedOnly deployment environments. There is no database schema update for AdvancedOnly deployment environments upgraded from V8.5.x.

• On Windows: install_root\bin\DBUpgrade.bat  -profileName deployment_manager_profile [-de deployment_environment_name]
• On Linux or UNIX: install_root/bin/DBUpgrade.sh  -profileName deployment_manager_profile [-de deployment_environment_name]

You can omit the -de parameter if there is only one deployment environment.

Important: Make sure that the DBUpgrade command has finished successfully before you move to the next step. The log is saved in deployment_manager_profile_root/logs/DBUpgrade_timestamp.log. Check the log for errors or exceptions. If running DBUpgrade was unsuccessful, correct the errors and restore your databases from the backup before running the command again.

Tip: If you are using Oracle, you might see an error similar to the following error:
Executing upgrade step: Upgrade 8.5.0 schema to 8.5.5 for database ProcessServerDatabase.
Error executing SQL statement: ORA-60019: Creating initial extent of size 14 in tablespace of extent size 8
On Oracle, IBM Business Process Manager stores large objects (LOBs) with the SECUREFILE option. For SECUREFILE, it is recommended to use a tablespace with the AUTOALLOCATE option. If you use UNIFORM SIZE extents, ensure that the UNIFORM SIZE is big enough. Given the default block size of 8K, specify a UNIFORM SIZE of at least 120K. IBM BPM does not explicitly prescribe the tablespace options; it relies on the default Oracle settings (such as AUTOALLOCATE) to automatically manage extents. If you use tablespaces with a UNIFORM SIZE less than 120K, create new tablespaces with the AUTOALLOCATE option and make it the default tablespace for IBM BPM database schema users.