What a phantastic morning, sitting in my arm chair with a cool drink, squinting in the bright sun and dreaming from my next vacation in the middle of nowhere .
Suddenly, the phone rings and bring me back to reality - with real problems.
A more excited customer told me he upgraded his BPM environment from v8.5.6. to 8.5.7.. Now he is complaining about serious pains as his environment cannot be started after the upgrade anymore and he need very quick help.
He knew that he needs to look at the BPMConfig logs and there he found the following errors:
CWUPO0001I: Running configuration action clearClassCache.antrunConfigActions script execution failed. Exit code: 1
BPMConfig E com.ibm.bpm.config.BPMConfig main CWMCB0047E: A connection to node Node1 on host dummyhost.com, SOAP port 8879 could not be
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Unable to find a valid IP for host dummyhost.com]
At a first glance, no valid IP can be found for host dummyhost.com. Pinging the host is working. So he thoughts, it might be a DNS resolution problem.
Hence, he tried including the host into etc/hosts file for DNS resolution but it did not resolve the issue.
Now he is asking me for help.
I knows that on the first start of the servers after the upgrade, important upgrade-related steps like the profile upgrade and the bootstrapProcessServerData command are performed. From the problem description it seems that anything at these actions failed.
I took a deeper look into the BPMConfig logs and found the following errors:
[28-03-2017 19:49:34:355 BST] 00000001 BPMConfig E
com.ibm.bpm.config.BPMConfig main Array index out of range: 0
java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 0
[28-03-2017 19:49:34:802 BST] 00000001 BPMConfig I
com.ibm.bpm.config.BPMConfig main The 'BPMConfig.bat -upgrade -profile C:\IBM\BPM\v8.5\profiles\DmgrProfile' command failed.
That means the DmgrProfile could not be upgraded (according to the stack trace the problem occurs on “setJvmSystemProperty”).
Looking further I discovered also errors like the following:
runConfigActions script execution failed. Exit code: 1
Exception caught while waiting for runConfigActions script to complete: <profile_root> \bin\runConfigActions.bat
Checking the runConfigActions.log of Dmgr, I found the following additional information:
<message>Postinstaller failed. Server will not continue startup</message>
According to that we now know that the runConfigActions as a postinstall step failed for some reason which in turn prevents the server to start.
It seems like both errors are connected. And yes, also the Profile upgrade instructions (see: http://www-01.ibm.com/support/docview.wss?uid=swg27047023 )
informs us about the following:
“When you start the server for the first time after you install the refresh pack, you might see an error message similar to the following message:
runConfigActions script execution failed. Exit code: 1 Exception caught while waiting for runConfigActions script to complete:
This error message indicates that a profile upgrade or bootstrapProcessServerData error occurred. …“
According to the findings I summarizes that overall the profile upgrade of the DmgrProfile was NOT successful preventing the environment to start. It seems that important profile upgrade steps were missing. And we got another indication for that because DBUpgrade logs and bootstrapProcessServerData logs were missing.
My client was very happy for this analysis of the symptoms and mentioned that he potentially missed to perform the DB upgrade steps
(see: http://www-01.ibm.com/support/docview.wss?uid=swg27047023 ).
Although re-running the missing DB upgrade and follow up with the subsequent steps might already resolve the issue, rolling back the environment to a state before the upgrade and running all the upgrade steps on a clean basis is the recommended approach.
The user decided to rollback and performed the upgrade again with all required steps. This time no error occurred, the upgrade was successful and the environment came up as desired.
Problem solved ! Time to dream on of a world without IT problems and relaxed vacation in the middle of nowhere .
And if this doesn't help, take two of these and call me in the morning.
Your Dr. Debug