IBM Cloud Orchestrator, Version 2.5.0.6

High-availability upgrade fails while enabling JDK 1.7

While upgrading a high-availability setup to 2.5.0.6, sometimes high-availability does not come online in the final phases of the upgrade, because Business Process Manager services do not start correctly.

Reason

The root cause is that sometimes Dmgr node fails to stop all the processes correctly. The following error is seen in the installation log file, especially while enabling the JDK 1.7.

ERROR: execute[Java7 stop Dmgr] 
(bpm::bpm_upgrade_enablejdk line 49) had an error: 
Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], 
but received '255'---- Begin output of
/opt/ibm/ico/BPM/v8.5/profiles/DmgrProfile/bin/stopServer.sh dmgr ----
STDOUT: ADMU0116I: Tool information is being logged in file
/opt/ibm/ico/BPM/v8.5/profiles/DmgrProfile/logs/dmgr/stopServer.log
ADMU0128I: Starting tool with the DmgrProfile profile
ADMU3100I: Reading configuration for server: dmgr
ADMU3201I: Server stop request issued. Waiting for stop status.
ADMU3111E: Server stop requested but failed to complete.
ADMU0111E: Program exiting with error:          
   com.ibm.websphere.management.exception.AdminException:
ADMU3060E: Timed out waiting for server shutdown.
ADMU1211I: To obtain a full trace of the failure, use the -trace option.
ADMU0211I: Error details may be seen in the file:          
         /opt/ibm/ico/BPM/v8.5/profiles/DmgrProfile/logs/dmgr/stopServer.log
STDERR:
----End output of/opt/ibm/ico/BPM/v8.5/profiles/DmgrProfile/bin/stopServer.sh dmgr ----
Ran /opt/ibm/ico/BPM/v8.5/profiles/DmgrProfile/bin/stopServer.sh dmgr 
returned 255; ignore_failure is set, continuing

Solution

It is advised to revert to the earlier snapshot of the system and rerun the upgrade after commenting the step to enable JDK 1.7 automatically as part of the upgrade. JDK 1.7 gets installed, but needs to be manually enabled. Steps for both commenting the JDK 1.7 enablement during upgrade, as well as manually enabling it are listed below.

Changing upgrade process to disable JDK 1.7

To remove the step, which automatically enables the JDK 1.7 from the installation procedure, comment out the following line include_recipe 'bpm::bpm_upgrade_enablejdk from the default.rb from the following directory data/orchestrator-chef-repo/chef-repo/cookbooks/bpm/recipes of the installer and then rerun the installer.

Manually enabling the JDK

After the installation is successful, you need to manually enable JDK 1.7 for Business Process Manager.

The following steps must be executed sequentially on the primary and slave nodes.

Note: Sometimes while stopping the Business Process Manager nodes and servers, the Business Process Manager processes are not killed properly. In this scenario, the processes need to be manually killed before restarting the services.

Switch TSAMP to manual mode using the following command:

samctrl -M t

On the primary node

From the Business Process Manager Home/bin directory (by default /opt/ibm/ico/BPM/v8.5/bin), run the following commands:
1. Set Java7 as default for new profiles.
```
managesdk.sh -setNewProfileDefault -sdkName 1.7_64
```
2. Set Java7 as default for scripting operations.
```
managesdk.sh -setCommandDefault -sdkName 1.7_64
```
3. Set Java7 as default for Dmgr profile.
```
managesdk.sh -enableProfile -profileName DmgrProfile -sdkname 1.7_64 -enableServers
```
4. Set Java7 as default for Node1 profile.
```
managesdk.sh -enableProfile -profileName Node1Profile -sdkname 1.7_64 -enableServers
```
From the Node1Profile/bin directory (by default /opt/ibm/ico/BPM/v8.5/profiles/Node1Profile/bin), run the following commands:
1. Stop the node.
```
stopNode.sh
```
2. Sync the node.
```
syncNode.sh #{bpm_server_hostname} 8879
```
  where #{bpm_server_hostname} is the host name of the primary server.
3. Stop the server.
```
stopServer.sh SingleClusterMember1
```
From the DMgr/bin directory (by default /opt/ibm/ico/BPM/v8.5/profiles/DMgrProfile/bin), run the following commands:
1. Stop the Dmgr.
```
stopServer.sh dmgr
```
2. Start the Dmgr.
```
startServer.sh dmgr
```
From the Node1Profile/bin directory (by default /opt/ibm/ico/BPM/v8.5/profiles/Node1Profile/bin), run the following commands:
1. Start the node.
```
startNode.sh
```
2. Start the server.
```
startServer.sh SingleClusterMember1
```

On the secondary node

From the Business Process Manager Home/bin directory (by default /opt/ibm/ico/BPM/v8.5/bin), run the following commands:
1. Set Java7 as default for Node2 profile.
```
managesdk.sh -enableProfile -profileName Node2Profile -sdkname 1.7_64 -enableServers
```
2. Set Java7 as default for new profiles on Node2.
```
managesdk.sh -setNewProfileDefault -sdkName 1.7_64
```
3. Set Java7 as default for scripting operations on Node2.
```
managesdk.sh -setCommandDefault -sdkName 1.7_64
```
From the Node2Profile/bin directory (by default /opt/ibm/ico/BPM/v8.5/profiles/Node2Profile/bin), run the following commands:
1. Stop node on Node2:
```
stopNode.sh
```
2. Sync node on Node2.
```
syncNode.sh #{bpm_server_hostname_HA} 8879
```
  where #{bpm_server_hostname_HA} is the name of the master server.
3. Start node on Node2.
```
startNode.sh
```
4. Stop Server on Node2.
```
stopServer.sh SingleClusterMember2
```
5. Start Server on Node2.
```
startServer.sh SingleClusterMember2
```

After these steps are executed, make sure the Business Process Manager is up and running properly. Change the TSAMP control back to automatic using the following command:

samctrl -M f