Select the Standard tab for upgrade instructions using the iFix installation script.

Select the Manual tab for the instructions on how to manually upgrade each component.

Important Notice: Ensure your environment has met all the requirements in the Prerequisites section above before applying this iFix.

1) Stop SmartCloud Orchestrator on Central Server 1
Change to directory: /iaas/scorchestrator/
Stop SmartCloud Orchestrator: ./SCOrchestrator.py --stop
===>>> Stopping Smart Cloud Orchestrator
...
===>>> Stopping Smart Cloud Orchestrator complete

Wait until all components have stopped

2) Backup your SCO installation
Backup the Central Servers and Region Server virtual machines:
for VMware hosted virtual machines, see Taking Snaphots for information about taking snapshots of the virtual machines.

3) Copy 2.3.0.1-CSI-ISCO-IF0004.tar on Central Server 1

4) Extract 2.3.0.1-CSI-ISCO-IF0004.tar in a directory of your choice <your_dir> by running the following command:
Note: Do not place the pakcage under /root path

tar -xf 2.3.0.1-CSI-ISCO-IF0004.tar

The following files are extracted:

   3.1.0.4-IBM-SCE-FP004-201407010432.zip
   3.1.0.4-IBM-SCE-IF001-201407170051.zip
   helper.py
   ibm-java-i386-sdk-6.0-16.0.i386.rpm
   ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm
   IBM_SmartCloud_Provisioning_Core_2.3.0.1_IF0004-2.3.0.1.fxtag
   ICCT_Install_2.3.0.1-17.zip
   IFIX0004_IWD_SCO2301_20140711-0842-753.tar
   ifix4.py
   il_install_package_23025.zip
   il_proxy_install_package_23025.zip  
   openstack.tar
   sce310_jre_linux_installer.bin
   SmartCloud_Orchestrator_2.3.0.1_IF0004.fxtag
   2.3.0.1-CSI-ISCO-IF0003
     -helper.py
     -ifix3.py
     -SmartCloud_Orchestrator_2.3.0.1_IF0003.fxtag
     -SE59389
       -se59389_fix.py
       -se59389_fix.tar.gz
     -scp-pdcollect
       -Components_nonroot.xml
       -Components.xml
       -Environment.xml
       -PDCOLLECT_Central_Server_Template.xml
     -ZZ00201
       -iwd0002_0006checksum.txt
       -iwd0002_0006.py
       -opt/ibm/.../maestro.util.jar
     -ZZ00234
       -automation.groovy
     -ZZ00240
       -00_SCOrchestrator_Toolkit_2301_20140403_ifix1.twx
     -ZZ00242
       -config_network.sh
     -ZZ00244_ZZ00246
       -plugin.com.ibm.orchestrator.rest-1.0.1.1.jar
   install_script
     -create_new_repo.py
     -helper.py
     -iwd_fix.py
     -java632sr16fp1.py
     -java664sr16fp1.py
     -sce310fp4.py
     -sce310fp4fix.py
     -sce310java664sr16fp1.py
     -update_pack.py
     -vilp23025.py
     -vils23025.py
     -was80javasdk32-ifpi19109.py
     -was85javasdk64-ifpi19108.py

5) Upgrading the Image Construction and Composition Tool (ICCT)

- Review the Upgrading the Image Construction and Composition Tool using the standard upgrade procedure or using the silent upgrade procedure.

- In step "Download and extract the compressed file to the computer where you want to upgrade or install
the Image Construction and Composition Tool fix pack" use ICCT_Install_2.3.0.1-17.zip

6) Prepare the 32-bit Java installation used by the VIL on Central Server 2

- In Security Bulletin: Multiple vulnerabilities in current IBM SDK for Java for WebSphere Application Server April 2014 CPU click on link named "PI19109" in section "For 8.0.0.0 through 8.0.0.8:"

- In PI19109: SHIP SDK 626 SR8 FOR WSAS V8.0.0.X you need to download the correct package (Linux 32-bit x86 AMD/Intel Java SDK) by clicking the FC link which lets you download the following package:

8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 (94943549 Bytes)

- When downloaded, copy 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 to the Central Server 1,
in the ifix directory <your_dir>.

7) Prepare the 64-bit Java installation used by the Business Process Manager on Central Server 4

- In Security Bulletin: Multiple vulnerabilities in current IBM SDK for Java for WebSphere Application Server April 2014 CPU click on link named "PI19108" in section "For V8.5.0.0 through 8.5.5.2 Full Profile:"

- In PI19108: SHIP SDK 626 SR8 FOR WSAS V8.5.0.X AND V8.5.5.X you need to download the correct package (Linux 64-bit x86 AMD/Intel Java SDK) by clicking the FC link which lets you download the following package:

8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 (103391758 Bytes)

- When downloaded, copy 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 to the Central Server 1,
in the ifix directory <your_dir>.

8) Run the iFix installation script on Central Server 1

- From the directory where you unpacked the iFix package, run the iFix installation script by using the following command:

 ./ifix4.py --cs2=<cs2> --cs3=<cs3> --cs4=<cs4> --rs=<regionservers> --cn=<compute_node> --wasa=wasadmin --wasp=passw0rd

Example: ./ifix4.py --cs2=10.10.0.12 --cs3=10.10.0.13 --cs4=10.10.0.14 --rs=10.10.0.15,10.10.0.16 --cn=10.10.0.17 --wasa=wasadmin --wasp=passw0rd

You can use ./ifix4.py -h to show the usage.

Options:
    -h, --help          show this help message and exit
    --cs2=CS2           hostname/ip address of central server 2
    --cs3=CS3           hostname/ip address of central server 3
    --cs4=CS4           hostname/ip address of central server 4
    --rs=RS             list of hostnames/ip addresses of region servers format
                        server1,server2,server3,...
    --cn=CN             (Optional)list of hostnames/ip addresses of compute node format
                        compute1,compute2,compute3,...
    --wasa=WASADMIN     virtual image library WAS administrator ID to be used
                        during the installation procedure
    --wasp=WASPASSWORD  virtual image library WAS administrator password

You can refer to log file ifix4.log in the script path for more detail information of the installation process.

Other log information:
VIL install/upgrade log: /tmp/fresh_install_vil.log, upgrade_vil.log on Central Server 2
VIL Proxy install/upgrade log: /tmp/fresh_install_proxy.log, upgrade_proxy.log on Region Server
IWD fix install log: /tmp/iwdifix4.log Central Server 3

9) Apply patches to OpenStack Components
Note: Use ./SCOrchestrator.py to stop all the services before applying the changes.

(1) use below command to find python directory:

$ python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

For example:
$ python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/usr/lib/python2.6/site-packages

From the command, '/usr/lib/python2.6/site-packages' is #{your_python_dir} which will be used in following steps.

(2) apply keystone patch on CS-2
login to CS-2 and carry out below commands:
$ scp root@$cs-1_ip:/data/repos/scp/patch/keystone.patch <your_directory>
$ cd #{your_python_dir}
$ patch -p1 -N -f < <your_directory>/keystone.patch

(3) apply OpenStack patches on each Region Server
login to Region Server and carry out below commands:

- apply nova patch
$ cd #{your_python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch
$ cd /usr
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch

Note: For this part, ignore the error message reporting "can't find file to patch" for below files:

nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.json.tpl
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.xml.tpl
nova/tests/integrated/test_api_samples.py
nova/tests/test_configdrive2.py
nova/tests/test_nova_manage.py
doc/api_samples/all_extensions/extensions-get-resp.json
doc/api_samples/all_extensions/extensions-get-resp.xml

- apply glance patch
$ cd #{python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/glance.patch

- apply cinder patch
$ cd #{python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/cinder.patch

(4) apply nova patch on each KVM compute node
login to KVM compute node and carry out below commands:
$ scp root@$rs_ip:/data/repos/scp/patch/nova.patch <your_directory>
$ cd #{your_python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch
$ cd /usr
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch

Note: For this part, ignore the error message reporting "can't find file to patch" for below files:
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.json.tpl
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.xml.tpl
nova/tests/integrated/test_api_samples.py
nova/tests/test_configdrive2.py
nova/tests/test_nova_manage.py
doc/api_samples/all_extensions/extensions-get-resp.json
doc/api_samples/all_extensions/extensions-get-resp.xml

10) Start SmartCloud Orchestrator on Central Server 1

- Stop SmartCloud Orchestrator on Central Server 1 (to enable a clean start in the following step)
Change to directory: /iaas/scorchestrator/
Stop SmartCloud Orchestrator: ./SCOrchestrator.py --stop
===>>> Stopping Smart Cloud Orchestrator
...
===>>> Stopping Smart Cloud Orchestrator complete

Wait until all components have stopped

- Start SmartCloud Orchestrator on Central Server 1
Change to directory: /iaas/scorchestrator/
Start SmartCloud Orchestrator: ./SCOrchestrator.py --start

Wait until all components have started

11) Additional manual steps required to install fix for APAR ZZ00240.
Note: This fix is also included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). If you have alreay applied 2.3.0.1-CSI-ISCO-IF0003 to your SCO environment, ignore this part.

Step (1). Copy the folder 2.3.0.1-CSI-ISCO-IF0003/ZZ00240 to a directory of your choice where the BPM brower is being run from.

Step (2). Import toolkit SCOrchestrator_Toolkit provided by this fix

Logon to IBM Process Designer with admin/$your_smartcloud_passowrd

In Process Center, select tab Toolkits and click on toolkit SCOrchestrator_Toolkit

Check that SCOrchestrator_Toolkit version 2301_20140213 is the current version
(top in the stack below "Current")

Click on Toolkits and then on Import Toolkit

In the Import Toolkit dialog, click on Browse... and select 00_SCOrchestrator_Toolkit_2301_20140403_ifix1.twx
provided with this fix and click OK

Click the Import button in dialog Import Toolkit - Will be imported - SCOrchestrator_Toolkit - 2301-20140403_ifix1 and wait for the import to be finished

Click on toolkit SCOrchestrator_Toolkit

Check that SCOrchestrator_Toolkit version 2301_20140403_ifix1 is the current version
(top in the stack below "Current")


Step (3). Upgrade dependency of toolkit SCOrchestrator_Scripting_Utilities_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Scripting_Utilities_Toolkit

Check that SCOrchestrator_Scripting_Utilities_Toolkit version 2301_20140226 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1


Step (4). Create new snapshot of toolkit SCOrchestrator_Scripting_Utilities_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Scripting_Utilities_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that SCOrchestrator_Scripting_Utilities_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (5). Upgrade dependency of toolkit SCOrchestrator_Support_IaaS_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_IaaS_Toolkit

Check that SCOrchestrator_Support_IaaS_Toolkit version 2301_20140227 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1


Step (6). Create new snapshot of toolkit SCOrchestrator_Support_IaaS_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_IaaS_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that SCOrchestrator_Support_IaaS_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (7). Upgrade dependency of toolkit SCOrchestrator_Support_vSys_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_vSys_Toolkit

Check that SCOrchestrator_Support_vSys_Toolkit version 2301_20140310 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_IaaS_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (8). Create new snapshot of toolkit SCOrchestrator_Support_vSys_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_vSys_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that SCOrchestrator_Support_vSys_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (9). Upgrade dependencies of toolkit TivSAM_Integration_Toolkit

In Process Center, click on Toolkits and click on TivSAM_Integration_Toolkit

Check that TivSAM_Integration_Toolkit version 2301_20140305 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Scripting_Utilities_Toolkit (2301_20140226) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (10). Create new snapshot of toolkit TivSAM_Integration_Toolkit

In Process Center, click on Toolkits and click on TivSAM_Integration_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that TivSAM_Integration_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (11). Upgrade dependencies of process app Sample_Support_Iaas_ProcessApp

In Process Center, click on Process Apps and click on Sample_Support_Iaas_ProcessApp

Check that Sample_Support_Iaas_ProcessApp version 2301_20140305 is the current version
(top in the stack below "Current")

Click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_IaaS_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (12). Upgrade dependencies of process app Sample_Support_vSys_ProcessApp

In Process Center, click on Process Apps and click on Sample_Support_vSys_ProcessApp

Check that Sample_Support_vSys_ProcessApp version 2301_20140310 is the current version
(top in the stack below "Current")

Click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_IaaS_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_vSys_Toolkit (2301_20140310) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (13). Upgrade dependencies of process app TSAMITK_SampleApp

In Process Center, click on Process Apps and click on TSAMITK_SampleApp

Check that TSAMITK_SampleApp version 2301_20140305 is the current version
(top in the stack below "Current")

Click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Scripting_Utilities_Toolkit (2301_20140226) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Integration_Toolkit (2301_20140305) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001

Please repeat the above described procedure for every toolkit or process application in your process centre.

Step (14). Upgrade REST endpoint in database

Logon to Central-Server-1

su - db2inst1
db2 connect to BPMDB
db2 set schema bpmuser
db2 "select guid,substr(default_value,1,29) as default_value,last_modified from lsw_env_var where name='restEndpoint'"

You should see one entry where default value is equals to https://localhost

Update this entry:
db2 "update LSW_ENV_VAR set default_value='https://<Central-Server-3-IP>' where name = 'restEndpoint' and default_value='https://localhost'&quot;

Double-check that the update was successful:

db2 "select guid,substr(default_value,1,29) as default_value,last_modified from lsw_env_var where name='restEndpoint'"

db2 commit
db2 connect reset

Stop/Start the BPM admin server on Central-Server-4
/opt/ibm/BPM/v8.5/profiles/Node1Profile/bin/stopServer.sh SingleClusterMember1
/opt/ibm/BPM/v8.5/profiles/Node1Profile/bin/startServer.sh SingleClusterMember1

12) Apply fix for APAR ZZ00256.
Note: This fix is also included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). If you have alreay applied 2.3.0.1-CSI-ISCO-IF0003 to your SCO environment, ignore this part.

If your SCO server OS locale is non-English, please perform the following steps to return the correct status of openstack services to SAAM.

On each of the SCO servers (Central-servers, region-servers and KVM compute nodes), find the file 'openstack-servicectrl.sh' in '/home/saam/' or '/home/ <yourmechid>/root/'.

(1) make a backup of openstack-servicectrl.sh
(2) edit 'openstack-servicectrl.sh',

Modify the line
CTRL_CMD="/sbin/service $SERVICE_ID"

to
CTRL_CMD="/etc/init.d/${SERVICE_ID}"
LANG=C

(3) check the output of openstack-servicectrl.sh and ensure it can get the correct status of openstack services.

For example, on central-server 2 run:

$ LANG=C /etc/init.d/openstack-keystone status
keystone (pid 2817) is running...

$ ./openstack-servicectrl.sh openstack-keystone status
root: openstack-servicectrl.sh (28621): openstack-keystone status monitor detected openstack-keystone online.

13) Apply fix for APAR SE58688 - KVM nodes going offline
Note: This fix is also included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). If you have alreay applied 2.3.0.1-CSI-ISCO-IF0003 to your SCO environment, ignore this part.

KVM compute nodes are going offline and can't deply any new patterns and VMs.
This is a bug of openstack https://bugs.launchpad.net/oslo-incubator/+bug/1211338.

On each kvm region server and compute node, backup the file /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py
Then change it as bellow.
find the words:
{"type": "Direct"}
modify it to:
{"type": "direct"}

14) Apply fix for ZZ00242 on Central Server 1.
Note: This fix is also included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). If you have alreay applied 2.3.0.1-CSI-ISCO-IF0003 to your SCO environment, ignore this part.

(1) backup file config_network.sh in $your_sco_install_media/installer/scripts/

(2) Update config_network.sh with the one in folder 2.3.0.1-CSI-ISCO-IF0003/ZZ00242

15) Apply fix for the DB2 PSIRT issue.
Note: This fix is also included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). If you have alreay applied 2.3.0.1-CSI-ISCO-IF0003 to your SCO environment, ignore this part.

(1) downloading the DB2 fix pack

Address the security vulnerabilities outlined in Security Bulletin: Security vulnerabilities have been identified in IBM DB2 shipped with SmartCloud Orchestrator (CVE-2013-6747, CVE-2014-0963)

Open the security bulletin IBM DB2 is impacted by multiple TLS/SSL security vulnerabilities (CVE-2013-6747, CVE-2014-0963) and download the fix for you release of IBM DB2

(2) Installing the DB2 fix pack on Central Server 1 and each Region Servers (if shared DB is not used)

Ensure the SCO processes have been stopped using SCOrchestrator.py stop as outlined in section III step 1 above
Note: The output of the command /iaas/scorchestrator/SCOrchestrator.py will list the servers that are running DB2. Update each of these servers, starting with central-server-1 and then moving on to each region server.

Perform following steps to install the DB2 fix.
<1> Copy the fix pack to central server 1 to a folder of your choice <db2fixpack>
<2> As user db2inst1 stop DB2 and DAS:
db2stop
/opt/ibm/db2/v10.1/das/bin/db2admin stop
<3> As user root extract and install the DB2 fix pack
cd <db2fixpack>
tar zxvf v10.1fp3a_linuxx64_server.tar.gz
cd server
./installFixPack -n -b /opt/ibm/db2/v10.1 -f db2lib
Please wait for the install to complete.

<4> Re-run the install command to verify the fix pack has been applied
./installFixPack -n -b /opt/ibm/db2/v10.1/ -f db2lib

Need to repeat the steps above on each Region Server which uses standalone DB2.

*********************************************************************************************
WARNING: If you ran the script ifix6.py as outlined on the Standard tab and it completed successfully, then most of the steps in this section can be skipped. You will need to refer to the instructions on the Standard tab for additional manual steps which still need to be executed after running the script ifix4.py.

Note: Be sure to execute all the steps described in this section to ensure that any of the fixes in the pile (i.e. iFix3 and iFix4) are included.
*********************************************************************************************

1) Inventory Tagging

To make it easier for support to find out which SmartCloud Orchestrator fixes are installed on your system, copy some files to /opt/ibm/SmartCloud_Orchestrator/properties/version/ on Central-Server-3.
(if not present on Central-Server-3 create the directory)

scp *.fxtag \
Central-Server-3:/opt/ibm/SmartCloud_Orchestrator/properties/version/

2) Manual steps to apply IWD fixes on Central Server 3.
Note: This fix is also included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). If you have already applied 2.3.0.1-CSI-ISCO-IF0003 to your SCO environment, ignore this part.

(1) Copy the following files from Cetral Server 1 to $your_folder on Central Server 3.

2.3.0.1-CSI-ISCO-IF0003/SmartCloud_Orchestrator_2.3.0.1_IF0003.fxtag
2.3.0.1-CSI-ISCO-IF0003/ZZ00234/automation.groovy
2.3.0.1-CSI-ISCO-IF0003/ZZ00244_ZZ00246/plugin.com.ibm.orchestrator.rest-1.0.1.1.jar
2.3.0.1-CSI-ISCO-IF0003/ZZ00201/opt/ibm/rainmaker/purescale.app/private/expanded/ibm/maestro.util-4.0.0.1/lib/maestro.util.jar

(2) Stop IWD server on Central Server 3:
service iwd stop

(3) Backup the following files on Central Server 3:

mkdir -p /tmp/$your_backup
cd /tmp/$your_backup
cp --preserve /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/maestro.util-4.0.0.1/lib/maestro.util.jar maestro.util.jar_ori
cp --preserve /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/orchestrator.ui-4.0.0.1/app/resources/automation.groovy automation.groovy_ori
cp --preserve /drouter/ramdisk2/mnt/raid-volume/raid0/usr/servers/kernelservices/plugins/bundles/com.ibm.orchestrator.task/1.0.1.0 plugin.com.ibm.orchestrator.rest-1.0.1.1.jar plugin.com.ibm.orchestrator.rest-1.0.1.1.jar_ori

(4) Update the files on Central Server 3

cd $your_folder
cp -f maestro.util.jar /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/maestro.util-4.0.0.1/lib/
cp -f automation.groovy /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/orchestrator.ui-4.0.0.1/app/resources/
cp -f plugin.com.ibm.orchestrator.rest-1.0.1.1.jar /drouter/ramdisk2/mnt/raid-volume/raid0/usr/servers/kernelservices/plugins/bundles/com.ibm.orchestrator.task/1.0.1.0/

(5) Start IWD server on Central Server 3:
service iwd start

3) Manual steps to update pdcollect files on Central Server 1.
Note: This fix is also included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). If you have already applied 2.3.0.1-CSI-ISCO-IF0003 to your SCO environment, ignore this part.

(1) Backup below files in '/iaas/pdcollect' on Central Server 1 (If you are using no-root pdcollect, please find the files in /home/<yourmechid>)
Components_nonroot.xml
Components.xml
Environment.xml
PDCOLLECT_Central_Server_Template.xml

(2) Update the files in /iaas/pdcollect with the ones in folder '2.3.0.1-CSI-ISCO-IF0003/scp-pdcollect/'


4) Upgrade Virtual Image Library (VIL) and VIL distributed proxy components on Central Server 2 and each Region server.

- Make sure VIL service and VIL proxy service are all stopped

(1)on Central Server 2, check vil service status by running the following command:

service vil status

If vil service is not stopped, run the following command to stop it.

service vil stop

(2)on each Region server, check vilProxy service status by running the following command:

service vilProxy status

If vilProxy service is not stopped, run the following command to stop it.

service vilProxy stop

- Upgrade VIL proxy on each Region Server where it is installed:

(1)copy il_proxy_install_package_23025.zip to the Region Server, in a directory of your choice.

(2)unpack il_proxy_install_package_23025.zip into a temporary directory <your_path>:

unzip il_proxy_install_package_23025.zip -d <your_path>

(3)navigate to <your_path> and upgrade Virtual Image Library proxy by running the following script:

cd <your_path>
./install_vil.sh -proxy -vilServer <hostnameFqdn>

Where
vilServer: Specifies the fully qualified host name or the IP address of the Virtual
Image Library server (central-server-2).
If you specify the fully qualified host name of the Virtual Image Library
server, the proxy must be able to resolve that host name. If it cannot,
add the IP address and host name to /etc/hosts before installing the
proxy.

- Upgrade VIL server and proxy on Central Server 2:

(1)copy il_install_package_23025.zip to Central Server 2, in a directory of your choice.

(2)unpack il_install_package_23025.zip into a temporary directory <your_path>:

unzip il_install_package_23025.zip -d <your_path>

(3)navigate to <your_path> and upgrade Virtual Image Library by running the following script:

cd <your_path>
./install_vil.sh -u wasadmin -p $smartcloud_password

5) Update the Java 64-bit installation on Central Server 1

- Check the Java current version:
/opt/ibm/java-x86_64-60/bin/java -fullversion

- Update to Java 6 SR16 if not already installed:
yum localupdate ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm
...
Updated:
ibm-java-x86_64-sdk.x86_64 0:6.0-16.0

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-x86_64-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16-20140418_01 (SR16)"

6) Update the Java 64-bit installation on Central Server 2 (used by Public Cloud Gateway)

- Check the Java current version:
/opt/ibm/java-x86_64-60/bin/java -fullversion

- Copy ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm to Central Server 2,
in a directory of your choice.

- Update to Java 6 SR16 if not already installed:
yum localupdate ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm
...
Updated:
ibm-java-x86_64-sdk.x86_64 0:6.0-16.0

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-x86_64-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16-20140418_01 (SR16)"

7) Install WAS interim fix 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 for VIL on Central Server 2

- In Security Bulletin: Multiple vulnerabilities in current IBM SDK for Java for WebSphere Application Server April 2014 CPU click on link named "PI19109" in section "For 8.0.0.0 through 8.0.0.8:"

- In PI19109: SHIP SDK 626 SR8 FOR WSAS V8.0.0.X you need to download the correct package (Linux 32-bit x86 AMD/Intel Java SDK) by clicking the FC link which lets you download the following package:

8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 (94943549 Bytes)

- When downloaded, copy 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 to the Central Server 2,
in a directory of your choice (<your_dir>).

- Stop VIL server if not already stopped, by running the following command:

service vil stop

- Install the interim fix IFPI19109, by running the following command:

/opt/IBM/InstallationManager/eclipse/tools/imcl install 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 -installationDirectory /opt/IBM/WebSphere/AppServer/
-repositories <your_dir>/8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109.zip

- If the installation completes successfully, the following message is displayed:
"Installed 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109_8.0.0.20140602_1627 to the /opt/IBM/WebSphere/AppServer directory."

- If the installation fails with the following message:
"ERROR: An interim fix for the Java SDK is installed already. Uninstall interim fix 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI08995
before installing a different Java SDK interim fix."

the interim fix IFPI08995 must first be uninstalled, by running the following command:
/opt/IBM/InstallationManager/eclipse/tools/imcl uninstall 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI08995 -installationDirectory /opt/IBM/WebSphere/AppServer/

and then the interim fix IFPI19109 can be installed as described above.

- Verify that the interim fix has been installed, checking that it is properly listed among the output of the following command:
/opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages

8) Update the Java 64-bit installation on Central Server 3 (used by SCUI)

- Check the Java current version:
/opt/ibm/java-x86_64-60/bin/java -fullversion

- Copy ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm to Central Server 3,
in a directory of your choice.

- Update to Java 6 SR16 if not already installed:
yum localupdate ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm
...
Updated:
ibm-java-x86_64-sdk.x86_64 0:6.0-16.0

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-x86_64-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16-20140418_01 (SR16)"

9) Update the Java 32-bit installation on Central Server 3 (used by IBM Workload Deployer)

- Check the Java current version:
/opt/ibm/java-i386-60/bin/java -fullversion

- Copy ibm-java-i386-sdk-6.0-16.0.i386.rpm to Central Server 3,
in a directory of your choice.

- Update to Java 6 SR16 if not already installed:
yum localupdate ibm-java-i386-sdk-6.0-16.0.i386.rpm
...
Updated:
ibm-java-i386-sdk.i386 0:6.0-16.0

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-i386-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxi3260sr16-20140418_01 (SR16)"

- Copy new security (unrestricted) policy files
cp /opt/ibm/java-i386-60/demo/jce/policy-files/unrestricted/*.jar /opt/ibm/java-i386-60/jre/lib/security/

10) Install WAS interim fix 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 for BPM on Central Server 4

- In Security Bulletin: Multiple vulnerabilities in current IBM SDK for Java for WebSphere Application Server April 2014 CPU click on link named "PI19108" in section "For V8.5.0.0 through 8.5.5.2 Full Profile:"

- In PI19108: SHIP SDK 626 SR8 FOR WSAS V8.5.0.X AND V8.5.5.X you need to download the correct package (Linux 64-bit x86 AMD/Intel Java SDK) by clicking the FC link which lets you download the following package:

8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 (103391758 Bytes)

- When downloaded, copy 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 to the Central Server 4,
in a directory of your choice (<your_dir>).

- Stop BPM if not already stopped, by running the following command (wait for services to stop and confirm with service bpm status afterwards):

service bpm stop

- Install the interim fix IFPI19108, by running the following command:

/opt/IBM/InstallationManager/eclipse/tools/imcl install 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 -installationDirectory /opt/ibm/BPM/v8.5
-repositories <your_dir>/8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108.zip

- If the installation fails with the following message:
"ERROR: An interim fix for the Java SDK is installed already. Uninstall interim fix 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI08994
before installing a different Java SDK interim fix."

the interim fix IFPI08995 must first be uninstalled, by running the following command:
/opt/IBM/InstallationManager/eclipse/tools/imcl uninstall 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI08994 -installationDirectory /opt/ibm/BPM/v8.5

and then the interim fix IFPI19109 can be installed as described above.
- Verify that the interim fix has been installed, checking that it is properly listed among the output of the following command:
/opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages

11) Update the 64-bit Java installation used by SmartCloud Entry on each VMware Region Server

- Make sure SmartCloud Entry is stopped on each VMware Region Server by running the following command:

service sce status
IBM SmartCloud Entry is stopped

- Copy sce310_jre_linux_installer.bin to the VMware Region Server, in a directory of your choice.

- Update to Java 6 SR16 from the sce310_jre_linux_installer.bin downloaded directory:

./sce310_jre_linux_installer.bin

Preparing to install...
Extracting the JRE from the installer archive...
Unpacking the JRE...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...

Launching installer...

===============================================================================
Choose Locale...
----------------

1- Deutsch
->2- English
...

CHOOSE LOCALE BY NUMBER:
===============================================================================
(created with InstallAnywhere)
-------------------------------------------------------------------------------

Preparing CONSOLE Mode Installation...

===============================================================================
Introduction
------------

InstallAnywhere will guide you through the JRE (Java Runtime Environment)
update for IBM SmartCloud Entry.

The JRE version of this update is 1.6.0-SR16.

It is strongly recommended that you quit all programs before continuing with
this installation.

Respond to each prompt to proceed to the next step in the installation. If you
want to change something on a previous step, type 'back'.

You may cancel this installation at any time by typing 'quit'.

PRESS <ENTER> TO CONTINUE:

===============================================================================
License Agreement
-----------------

Installation and use of IBM SmartCloud Entry JRE Update requires acceptance of
the following License Agreement:

International Program License Agreement

...

DO YOU ACCEPT THE TERMS OF THIS LICENSE AGREEMENT? (Y/N): Y

===============================================================================
IBM SmartCloud Entry Install Locations
--------------------------------------

The following IBM SmartCloud Entry installation locations require the JRE
version to be updated to version 1.6.0-SR16 FP1.

Select the install location to proceed with the JRE update.

->1- /opt/ibm/SCE31

ENTER THE NUMBER FOR YOUR CHOICE, OR PRESS <ENTER> TO ACCEPT THE DEFAULT::

===============================================================================
Pre-Installation Summary
------------------------

Please review the following information before continuing:

Installation folder:
/opt/ibm/SCE31

Product selected for update:
IBM SmartCloud Entry 3.1

Current JRE version:
1.6.0-SR13

Update JRE verson:
1.6.0-SR16

PRESS <ENTER> TO CONTINUE:

===============================================================================
Ready To Install
----------------

InstallAnywhere is now ready to update the JRE version of IBM SmartCloud Entry
3.1 at the following installation location:

/opt/ibm/SCE31

PRESS <ENTER> TO INSTALL:

===============================================================================
Installing...
-------------

[==================|==================|==================|==================]
[------------------|------------------|------------------|------------------]

===============================================================================
Installation Complete
---------------------

Congratulations. The IBM SmartCloud Entry JRE update installed successfully at
the following location:

/opt/ibm/SCE31

PRESS <ENTER> TO EXIT THE INSTALLER:

- Check that the updated Java version used by SmartCloud Entry is Java 6 SR16 FP1:
/opt/ibm/SCE31/jre/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16-20140418_01 (SR16)"

12) Upgrade OPENSTACK components

- Update yum repos with new OPENSTACK packages

(1)On Central Server 1, extract openstack.tar to your ifix path

tar -xf openstack.tar

Two folders are extracted, openstack_norach and openstack_x86.

Copy rpm pakcages in the above two folders to SCO yum repo.

copy -rf openstack_norach/* /data/repos/scp/ibm-rpms/openstack_noarch/
copy -rf openstack_x86/* /data/repos/scp/ibm-rpms/openstack_x86/

Run below commands to update yum repo:

yum clean all
createrepo /data/repos/scp/

(2)On each SCO Region server, perform same actions like (1) to update its yum repo.

- Backup configuration files

(1)On Central Server 2, backup below files/folders
/root/keystonerc
/etc/keystone
/etc/iaasgateway
/etc/my.cnf
/etc/qpid

(2)On each Region Server, backup below files/folders
/root/.SCE31
/root/openrc
/root/keystonerc
/etc/cinder
/etc/nova
/etc/glance
/etc/qpid
/etc/my.cnf

(3)On each Compute node, backup below files/folders
/etc/nova
/etc/my.cnf
/root/openrc

- Upgrade OPENSTACK rpm packages

On Central Server 2, each region server and each compute node, perform below action:

(1)Copy SP7_rpm_list to a directory of your choice on the server, <your_path>

(2)Run below command to update OPENSTACK components:
cd <your_path>
yum clean all
for i in $(cat SP7_rpm_list); do yum update $i;done

- Restore configuration files

(1)On Central Server 2:
/root/keystonerc
/etc/keystone
/etc/iaasgateway
/etc/my.cnf
/etc/qpid

(2)On each Region Server:
/root/.SCE31
/root/openrc
/root/keystonerc
/etc/cinder
/etc/nova
/etc/glance
/etc/qpid
/etc/my.cnf

(3)On each Compute node:
/etc/nova
/etc/my.cnf
/root/openrc

- Upgrade SCE on each Vmware Region server

(1)Copy 3.1.0.4-IBM-SCE-FP004-201407010432.zip to a directory of your choice on the server, <your_path>

(2)Extart 3.1.0.4-IBM-SCE-FP004-201407010432.zip in the directory where you copied it.

(3)Backup /opt/ibm/SCE31/program/skc.ini

(4)Upgrade SCE：

Login into SmartCloud Entry console running
telnet localhost 7777

You will see a prompt like
osgi>

In this console type the following commands (the commands are in italics, then you can also see an example of the output):
osgi> showrepos

Metadata repositories:

Artifacts repositories:

repositories:

Artifacts repositories:

file:/C:/Users/IBM_ADMIN/.eclipse/207580638/p2/org.eclipse.

equinox.p2.core/cache/

If the repository that is storing the extracted files is not available, use the addrepo command to add that repository.
osgi> addrepo file:<the absolute path when you unpacked the zip file>
SKC update repository added

Install the updates by using the installupdates command.
osgi> installupdates
SKC updates to install:
com.ibm.cfs.product 3.1.0.3-201403100300 ==> com.ibm.cfs.product 3.1.0.4-201407010432
SKC update done

When the update is complete, activate the changes by using the close command to end the OSGi session, then restarting SmartCloud Entry.
osgi> close

(5)Restore the original copy of /opt/ibm/SCE31/program/skc.ini

(6)Restart SmartCloud Entry running
service sce restart

13) Manual steps to install fix for vCenter 5.5 support on each VMware Region Server.

(1) Copy package '3.1.0.4-IBM-SCE-IF001-201407170051.zip' to the region server $your_folder.

(2) On VMware Region Server, extact the fix package:
cd $your_folder
unzip 3.1.0.4-IBM-SCE-IF001-201407170051.zip

(3) Ensure SCE service is stopped.
service sce status
If SCE service is runnning, stop it with below command.
service sce stop

(4) Backup configure file:
cp /opt/ibm/SCE31/program/skc.ini $your_folder/

(5) Install SCE ifix:

Login into SmartCloud Entry console running
telnet localhost 7777

You will see a prompt like
osgi>

In this console type the following commands (the commands are in italics, then you can also see an example of the output):
osgi> showrepos

Metadata repositories:

Artifacts repositories:

repositories:

Artifacts repositories:

file:/C:/Users/IBM_ADMIN/.eclipse/207580638/p2/org.eclipse.

equinox.p2.core/cache/

If the repository that is storing the extracted files is not available, use the addrepo command to add that repository.
osgi> addrepo file:<the absolute path when you unpacked the zip file>
SKC update repository added

Install the updates by using the installupdates command.
osgi> installupdates
SKC updates to install:
com.ibm.cfs.product 3.1.0.4-201407010432 ==> com.ibm.cfs.product 3.1.0.4-201407162230
SKC update done

When the update is complete, activate the changes by using the close command to end the OSGi session, then restarting SmartCloud Entry.
osgi> close

(6)Restore the original copy of /opt/ibm/SCE31/program/skc.ini

(7)Add "-Dhttps.protocols=TLSv1" in /opt/ibm/SCE31/program/skc.ini

(8)Restart SmartCloud Entry running
service sce restart

(9)Start SCE service:
service sce start

14) Manual steps to apply the IWD cumulative fix on Central Server 3.

(1) Copy IFIX0004_IWD_SCO2301_20140711-0842-753.tar on Central Server 3.
(2) Extract IFIX0004_IWD_SCO2301_20140711-0842-753.tar in a directory of your choice by running the following command:
tar -xf IFIX0004_IWD_SCO2301_20140711-0842-753.tar

(3) Change directory to the one where IFIX0004_IWD_SCO2301_20140711-0842-753 was extracted.

The following files are extracted:
checksums_20140307-0054-235
checksums_20140411-0853-495
checksums_20140514-0635-815
checksums_20140515-1212-353
checksums_20140602-0115-622
checksums_20140711-0842-753
iwd_db2_sco2301_20140711-0842-753.tar
iwd_sco2301_20140711-0842-753.tar
README.txt
removefiles.txt
version
install_fix.sh

(4) Modify DB2 database
Connect to DB2 server on Central Server 1, change user to db2inst1 and backup two databases, rainmake and storhous.
Execute all SQL commands included in SQL files in iwd_db2_sco2301_20140711-0842-753.tar archive.
SQL files:
opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.cloud-4.0.0.1/sql/db2/create-4.0.0.1-ifix.sql
opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.users-4.0.0.1/sql/db2/create-4.0.0.1-ifix.sql

Example:
Copy iwd_db2_sco2301_20140711-0842-753.tar file to Central Server 1. Extract it to a directory of your choice.
su - db2inst1
db2 connect to rainmake
db2 -tvsf <path_to_file>/opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.cloud-4.0.0.1/sql/db2/create-4.0.0.1-ifix.sql
db2 -tvsf <path_to_file>/opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.users-4.0.0.1/sql/db2/create-4.0.0.1-ifix.sql

or line by line:
db2 ALTER TABLE groups ADD COLUMN tmpname VARCHAR(64)
...
(5) Use script to intall the cumulative fix
Run command: ./install_fix.sh (Note: this need to be carry out in current path of install_fix.sh)
The script makes a backup copy of all files under /opt/ibm/rainmaker-backup/20140711-0842-753, verifies all files that are being modified by the IFIX installer and installs the fix.

(6) Manual steps to intall the cumulative fix
*WARNING: If you ran the script ./install_fix.sh from the Standard tab above and it completed successfully then the steps in this part may be skipped.

-Stop the workload deployer.
In order to stop the workload deployer, log as root into the Central Server 3 host (workload deployer machine) and execute following command:
service iwd stop

-Make a backup
It is strongly recommended to create a backup copy of all files that will be replaced with the iFix before applying the iFix.
Note: The backuped files should not be stored in IWD folders tree (locations /opt/ibm/* /drouter/*)

Backup the files list in 'checksums_20140711-0842-753' and 'removefiles.txt'.
Backup two databases: rainmake and storhous.

-Install the fix
Verify that IWD component is at required version to apply the iFix. Go to folder where iFix is extracted and run following command for each of the provided checksum files:

md5sum -c checksumfile

If all files are identical (OK status) the fix procedure may be continued.

Ensure you have backuped the files required in Step 3.2.

Replace all IWD files which are listed in 'checksums_20140711-0842-753' with the ones in package 'iwd_sco2301_20140711-0842-753'.
For example,
tar -xvf iwd_sco2301_20140711-0842-753.tar
cp ./opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.openstack.shim-4.0.0.1/lib/rainmaker.openstack.shim.jar /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.openstack.shim-4.0.0.1/lib/rainmaker.openstack.shim.jar
...

Remove all files listed in "removefiles.txt" file.

Validate fix installation
md5sum -c checksums_20140711-0842-753

If all files are identical (OK status) the fix is installed successfully.

In case of any failure, please try the fix again or restore the files from backup.

-Start Worklad Deployer
Restart the workload deployer by running:
service iwd start