Select the Standard tab for upgrade instructions using the iFix installation script.

Select the Manual tab for the instructions on how to manually upgrade each component.

Important Notice: Ensure your environment has met all the requirements in the Prerequisites section above before applying this iFix.

1) Stop SmartCloud Orchestrator on Central Server 1

Change to directory: /iaas/scorchestrator/
Stop SmartCloud Orchestrator: ./SCOrchestrator.py --stop
===>>> Stopping Smart Cloud Orchestrator
...
===>>> Stopping Smart Cloud Orchestrator complete

Wait until all components have stopped


2) Backup your SCO installation

Backup the Central Servers and Region Server virtual machines:
for VMware hosted virtual machines, see Taking Snaphots for information about taking snapshots of the virtual machines.


3) Copy and extract 2.3.0.1-CSI-ISCO-IF0006.tar on Central Server 1

Extract 2.3.0.1-CSI-ISCO-IF0006.tar in a directory of your choice <your_dir> by running the following command:

Note: Do not place the package under /root path

tar -xvf 2.3.0.1-CSI-ISCO-IF0006.tar

The following files are extracted:

SmartCloud_Orchestrator_2.3.0.1_IF0006.fxtag
   ifix6.py
   helper.py
   install_files/
     -02984.999.000.rtc-190749-nova-147e3d8da-ifix-el6.tgz
     -3.1.0.4-IBM-SCE-IF003-201410100432.zip
     -8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI20798.zip
     -8.5.0.0-WS-BPM-IFJR47778.zip
     -8.5.0.0-WS-BPM-IFJR47937.zip
     -8.5.0.0-WS-BPM-IFJR48541.zip
     -8.5.0.0-WS-BPM-IFJR48570.zip
     -8.5.0.0-WS-BPM-IFJR48704.zip
     -8.5.0.0-WS-BPM-IFJR49864.zip
     -8.5.0.0-WS-BPM-IFJR51596.zip
     -8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI20797.zip
     -bpm8.5.0.0IFJR.py
     -db2v101fp3a.py
     -helper.py
     -HybridCSB-API.war
     -ibm-java-i386-sdk-6.0-16.1.i386.rpm
     -ibm-java-x86_64-sdk-6.0-16.1.x86_64.rpm
     -ICCT_Install_2.3.0.1-20.zip
     -IFIX_IWD_SCO2301_IFIX06_20141119-1243-200.tar
     -il_install_package_23027.zip
     -il_proxy_install_package_23027.zip
     -index.gt
     -iwd_fix.py
     -java632sr16fp1.py
     -java664sr16fp1.py
     -keystone_memcache.zip
     -ldapauth.py
     -n3.app_1.0.0.20141021-1433.jar
     -n3.orchestrator.app_1.0.0.20141021-1433.jar
     -nova-cloud-modify
     -openstack-nova-201411241107.py
     -plugin.com.ibm.orchestrator.BPMInvoker-1.0.1.1.jar
     -plugin.com.ibm.orchestrator.rest-1.0.1.1.jar
     -rtc-190961-nova-283ad24da-ifix-el6.tgz
     -sce310fp4aparse60379.py
     -sce310java664sr16fp1.py
     -sce310_jre_linux_installer.bin
     -smartcloud-2013.1-1.1.4.ibm.201410270110.noarch.rpm
     -smartcloud-2013.1-1.1.4.ibm.201412010336.noarch.rpm
     -smartcloud-201412010336.py    
     -vilp23027.py
     -vils23027.py
     -vmware.properties_ifix6
     -was80java32-ifpi20798.py
     -was85javasdk64-ifpi20797.py    

   2.3.0.1-CSI-ISCO-IF0005/
     -SmartCloud_Orchestrator_2.3.0.1_IF0005.fxtag
     -ifix5.py
     -0001-enable-multiple-keystone-all-worker-processes.patch              
     -PSIRT1876.patch                          
     -iaasgateway-2013.1-1.1.4.ibm.201409112157.noarch.rpm
     -rtc-184008-keystone-11c387264-ifix-el6.tar.gz
     -8.5.0.0-WS-BPMPCPD-IFJR48696.zip                          
     -iaasgateway_http.cfg                                  
     -keystone_zz00267.patch
     -smartcloud-2013.1-1.1.3.ibm.201407310025.noarch.rpm
     -smartcloud-2013.1-1.1.4.ibm.201407310043.noarch.rpm
     -helper.py                                                
     -IFIX005_IWD_SCO2301_20140902-0433-971.tar
     -install_script/
        -9mjhzl_iaasgateway.py
        -9mjhzl_keystone.py  
        -bpm_jr48696.py  
        -helper.py  
        -iwd_fix.py
        -psirt_1838.py  
        -psirt_1876.py  
        -zz00267.py

      -2.3.0.1-CSI-ISCO-IF0004/
        -3.1.0.4-IBM-SCE-FP004-201407010432.zip
        -3.1.0.4-IBM-SCE-IF001-201407170051.zip
        -helper.py
        -ibm-java-i386-sdk-6.0-16.0.i386.rpm
        -ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm
        -IBM_SmartCloud_Provisioning_Core_2.3.0.1_IF0004-2.3.0.1.fxtag
        -ICCT_Install_2.3.0.1-17.zip
        -ifix4.py
        -il_install_package_23025.zip
        -il_proxy_install_package_23025.zip  
        -openstack.tar
        -SP7_rpm_list
        -sce310_jre_linux_installer.bin
        -SmartCloud_Orchestrator_2.3.0.1_IF0004.fxtag
        -install_script/
           -create_new_repo.py
           -helper.py
           -iwd_fix.py
           -java632sr16fp1.py
           -java664sr16fp1.py
           -sce310fp4.py
           -sce310fp4fix.py
           -sce310java664sr16fp1.py
           -update_pack.py
           -vilp23025.py
           -vils23025.py
           -was80javasdk32-ifpi19109.py
           -was85javasdk64-ifpi19108.py
           
        -2.3.0.1-CSI-ISCO-IF0003/
           -helper.py
           -ifix3.py
           -SmartCloud_Orchestrator_2.3.0.1_IF0003.fxtag
           -SE59389
              -se59389_fix.py
              -se59389_fix.tar.gz
           -scp-pdcollect
              -Components_nonroot.xml
              -Components.xml
              -Environment.xml
              -PDCOLLECT_Central_Server_Template.xml
           -ZZ00201/
              -iwd0002_0006checksum.txt
              -iwd0002_0006.py
              -opt/ibm/.../maestro.util.jar
           -ZZ00234
              -automation.groovy
           -ZZ00240
              -00_SCOrchestrator_Toolkit_2301_20140403_ifix1.twx
           -ZZ00242
              -config_network.sh
           -ZZ00244_ZZ00246
              -plugin.com.ibm.orchestrator.rest-1.0.1.1.jar


4) Upgrading the Image Construction and Composition Tool (ICCT)

- Review the topic Upgrading the Image Construction and Composition Tool in the IBM Knowledge Center for details..

- In step "Download and extract the compressed file to the computer where you want to upgrade or install the Image Construction and Composition Tool fix pack" use ICCT_Install_2.3.0.1-20.zip


5) Apply fix for the DB2 security vulnerabilities CVE-2013-6747 and CVE-2014-0963.

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

Follow the steps described in "20) Apply fix for the DB2 security vulnerabilities CVE-2013-6747 and CVE-2014-0963"
under "(1) Downloading the DB2 fix pack" on the Manual tab (above) to download that fix.

When downloaded, copy v10.1fp3a_linuxx64_server.tar.gz to the Central Server 1, in the ifix directory <your_dir>/2.3.0.1-CSI-ISCO-IF0006/installfiles.

Running the iFix installation script in the next step will check if the fix file is present and install if required.

Follow the steps described in "20) Apply fix for the DB2 security vulnerabilities CVE-2013-6747 and CVE-2014-0963"
under "(2) Installing the DB2 fix pack on Central Server 1 and each Region Servers (if shared DB is not used)"
on the Manual tab (above) if you want to install that fix manually.


6) Download additional BPM fix for APAR JR51814 to be installed when applying fix for APAR ZZ00292 (Exposed Workflows in BPM REST causing 2min respnse)

Note: This is an optional step for applying fix for APAR ZZ00292 which is described in "30) Apply fix for APAR ZZ00292 (Exposed Workflows in BPM REST causing 2min respnse)" on the Manual tab (above).

Login to "IBM Support: Fix Central" (FC) and download interim fix "8.5.0.0-WS-BPM-IFJR51814" for APAR JR51814 for BPM 8.5.0.0 if officially available from FC

Select "IBM Business Process Manager Standard" as product selector, "8.5.0.0" as installed version and "Linux 64-bit,x86_64" as platform.

Select "Browse for fixes" and from the list of fixes select/download interim fix: 8.5.0.0-WS-BPM-IFJR51814.

When downloaded, copy 8.5.0.0-WS-BPM-IFJR51814.zip to the Central Server 1, in the ifix directory <your_dir>/2.3.0.1-CSI-ISCO-IF0006/installfiles.

Running the iFix installation script in the next step will check if the fix file is present and install if required.


7) Run the iFix installation script on Central Server 1

WARNING: Review the steps on the Manual tab (above) if you don't want to run iFix installation script on Central Server 1.

- From the directory where you unpacked the iFix package, run the iFix installation script by using the following command:

./ifix6.py --cs2=<cs2> --cs3=<cs3> --cs4=<cs4> --rs=<regionservers> --cn=<compute_node> --wasa=<wasadmin> --wasp=<password>

Example: ./ifix6.py --cs2=10.10.0.12 --cs3=10.10.0.13 --cs4=10.10.0.14 --rs=10.10.0.15,10.10.0.16 --cn=10.10.0.17 --wasa=wasadmin --wasp=passw0rd

You can use ./ifix6.py -h to show the usage.
Options:
      -h, --help          show this help message and exit
      --cs2=CS2           hostname/ip address of central server 2
      --cs3=CS3           hostname/ip address of central server 3
      --cs4=CS4           hostname/ip address of central server 4
      --rs=RS             (optional)list of hostnames/ip addresses of region
 servers format server1,server2,server3,...
      --cn=CN             (optional)list of hostnames/ip addresses of compute
 nodes format compute1,compute2,compute3,...
      --wasa=WASADMIN     virtual image library WAS administrator ID to be used
 during the installation procedure
      --wasp=WASPASSWORD  virtual image library WAS administrator password  


You can refer to log file ifix6.log in the script path for more detail information of the installation process.
Other log information:
VIL install/upgrade log: /tmp/fresh_install_vil.log, upgrade_vil.log on Central Server 2
VIL Proxy install/upgrade log: /tmp/fresh_install_proxy.log, upgrade_proxy.log on Region Server
IWD fix install log: /tmp/iwdifix6.log Central Server 3

Note: If you have already applied ifix4 manually in your environment, please manually create a file '/home/db2inst1/.db2ifix0004' on Central Server 1 before running script ifix6.py.


8) Apply fix for Iaasgateway cluster support

Note: This is an optional step which is intended for large environments.

Follow the steps described in "19) Apply fix for Iaasgateway cluster support" on the Manual tab (above)


9) Enable IWD notifications

Note: This is an optional step which is intended for large environments only (thousands of instances).

Follow the steps described in "14) Manual steps to apply the IWD cumulative fix on Central Server 3"
under "(6) Enable IWD notifications" on the Manual tab (above)


10) Start SmartCloud Orchestrator on Central Server 1

- Stop SmartCloud Orchestrator on Central Server 1 (to enable a clean start in the following step)
Change to directory: /iaas/scorchestrator/
Stop SmartCloud Orchestrator: ./SCOrchestrator.py --stop
===>>> Stopping Smart Cloud Orchestrator
...
===>>> Stopping Smart Cloud Orchestrator complete

Wait until all components have stopped

- Start SmartCloud Orchestrator on Central Server 1
Change to directory: /iaas/scorchestrator/
Start SmartCloud Orchestrator: ./SCOrchestrator.py --start

Wait until all components have started


11) Additional manual steps required to install fix for APAR ZZ00240.

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

Follow the steps described in "21) Apply fix for APAR ZZ00240" on the Manual tab (above)


12) Apply fix for APAR ZZ00256.

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

Follow the steps described in "22) Apply fix for APAR ZZ00256" on the Manual tab (above)


13) Apply fix for APAR SE58688 - KVM nodes going offline

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

Follow the steps described in "23) Apply fix for APAR SE58688 - KVM nodes going offline" on the Manual tab (above)


14) Apply fix for ZZ00242 on Central Server 1

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

Follow the steps described in "24) Apply fix for ZZ00242 on Central Server 1" on the Manual tab (above)


15) (Optional) Apply fix for APAR SE59801

Note: This fix is already included in iFix5 (2.3.0.1-CSI-ISCO-IF0005). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0005.

Follow the steps described in "25) (Optional) Apply fix for APAR SE59801" on the Manual tab (above)


16) Apply workaround for issue (Unable to provision VMs if user not in admin project) on Central Server 3

Follow the steps described in "14) Manual steps to apply the IWD cumulative fix on Central Server 3"
under "(5) Workaround for problem reported in PMR 63576,024,677 (Unable to provision VMs if user not in admin project)"
on the Manual tab (above)

*********************************************************************************************
WARNING: If you ran the script ifix6.py as outlined on the Standard tab and it completed successfully, then most of the steps in this section can be skipped. You will need to refer to the instructions on the Standard tab for additional manual steps which still need to be executed after running the script ifix6.py.

Note: Be sure to execute all the steps described in this section to ensure that any of the fixes in the pile (i.e. iFix3, iFix4, iFix5, and iFix6) are included.
*********************************************************************************************
1) Inventory Tagging

To make it easier for support to find out which SmartCloud Orchestrator fixes are installed on your system, copy some files to
/opt/ibm/SmartCloud_Orchestrator/properties/version/ on Central-Server-3 (if not present on Central-Server-3 create the directory)

scp *.fxtag \
Central-Server-3:/opt/ibm/SmartCloud_Orchestrator/properties/version/

Note: For IFIX6 copy the provided FXTAG file SmartCloud_Orchestrator_2.3.0.1_IF0006.fxtag


2) Apply IWD fixes on Central Server 3.

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

This fix depends on IFIX005_IWD_SCO2301_20140902-0433-971/iwd_sco2301_20140902-0433-971.tar and can not be installed separately. This fix will not work without IFIX005_IWD_SCO2301_20140902-0433-971/iwd_sco2301_20140902-0433-971.tar installed.

(2.1) Copy the following files from Central Server 1 to $your_folder on Central Server 3.

2.3.0.1-CSI-ISCO-IF0004/2.3.0.1-CSI-ISCO-IF0003/SmartCloud_Orchestrator_2.3.0.1_IF0003.fxtag
2.3.0.1-CSI-ISCO-IF0004/2.3.0.1-CSI-ISCO-IF0003/ZZ00234/automation.groovy
2.3.0.1-CSI-ISCO-IF0004/2.3.0.1-CSI-ISCO-IF0003/ZZ00244_ZZ00246/plugin.com.ibm.orchestrator.rest-1.0.1.1.jar

(2.2) Stop IWD server on Central Server 3:

service iwd stop

(2.3) Backup the following files on Central Server 3:

mkdir -p /tmp/$your_backup
cd /tmp/$your_backup
cp --preserve /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/maestro.util-4.0.0.1/lib/maestro.util.jar maestro.util.jar_ori
cp --preserve /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/orchestrator.ui-4.0.0.1/app/resources/automation.groovy automation.groovy_ori
cp --preserve /drouter/ramdisk2/mnt/raid-volume/raid0/usr/servers/kernelservices/plugins/bundles/com.ibm.orchestrator.task/1.0.1.0 plugin.com.ibm.orchestrator.rest-1.0.1.1.jar plugin.com.ibm.orchestrator.rest-1.0.1.1.jar_ori

(2.4) Update the files on Central Server 3

cd $your_folder
cp -f maestro.util.jar /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/maestro.util-4.0.0.1/lib/
cp -f automation.groovy /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/orchestrator.ui-4.0.0.1/app/resources/
cp -f plugin.com.ibm.orchestrator.rest-1.0.1.1.jar /drouter/ramdisk2/mnt/raid-volume/raid0/usr/servers/kernelservices/plugins/bundles/com.ibm.orchestrator.task/1.0.1.0/

(2.5) Start IWD server on Central Server 3:

service iwd start


3) Update pdcollect files on Central Server 1.

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

(1) Backup below files in '/iaas/pdcollect' on Central Server 1 (If you are using no-root pdcollect, please find the files in /home/<yourmechid>)
Components_nonroot.xml
Components.xml
Environment.xml
PDCOLLECT_Central_Server_Template.xml

(2) Update the files in /iaas/pdcollect with the ones in folder '2.3.0.1-CSI-ISCO-IF0004/2.3.0.1-CSI-ISCO-IF0003/scp-pdcollect/'


4) Upgrade Virtual Image Library (VIL) and VIL distributed proxy components with BL 23027 on Central Server 2 and each Region server

- Make sure VIL service and VIL proxy service are all stopped

(1) on Central Server 2, check vil service status by running the following command:

service vil status

If vil service is not stopped, run the following command to stop it.

service vil stop

(2) on each Region server, check vilProxy service status by running the following command:

service vilProxy status

If vilProxy service is not stopped, run the following command to stop it.

service vilProxy stop

- Upgrade VIL proxy on each Region Server where it is installed:

(1) copy il_proxy_install_package_23027.zip to the Region Server, in a directory of your choice.

(2) unpack il_proxy_install_package_23027.zip into a temporary directory <your_path>:

unzip il_proxy_install_package_23027.zip -d <your_path>

(3) navigate to <your_path> and upgrade Virtual Image Library proxy by running the following script:

cd <your_path>
./install_vil.sh -proxy -vilServer <hostnameFqdn>

Where
vilServer: Specifies the fully qualified host name or the IP address of the Virtual
Image Library server (central-server-2).
If you specify the fully qualified host name of the Virtual Image Library
server, the proxy must be able to resolve that host name. If it cannot,
add the IP address and host name to /etc/hosts before installing the
proxy.

- Upgrade VIL server and proxy on Central Server 2:

(1) copy il_install_package_23027.zip to Central Server 2, in a directory of your choice.

(2) unpack il_install_package_23027.zip into a temporary directory <your_path>:

unzip il_install_package_23027.zip -d <your_path>

(3) navigate to <your_path> and upgrade Virtual Image Library by running the following script:

cd <your_path>
./install_vil.sh -u wasadmin -p $smartcloud_password


5) Update the Java 64-bit installation on Central Server 1

- Check the Java current version:
/opt/ibm/java-x86_64-60/bin/java -fullversion

- Update to Java 6 SR16 FP1 if not already installed:
yum localupdate ibm-java-x86_64-sdk-6.0-16.1.x86_64.rpm
...
Updated:
ibm-java-x86_64-sdk.x86_64 0:6.0-16.1

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-x86_64-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16fp1-20140706_01 (SR16 FP1)"


6) Update the Java 64-bit installation on Central Server 2 (used by Public Cloud Gateway)

- Check the Java current version:
/opt/ibm/java-x86_64-60/bin/java -fullversion

- Copy ibm-java-x86_64-sdk-6.0-16.1.x86_64.rpm to Central Server 2,
in a directory of your choice.

- Update to Java 6 SR16 FP1 if not already installed:
yum localupdate ibm-java-x86_64-sdk-6.0-16.1.x86_64.rpm
...
Updated:
ibm-java-x86_64-sdk.x86_64 0:6.0-16.1

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-x86_64-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16fp1-20140706_01 (SR16 FP1)"


7) Install WAS interim fix 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI20798.zip for VIL on Central Server 2

Note: This fix is included in the IFIX6 package. Use 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI20798.zip under installfiles in the directory where you extracted 2.3.0.1-CSI-ISCO-IF0006.tar on Central Server 1.

8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI20798.zip (94977587 Bytes)

- Copy 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI20798.zip to the Central Server 2,
in a directory of your choice (<your_dir>).

- Stop VIL server if not already stopped, by running the following command:

service vil stop

- Install the interim fix IFPI20798, by running the following command:

/opt/IBM/InstallationManager/eclipse/tools/imcl install 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI20798 -installationDirectory /opt/IBM/WebSphere/AppServer/
-repositories <your_dir>/8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI20798.zip

- If the installation completes successfully, the following message is displayed:
"Installed 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI20798_8.0.0.20140721_1021 to the /opt/IBM/WebSphere/AppServer directory."

- If the installation fails with the following message:
ERROR: An interim fix for the Java SDK is installed already. Uninstall interim fix 8.0.0.0-WS-WASJavaSDK-LinuxX32-<interim_fix_code> before installing a different Java SDK interim fix."
then the interim fix <interim_fix_code> must first be uninstalled, by running the following command:

/opt/IBM/InstallationManager/eclipse/tools/imcl uninstall 8.0.0.0-WS-WASJavaSDK-LinuxX32-<interim_fix_code> -installationDirectory /opt/IBM/WebSphere/AppServer/

and then the interim fix IFPI20798 can be installed as described above.

- Verify that the interim fix has been installed, checking that it is properly listed among the output of the following command:

/opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages


8) Update the Java 64-bit installation on Central Server 3 (used by SCUI)

- Check the Java current version:
/opt/ibm/java-x86_64-60/bin/java -fullversion

- Copy ibm-java-x86_64-sdk-6.0-16.1.x86_64.rpm to Central Server 3,
in a directory of your choice.

- Update to Java 6 SR16 FP1 if not already installed:
yum localupdate ibm-java-x86_64-sdk-6.0-16.1.x86_64.rpm
...
Updated:
ibm-java-x86_64-sdk.x86_64 0:6.0-16.1

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-x86_64-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16fp1-20140706_01 (SR16 FP1)"


9) Update the Java 32-bit installation on Central Server 3 (used by IBM Workload Deployer)

- Check the Java current version:
/opt/ibm/java-i386-60/bin/java -fullversion

- Copy ibm-java-i386-sdk-6.0-16.1.i386.rpm to Central Server 3,
in a directory of your choice.

- Update to Java 6 SR16 FP1 if not already installed:
yum localupdate ibm-java-i386-sdk-6.0-16.1.i386.rpm
...
Updated:
ibm-java-i386-sdk.i386 0:6.0-16.1

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-i386-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxi3260sr16fp1-20140706_01 (SR16 FP1)"

- Copy new security (unrestricted) policy files
cp /opt/ibm/java-i386-60/demo/jce/policy-files/unrestricted/*.jar /opt/ibm/java-i386-60/jre/lib/security/


10) Apply WAS interim fix 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI20797.zip for BPM on Central Server 4

Note: This fix is included in the IFIX6 package. Use 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI20797.zip under installfiles in the directory where you extracted 2.3.0.1-CSI-ISCO-IF0006.tar on Central Server 1.

8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI20797.zip (103430562 Bytes)

- Copy 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI20797.zip to the Central Server 4,
in a directory of your choice (<your_dir>).

- Stop BPM if not already stopped, by running the following command (wait for services to stop and confirm with
service bpm status afterwards):

service bpm stop

- Install the interim fix IFPI20797, by running the following command:

/opt/IBM/InstallationManager/eclipse/tools/imcl install 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI20797 -installationDirectory /opt/ibm/BPM/v8.5
-repositories <your_dir>/8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI20797.zip

- If the installation completes successfully, the following message is displayed:
"Installed 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI20797_8.5.0.20140730_1058 to the /opt/ibm/BPM/v8.5 directory."

- If the installation fails with the following message:
ERROR: An interim fix for the Java SDK is installed already. Uninstall interim fix 8.5.0.0-WS-WASJavaSDK-LinuxX64-<interim_fix_code> before installing a different Java SDK interim fix."
then the interim fix <interim_fix_code> must first be uninstalled, by running the following command:

/opt/IBM/InstallationManager/eclipse/tools/imcl uninstall 8.5.0.0-WS-WASJavaSDK-LinuxX64-<interim_fix_code> -installationDirectory /opt/ibm/BPM/v8.5

and then the interim fix IFPI20797 can be installed as described above.

- Verify that the interim fix has been installed, checking that it is properly listed among the output of the following command:

/opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages


11) Update the 64-bit Java installation used by SmartCloud Entry on each VMware Region Server

- Make sure SmartCloud Entry is stopped on each VMware Region Server
by running the following command:

service sce status
IBM SmartCloud Entry is stopped

- Copy sce310_jre_linux_installer.bin to the VMware Region Server, in a directory of your choice.

- Update to Java 6 SR16 FP1 from the sce310_jre_linux_installer.bin downloaded directory:

./sce310_jre_linux_installer.bin

Preparing to install...
Extracting the JRE from the installer archive...
Unpacking the JRE...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...

Launching installer...

===============================================================================
Choose Locale...
----------------

1- Deutsch
->2- English
...

CHOOSE LOCALE BY NUMBER:
===============================================================================
(created with InstallAnywhere)
-------------------------------------------------------------------------------

Preparing CONSOLE Mode Installation...

===============================================================================
Introduction
------------

InstallAnywhere will guide you through the JRE (Java Runtime Environment)
update for IBM SmartCloud Entry.

The JRE version of this update is 1.6.0-SR16.

It is strongly recommended that you quit all programs before continuing with
this installation.

Respond to each prompt to proceed to the next step in the installation. If you
want to change something on a previous step, type 'back'.

You may cancel this installation at any time by typing 'quit'.

PRESS <ENTER> TO CONTINUE:

===============================================================================
License Agreement
-----------------

Installation and use of IBM SmartCloud Entry JRE Update requires acceptance of
the following License Agreement:

International Program License Agreement

...

DO YOU ACCEPT THE TERMS OF THIS LICENSE AGREEMENT? (Y/N): Y

===============================================================================
IBM SmartCloud Entry Install Locations
--------------------------------------

The following IBM SmartCloud Entry installation locations require the JRE
version to be updated to version 1.6.0-SR16 FP1.

Select the install location to proceed with the JRE update.

->1- /opt/ibm/SCE31

ENTER THE NUMBER FOR YOUR CHOICE, OR PRESS <ENTER> TO ACCEPT THE DEFAULT::

===============================================================================
Pre-Installation Summary
------------------------

Please review the following information before continuing:

Installation folder:
/opt/ibm/SCE31

Product selected for update:
IBM SmartCloud Entry 3.1

Current JRE version:
1.6.0-SR16

Update JRE version:
1.6.0-SR16 FP1

PRESS <ENTER> TO CONTINUE:

===============================================================================
Ready To Install
----------------

InstallAnywhere is now ready to update the JRE version of IBM SmartCloud Entry
3.1 at the following installation location:

/opt/ibm/SCE31

PRESS <ENTER> TO INSTALL:

===============================================================================
Installing...
-------------

[==================|==================|==================|==================]
[------------------|------------------|------------------|------------------]

===============================================================================
Installation Complete
---------------------

Congratulations. The IBM SmartCloud Entry JRE update installed successfully at
the following location:

/opt/ibm/SCE31

PRESS <ENTER> TO EXIT THE INSTALLER:

- Check that the updated Java version used by SmartCloud Entry is Java 6 SR16 FP1:
/opt/ibm/SCE31/jre/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16-20140418_01 (SR16)"
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16fp1-20140706_01 (SR16 FP1)"


12) Upgrade OPENSTACK components

Note: This fix is already included in iFix4 (2.3.0.1-CSI-ISCO-IF0004). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0004 or a higher IFIX version to your SCO environment.

- Update yum repos with new OPENSTACK packages

(1) On Central Server 1, extract openstack.tar to your ifix path

tar -xf openstack.tar

Two folders are extracted, openstack_norach and openstack_x86.

Copy rpm packages in the above two folders to SCO yum repo.

copy -rf openstack_norach/* /data/repos/scp/ibm-rpms/openstack_noarch/
copy -rf openstack_x86/* /data/repos/scp/ibm-rpms/openstack_x86/

Run below commands to update yum repo:

yum clean all
createrepo /data/repos/scp/

(2) On each SCO Region server, perform same actions like (1) to update its yum repo.


- Backup configuration files

(1) On Central Server 2, backup below files/folders

/root/keystonerc
/etc/keystone
/etc/iaasgateway
/etc/my.cnf
/etc/qpid

(2) On each Region Server, backup below files/folders

/root/.SCE31
/root/openrc
/root/keystonerc
/etc/cinder
/etc/nova
/etc/glance
/etc/qpid
/etc/my.cnf

(3) On each Compute node, backup below files/folders

/etc/nova
/etc/my.cnf
/root/openrc

- Upgrade OPENSTACK rpm packages

On Central Server 2, each region server and each compute node, perform below action:

(1) Copy SP7_rpm_list to a directory of your choice on the server, <your_path>

(2) Run below command to update OPENSTACK components:
cd <your_path>
yum clean all
for i in $(cat SP7_rpm_list); do yum update $i;done

- Restore configuration files

(1) On Central Server 2:

/root/keystonerc
/etc/keystone
/etc/iaasgateway
/etc/my.cnf
/etc/qpid

(2) On each Region Server:

/root/.SCE31
/root/openrc
/root/keystonerc
/etc/cinder
/etc/nova
/etc/glance
/etc/qpid
/etc/my.cnf

(3) On each Compute node:

/etc/nova
/etc/my.cnf
/root/openrc

- Upgrade SCE on each Vmware Region server

(1) Copy 3.1.0.4-IBM-SCE-FP004-201407010432.zip to a directory of your choice on the server, <your_path>

(2) Extart 3.1.0.4-IBM-SCE-FP004-201407010432.zip in the directory where you copied it.

(3) Backup /opt/ibm/SCE31/program/skc.ini

(4) Upgrade SCE:

Login into SmartCloud Entry console running
telnet localhost 7777

You will see a prompt like
osgi>

In this console type the following commands (the commands are in italics, then you can also see an example of the output):
osgi> showrepos

Metadata repositories:

Artifacts repositories:

repositories:

Artifacts repositories:

file:/C:/Users/IBM_ADMIN/.eclipse/207580638/p2/org.eclipse.

equinox.p2.core/cache/

If the repository that is storing the extracted files is not available, use the addrepo command to add that repository.
osgi> addrepo file:<the absolute path when you unpacked the zip file>
SKC update repository added

Install the updates by using the installupdates command.
osgi> installupdates
SKC updates to install:
com.ibm.cfs.product 3.1.0.3-201403100300 ==> com.ibm.cfs.product 3.1.0.4-201407010432
SKC update done

When the update is complete, activate the changes by using the close command to end the OSGi session, then restarting SmartCloud Entry.
osgi> close

(5) Restore the original copy of /opt/ibm/SCE31/program/skc.ini

(6) Restart SmartCloud Entry running
service sce restart

- Apply patches to OpenStack Components

Note: Please use ./SCOrchestrator.py to stop all the services before applying the changes.

(1) use below command to find python directory:

$ python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

For example:
$ python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/usr/lib/python2.6/site-packages

From the command, '/usr/lib/python2.6/site-packages' is #{your_python_dir} which will be used in following steps.

(2) apply keystone patch on CS-2

login to CS-2 and carry out below commands:
$ scp root@$cs-1_ip:/data/repos/scp/patch/keystone.patch <your_directory>
$ cd #{your_python_dir}
$ patch -p1 -N -f < <your_directory>/keystone.patch

(3) apply OpenStack patches on each Region Server

login to Region Server and carry out below commands:

-- apply nova patch
$ cd #{your_python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch
$ cd /usr
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch

Note: For this part, please ignore the error message reporting "can't find file to patch" for below files:
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.json.tpl
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.xml.tpl
nova/tests/integrated/test_api_samples.py
nova/tests/test_configdrive2.py
nova/tests/test_nova_manage.py
doc/api_samples/all_extensions/extensions-get-resp.json
doc/api_samples/all_extensions/extensions-get-resp.xml

-- apply glance patch
$ cd #{python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/glance.patch

-- apply cinder patch
$ cd #{python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/cinder.patch

(4) apply nova patch on each KVM compute node

login to KVM compute node and carry out below commands:
$ scp root@$rs_ip:/data/repos/scp/patch/nova.patch <your_directory>
$ cd #{your_python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch
$ cd /usr
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch

Note: For this part, please ignore the error message reporting "can't find file to patch" for below files:
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.json.tpl
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.xml.tpl
nova/tests/integrated/test_api_samples.py
nova/tests/test_configdrive2.py
nova/tests/test_nova_manage.py
doc/api_samples/all_extensions/extensions-get-resp.json
doc/api_samples/all_extensions/extensions-get-resp.xml


13) Apply fix for vCenter 5.5 support on each VMware Region Server.

Note: This fix is already included in iFix4 (2.3.0.1-CSI-ISCO-IF0004). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0004 or a higher IFIX version to your SCO environment.

(1) Copy package '3.1.0.4-IBM-SCE-IF001-201407170051.zip' to the region server $your_folder.

(2) On VMware Region Server, extract the fix package:
cd $your_folder
unzip 3.1.0.4-IBM-SCE-IF001-201407170051.zip

(3) Ensure SCE service is stopped.
service sce status
If SCE service is running, stop it with below command.
service sce stop

(4) Backup configure file:
cp /opt/ibm/SCE31/program/skc.ini $your_folder/

(5) Install SCE ifix:

Login into SmartCloud Entry console running
telnet localhost 7777

You will see a prompt like
osgi>

In this console type the following commands (the commands are in italics, then you can also see an example of the output):
osgi> showrepos

Metadata repositories:

Artifacts repositories:

repositories:

Artifacts repositories:

file:/C:/Users/IBM_ADMIN/.eclipse/207580638/p2/org.eclipse.

equinox.p2.core/cache/

If the repository that is storing the extracted files is not available, use the addrepo command to add that repository.
osgi> addrepo file:<the absolute path when you unpacked the zip file>
SKC update repository added

Install the updates by using the installupdates command.
osgi> installupdates
SKC updates to install:
com.ibm.cfs.product 3.1.0.4-201407010432 ==> com.ibm.cfs.product 3.1.0.4-201407162230
SKC update done

When the update is complete, activate the changes by using the close command to end the OSGi session, then restarting SmartCloud Entry.
osgi> close

(6)Restore the original copy of /opt/ibm/SCE31/program/skc.ini

(7)Add "-Dhttps.protocols=TLSv1" in /opt/ibm/SCE31/program/skc.ini

(8)Restart SmartCloud Entry running
service sce restart

(9)Start SCE service:
service sce start


14) Apply IWD cumulative fix on Central Server 3.

(1) Copy IFIX_IWD_SCO2301_IFIX06_20141119-1243-200.tar on Central Server 3.

(2) Extract IFIX_IWD_SCO2301_IFIX06_20141119-1243-200.tar in a directory of your choice by running the following command:
tar -xf IFIX_IWD_SCO2301_IFIX06_20141119-1243-200.tar

(3) Change directory to the one where IFIX_IWD_SCO2301_IFIX06_20141119-1243-200 was extracted.

The following files are extracted:
install_fix.sh
README.txt
version
checksums_20140307-0054-235
checksums_20140411-0853-495
checksums_20140514-0635-815
checksums_20140515-1212-353
checksums_20140602-0115-622
checksums_20140711-0842-753
checksums_20140827-0603-225
checksums_20140902-0433-971
checksums_20141023-0416-459
checksums_20141023-1136-786
checksums_20141104-0907-146
checksums_20141119-1243-200
iwd_db2_sco2301_20140902-0433-971.tar
iwd_sco2301_20141119-1243-200.tar
removefiles.txt

(4) Follow the steps in README.txt you extracted in the previous step

Note: The README describes two ways to install the IWD cumulative fix, the automated installation of the fix under 3.1 using the script install_fix.sh or the manual installation of the fix under 3.2. Please follow one of these two procedures.

The README also contains an optional step to enable notifications under "3.3 Enable notification (optional)"
which is outlined in more detail in the following step (5).

(5) Workaround for problem reported in PMR 63576,024,677 (Unable to provision VMs if user not in admin project)

Note: This is a mandotory step which is required to circumvent the problem reported in PMR 63576,024,677.
The final fix will be made available as part of the resolution of the related APAR.

(5.1) Open config file : /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/scp.ui-1.0.0/config/openstack.config on Central Server 3:

(5.2) Modify section "openstack" by adding "cache_service_versions" parameter with value false. After this change, openstack section should be similar to one below.

"openstack": {
"server_operation_timeout_sec": 3600,
"server_operation_check_interval_sec": 10,
"cache_service_versions": false,
"image_service":"vil"
},

(5.3) Restart IWD service on Central Server 3:

Use 'service iwd restart' to restart iwd service and wait about 15min for its initialization.

Note: Restarting IWD service is also be done as part of enabling IWD notifications in the following step.

(6) Enable IWD notifications

Note: This is an optional step which is intended for large environments only (thousands of instances).

(6.1) Update [DEFAULT] section in /etc/nova/nova.conf on each region server

Put the four lines under section #### QPID ####

notification_driver=nova.openstack.common.notifier.rabbit_notifier
instance_usage_audit=True
notify_on_state_change=vm_and_task_state
instance_usage_audit_period=hour

After the update on the region server restart openstack-nova-compute or openstack-smartcloud service

(6.2) Update /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.openstack.notifications-4.0.0.1/config/zero.config on Central Server 3:

/config/openstack/notifications = {
"general": {
"queue_factory_class": "org.apache.qpid.jndi.PropertiesFileInitialContextFactory",
"notifications_topic": "nova/notifications.info",
"vm_expiration_sec": 6000,
"vm_load_on_init": true,
"resolving_threads": 5,
"resolving_threads_processing_delay_sec": 5,
"consumer_connection_timeout_sec": 7200,
"consumer_error_delay_sec": 60

},
"regions": {
"<region-1-name>" : { <------Note: here is the region server name
"brokerlist" : "tcp://<region-1-ip>:5672" <------Note: here is the region server ip
},
"<region-2-name>" " { <------Note: For each of the region server, need a part like that.
"brokerlist" : "tcp://<region-2-ip>:5672"
}
}
}

Note: The region server name can be either obtained from /root/openrc file on the region server (check for setting of environment variable NOVA_REGION_NAME) or by executing the command "keystone endpoint-list" on Central Server 2.

** consumer_connection_timeout_sec - This parameters described how long IWD will wait for any notification from single region before it will try to reastablish connection to notification server. Normally OpenStack (with default configuration) is sending update about vm at least once each hour. If IWD won't get any notification for time described in parameter, it will drop the notification connection and try to reastablish new one

** consumer_error_delay_sec - Time between attempts to establish connection to notification service (in case of connection failure)


(6.3) Ensure CS-3 can use "telnet $region_server_ip 5672" to connect to region server

Here need to ensure each region server have enable the port 5672 access to the cs-3.
There are two ways to enable region server port 5672 access to the cs-3.

method 1:
On region server:
vim /etc/sysconfig/iptables

add below two lines, right before the iprules of port 5672

-A INPUT -s $cs-3_ip/32 -p udp -m udp --dport 5671:5672 -j ACCEPT
-A INPUT -s $cs-3_ip/32 -p tcp -m tcp --dport 5671:5672 -j ACCEPT

Then run 'service iptables restart'
Try command "telnet $region_server_ip 5672" on cs-3. If it can connect to the region server, then you can restart iwd service.

method 2:
On region server:
iptables-save >/tmp/iptables.save

Then modify the file /tmp/iptables.save and add two lines like below:

-A INPUT -j nova-api-INPUT <---right after this line add below two lines.
-A INPUT -s $cs-3_ip/32 -p udp -m udp --dport 5671:5672 -j ACCEPT
-A INPUT -s $cs-3_ip/32 -p tcp -m tcp --dport 5671:5672 -j ACCEPT

Carry out the command to restore iptables:
iptables-restore < /tmp/iptables.save

Then restart iptables.

(6.4) Restart IWD service

On CS-3, use 'service iwd restart' to restart iwd service and wait about 15min for its initialization.

(6.5) Try command "telnet $region_server_ip 5672" on cs-3. If telnet is not installed on cs-3, please run 'yum install telnet' to install it.

You should get a message on the shell like this:

Trying $region_server_ip...
Connected to $region_server_ip.
Escape character is '^]'.


15) Apply fix for PSIRT 1838

Note: This fix is already included in iFix5 (2.3.0.1-CSI-ISCO-IF0005). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0005 to your SCO environment.

(1) copy the file rtc-184008-keystone-11c387264-ifix-el6.tar.gz to a directory of your choice <your_dir> on Central Server 2.

(2) extract rtc-184008-keystone-11c387264-ifix-el6.tar.gz by following command:

cd <your_dir>; tar xvf rtc-184008-keystone-11c387264-ifix-el6.tar.gz

(3) backup the keystone configure file /etc/keystone

(4) install keystone rpms with command:

cd noarch; rpm -Uvh openstack-keystone-2013.1.5.1-201408010306.ibm.13.noarch.rpm python-keystone-2013.1.5.1-201408010306.ibm.13.noarch.rpm

(5) check installed keystone version:

rpm -qa |grep keystone


16) Apply fix for PSIRT 1876

Note: This fix is already included in iFix5 (2.3.0.1-CSI-ISCO-IF0005). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0005 to your SCO environment.

(1) copy PSIRT1876.patch to a directory of your choice <your_dir> on Central Server 2.

(2) get python path with command: python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

(3) backup file $python_path/keystone/token/controllers.py

(4) patch keystone with following command:

cd $python_path; patch -p1 -N -f < <your_dir>/PSIRT1876.patch


17) Apply fix for APAR ZZ00267

Note: This fix is already included in iFix5 (2.3.0.1-CSI-ISCO-IF0005). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0005 to your SCO environment.

(1) copy keystone_zz00267.patch to a directory of your choice <your_dir> on Central Server 2.

(2) get python path with command: python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

(3) backup file: mv $python_path/keystone/middleware/ldapauth.py /tmp

(4) patch keystone with following command:

cd $python_path; patch -p1 -N -f < <your_dir>/keystone_zz00267.patch


18) Apply fix for keystone multi-worker support

Note: This fix is already included in iFix5 (2.3.0.1-CSI-ISCO-IF0005). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0005 to your SCO environment.

(1) copy 0001-enable-multiple-keystone-all-worker-processes.patch to a directory of your choice <your_dir> on Central Server 2.

(2) get python path with command: python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

(3) Backup below files before applying the fix.

/usr/bin/keystone-all
$python_path/keystone/common/config.py
$python_path/keystone/common/wsgi.py

(4) Stop the keystone service using command 'service openstack-keystone stop'.

(5) Apply the fix

cd $python_path
patch -p1 < <your_dir>/0001-enable-multiple-keystone-all-worker-processes.patch

You can get the output like:
... ...
|Change-Id: If74f13bc2898e880649ee809967f5b5859b793c6
|---
| bin/keystone-all | 37 ++-
| etc/keystone.conf.sample | 8 +
| keystone/common/config.py | 2 +
| keystone/common/wsgi.py | 31 ++-
| keystone/openstack/common/loopingcall.py | 148 ++++++++++
| keystone/openstack/common/service.py | 446 ++++++++++++++++++++++++++++++
| keystone/openstack/common/threadgroup.py | 122 ++++++++
| 7 files changed, 785 insertions(+), 9 deletions(-)
| create mode 100644 keystone/openstack/common/loopingcall.py
| create mode 100644 keystone/openstack/common/service.py
| create mode 100644 keystone/openstack/common/threadgroup.py
|
|diff --git a/bin/keystone-all b/bin/keystone-all
|index 2fdc8c7..a28f31c 100755
|--- a/bin/keystone-all
|+++ b/bin/keystone-all
--------------------------
File to patch: /usr/bin/keystone-all <======== Note: here you need to input the file path here.
patching file /usr/bin/keystone-all
can't find file to patch at input line 159
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/etc/keystone.conf.sample b/etc/keystone.conf.sample
|index 9e66eb6..b70edbc 100644
|--- a/etc/keystone.conf.sample
|+++ b/etc/keystone.conf.sample
--------------------------
File to patch: <======== Note: here you need to click 'Enter'
Skip this patch? [y] y <======== Note: here you need to input 'y'
Skipping patch.
1 out of 1 hunk ignored
patching file keystone/common/config.py
patching file keystone/common/wsgi.py
patching file keystone/openstack/common/loopingcall.py
patching file keystone/openstack/common/service.py
patching file keystone/openstack/common/threadgroup.py

(6) Modify the file /etc/keystone/keystone.conf and specify the number of workers you want.

Typically these would be set to the number of CPUs.

For example:
[DEFAULT]
....
public_workers = 2
admin_workers = 2
.....

(7) start keystone service using 'service openstack-keystone start'

And ensure it is up and running with 'service openstack-keystone status'
Also, you can check the number of keystone process with 'ps -ef | grep keystone'


19) Apply fix for Iaasgateway cluster support

Note: This is an optional step which is intended for large environments.

This fix is already included in iFix5 (2.3.0.1-CSI-ISCO-IF0005). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0005 to your SCO environment.

Be aware of the additional step "(4) adapt SCOrchestrator configuration to enable Iaasgateway cluster management" not described in the README 2.3.0.1-CSI-ISCO-IF0005.

Refer to "IBM SmartCloud Orchestrator Version 2.3: Capacity Planning, Performance, and Management Guide" mentioned in "VI. ADDITIONAL INFORMATION" for considerations regarding capacity planning, performance optimization and management best practices to achieve service stability.

(1) copy iaasgateway-2013.1-1.1.4.ibm.201409112157.noarch.rpm to a directory of your choice <your_dir> on Central Server 2.

(2) backup below file before applying the fix.

/etc/iaasgateway/iaasgateway.conf

(3) applying the fix.

(3.1) prepare http server as loadbalancer

- Check if there is already a http server on Central Server 2

service httpd status

If there is already a http server, stop it with below command:

service httpd stop

If there is no http server installed, use below command to install one:

yum install httpd

- Update httpd.conf with loadbalancer configuration

Modify the file /etc/httpd/conf/httpd.conf with below two changes:

(1) update listen port to gateway port

# Listen 80
Listen 9973

(2) append loadbalancer configuration to the end the file

<VirtualHost *:9973>
ProxyRequests off
<Proxy balancer://mycluster>
# three node gateway cluster
BalancerMember http://127.0.0.1:12001
BalancerMember http://127.0.0.1:12002
BalancerMember http://127.0.0.1:12003

Order Deny,Allow
Deny from none
Allow from all

ProxySet lbmethod=byrequests
</Proxy>

# path of requests to balance "/" -> everything
ProxyPass / balancer://mycluster/
</VirtualHost>

(3.2) patch the iaasgateway for clustering

cd <your_dir>
rpm -Uvh iaasgateway-2013.1-1.1.3.ibm.201408250136.noarch.rpm

Please ignore the warning message "/etc/iaasgateway/iaasgateway.conf created as /etc/iaasgateway/iaasgateway.conf.rpmnew"

(3.3) prepare configure file for cluster members

Perform below commands:

cd /etc/iaasgateway/
cp iaasgateway.conf iaasgateway00.conf
vi iaasgateway00.conf

#It should look like below before applying this fix:
[service]
iaasgateway_listen = <central-server-2-ip>
iaasgateway_listen_port = 9973

#Update it to:
iaasgateway_listen = 127.0.0.1
iaasgateway_listen_port = 1200X
iaasgateway_user_entry = <central-server-2-ip>
iaasgateway_user_entry_port = 9973

# copy configure files and update port
cp iaasgateway00.conf iaasgateway01.conf
sed -i 's/1200X/12001/' iaasgateway01.conf
cp iaasgateway00.conf iaasgateway02.conf
sed -i 's/1200X/12002/' iaasgateway02.conf
cp iaasgateway00.conf iaasgateway03.conf
sed -i 's/1200X/12003/' iaasgateway03.conf

(3.4) prepare init scripts and update configure file

cd /etc/init.d/
cp openstack-iaasgateway openstack-iaasgateway01
cp openstack-iaasgateway openstack-iaasgateway02
cp openstack-iaasgateway openstack-iaasgateway03
sed -i 's/prog=openstack-iaasgateway/prog=openstack-iaasgateway01/' openstack-iaasgateway01
sed -i 's/iaasgateway.conf/iaasgateway01.conf/' openstack-iaasgateway01
sed -i 's/prog=openstack-iaasgateway/prog=openstack-iaasgateway02/' openstack-iaasgateway02
sed -i 's/iaasgateway.conf/iaasgateway02.conf/' openstack-iaasgateway02
sed -i 's/prog=openstack-iaasgateway/prog=openstack-iaasgateway03/' openstack-iaasgateway03
sed -i 's/iaasgateway.conf/iaasgateway03.conf/' openstack-iaasgateway03

(3.5) startup of cluster

Perform below commands to start iaasgateway cluster service

service openstack-iaasgateway stop
Stopping openstack-iaasgateway: [ OK ]

service openstack-iaasgateway01 start
Starting openstack-iaasgateway01: [ OK ]

service openstack-iaasgateway02 start
Starting openstack-iaasgateway02: [ OK ]

service openstack-iaasgateway03 start
Starting openstack-iaasgateway03: [ OK ]

service httpd start
Starting httpd: [ OK ]

(3.6) check the iaasgateway service status

- try to open below link in browser and it should be the same as it was before applying this fix.
http://<central-server-2-ip>:9973/providers

- check listening ports with below command:
# netstat -nap | grep 1200 | grep LISTEN
tcp 0 0 127.0.0.1:12001 0.0.0.0:* LISTEN 7269/python
tcp 0 0 127.0.0.1:12002 0.0.0.0:* LISTEN 7286/python
tcp 0 0 127.0.0.1:12003 0.0.0.0:* LISTEN 7303/python

- check loadbalancer listening:
# netstat -nap | grep 9973 | grep LISTEN
tcp 0 0 :::9973 :::* LISTEN 7321/httpd

- try to login SCO UI

(4) adapt SCOrchestrator configuration to enable Iaasgateway cluster management

(4.1) adapt SCOEnvironment.xml in directory /iaas/scorchestrator/ on Central Server 1

- Replace line 7 (last line below) by line 7 to 10 (first four lines below) in SCOEnvironment.xml and save your changes

7,10c7
< <component name="openstack-iaasgateway01"/>
< <component name="openstack-iaasgateway02"/>
< <component name="openstack-iaasgateway03"/>
< <component name="httpd"/>
---
> <component name="openstack-iaasgateway"/>

(4.2) adapt SCOComponents.xml in directory /iaas/scorchestrator/ on Central Server 1

- Replace line 27 (last line below) by line 27 to 30 (first four lines below) in SCOComponents.xml and save your changes

27,30c27
< <component name="openstack-iaasgateway01" openstackService="true" scriptName="openstack-servicectrl.sh" workdir="/tmp" startPrio="240" stopPrio="60"/>
< <component name="openstack-iaasgateway02" openstackService="true" scriptName="openstack-servicectrl.sh" workdir="/tmp" startPrio="240" stopPrio="60"/>
< <component name="openstack-iaasgateway03" openstackService="true" scriptName="openstack-servicectrl.sh" workdir="/tmp" startPrio="240" stopPrio="60"/>
< <component name="httpd" openstackService="true" scriptName="openstack-servicectrl.sh" workdir="/tmp" startPrio="240" stopPrio="60"/>
---
> <component name="openstack-iaasgateway" openstackService="true" scriptName="openstack-servicectrl.sh" workdir="/tmp" startPrio="240" stopPrio="60"/>

(4.3) run SCOrchestrator from directory /iaas/scorchestrator/ and check if the iaasgateway cluster services are listed/managed by the script

[root@sco-cs-1 scorchestrator]# ./SCOrchestrator.py -p openstack-iaasgateway,httpd
===>>> Collecting Status for Smart Cloud Orchestrator
===>>> Please wait ======>>>>>>

Component Hostname Status
------------------------------------------------------------------
httpd 172.17.58.212 online
openstack-iaasgateway01 172.17.58.212 online
openstack-iaasgateway02 172.17.58.212 online
openstack-iaasgateway03 172.17.58.212 online

===>>> Status Smart Cloud Orchestrator complete

(4.4) Enable httpd and iaasgateway cluster to be started during boot of Central Server 2

Execute following commands:

[root@sco-cs-2 scorchestrator]# chkconfig openstack-iaasgateway off
[root@sco-cs-2 scorchestrator]# chkconfig openstack-iaasgateway01 on
[root@sco-cs-2 scorchestrator]# chkconfig openstack-iaasgateway02 on
[root@sco-cs-2 scorchestrator]# chkconfig openstack-iaasgateway03 on
[root@sco-cs-2 scorchestrator]# chkconfig httpd on



20) Apply fix for the DB2 security vulnerabilities CVE-2013-6747 and CVE-2014-0963.

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

(1) Downloading the DB2 fix pack

In order to address security vulnerabilities CVE-2013-6747 and CVE-2014-0963, the steps described in security bulletin here http://www.ibm.com/support/docview.wss?uid=swg21675724 should be implemented.

Clicking the above link brings you to:
"Security Bulletin: Security vulnerabilities have been identified in IBM DB2 shipped with SmartCloud Orchestrator (CVE-2013-6747, CVE-2014-0963)"

From here click:
"IBM DB2 is impacted by multiple TLS/SSL security vulnerabilities (CVE-2013-6747, CVE-2014-0963)"
Which brings you to:
http://www.ibm.com/support/docview.wss?uid=swg21671732

From here click:
"http://www.ibm.com/support/docview.wss?uid=swg24037557"
Note: This is for version V10.1 FP3a - if you have already upgraded DB2, select the version that matches your environment.
Which brings you to:
"DB2 Version 10.1 Fix Pack 3a for Linux, UNIX, and Windows"

Under the section "Download fix pack images per operating system" expand "Linux 64-bit, x86-64 on AMD64 and Intel EM64T"
Click "DB2 Server Fix Pack"
This brings you to Fix Central and "interim fix: DB2-linuxx64-server-10.1.0.3-FP003a"
Check the box and click Continue
Login to FC if prompted
Under "Download files using https" you will be able to download:
v10.1fp3a_linuxx64_server.tar.gz (733 MB)

(2) Installing the DB2 fix pack on Central Server 1 and each Region Servers (if shared DB is not used)

Ensure the SCO processes have been stopped using SCOrchestrator.py stop as outlined in section III step 1 above
Note: The output of the command /iaas/scorchestrator/SCOrchestrator.py will list the servers that are running DB2. Update each of these servers, starting with central-server-1 and then moving on to each region server.

Perform following steps to install the DB2 fix.

(2.1) Copy the fix pack to central server 1 to a folder of your choice <db2fixpack>

(2.2) As user db2inst1 stop DB2 and DAS:

db2stop
/opt/ibm/db2/v10.1/das/bin/db2admin stop

(2.3) As user root extract and install the DB2 fix pack

cd <db2fixpack>
tar zxvf v10.1fp3a_linuxx64_server.tar.gz
cd server
./installFixPack -n -b /opt/ibm/db2/v10.1 -f db2lib

Please wait for the install to complete.

(2.4) Re-run the install command to verify the fix pack has been applied

./installFixPack -n -b /opt/ibm/db2/v10.1/ -f db2lib

Perform the steps described under (2) on each Region Server which uses standalone DB2.


21) Apply fix for APAR ZZ00240.

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.


Step (11.1). Copy the folder 2.3.0.1-CSI-ISCO-IF0006/2.3.0.1-CSI-ISCO-IF0005/2.3.0.1-CSI-ISCO-IF0004/2.3.0.1-CSI-ISCO-IF0003/ZZ00240 to a directory of your choice where the BPM brower is being run from.

Step (11.2). Import toolkit SCOrchestrator_Toolkit provided by this fix

Logon to IBM Process Designer with admin/$your_smartcloud_password

In Process Center, select tab Toolkits and click on toolkit SCOrchestrator_Toolkit

Check that SCOrchestrator_Toolkit version 2301_20140213 is the current version
(top in the stack below "Current")

Click on tab Toolkits and then on Import Toolkit

In the Import Toolkit dialog, click on Browse... and select 00_SCOrchestrator_Toolkit_2301_20140403_ifix1.twx
provided with this fix and click OK

Click the Import button in dialog Import Toolkit - Will be imported - SCOrchestrator_Toolkit-2301-20140403_ifix1
and wait for the import to be finished

Click on toolkit SCOrchestrator_Toolkit

Check that SCOrchestrator_Toolkit version 2301_20140403_ifix1 is the current version
(top in the stack below "Current")


Step (11.3). Upgrade dependency of toolkit SCOrchestrator_Scripting_Utilities_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Scripting_Utilities_Toolkit

Check that SCOrchestrator_Scripting_Utilities_Toolkit version 2301_20140226 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1


Step (11.4). Create new snapshot of toolkit SCOrchestrator_Scripting_Utilities_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Scripting_Utilities_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that SCOrchestrator_Scripting_Utilities_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (11.5). Upgrade dependency of toolkit SCOrchestrator_Support_IaaS_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_IaaS_Toolkit

Check that SCOrchestrator_Support_IaaS_Toolkit version 2301_20140227 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1


Step (11.6). Create new snapshot of toolkit SCOrchestrator_Support_IaaS_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_IaaS_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that SCOrchestrator_Support_IaaS_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (11.7). Upgrade dependency of toolkit SCOrchestrator_Support_vSys_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_vSys_Toolkit

Check that SCOrchestrator_Support_vSys_Toolkit version 2301_20140310 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_IaaS_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (11.8). Create new snapshot of toolkit SCOrchestrator_Support_vSys_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_vSys_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that SCOrchestrator_Support_vSys_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (11.9). Upgrade dependencies of toolkit TivSAM_Integration_Toolkit

In Process Center, click on Toolkits and click on TivSAM_Integration_Toolkit

Check that TivSAM_Integration_Toolkit version 2301_20140305 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Scripting_Utilities_Toolkit (2301_20140226) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (11.10). Create new snapshot of toolkit TivSAM_Integration_Toolkit

In Process Center, click on Toolkits and click on TivSAM_Integration_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that TivSAM_Integration_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (11.11). Upgrade dependencies of process app Sample_Support_Iaas_ProcessApp

In Process Center, click on Process Apps and click on Sample_Support_Iaas_ProcessApp

Check that Sample_Support_Iaas_ProcessApp version 2301_20140305 is the current version
(top in the stack below "Current")

Click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_IaaS_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (11.12). Upgrade dependencies of process app Sample_Support_vSys_ProcessApp

In Process Center, click on Process Apps and click on Sample_Support_vSys_ProcessApp

Check that Sample_Support_vSys_ProcessApp version 2301_20140310 is the current version
(top in the stack below "Current")

Click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_IaaS_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_vSys_Toolkit (2301_20140310) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (11.13). Upgrade dependencies of process app TSAMITK_SampleApp

In Process Center, click on Process Apps and click on TSAMITK_SampleApp

Check that TSAMITK_SampleApp version 2301_20140305 is the current version
(top in the stack below "Current")

Click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Scripting_Utilities_Toolkit (2301_20140226) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Integration_Toolkit (2301_20140305) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001

Please repeat the above described procedure for every toolkit or process application in your process centre.

Step (11.14). Upgrade REST endpoint in database

Logon to Central-Server-1

su - db2inst1
db2 connect to BPMDB
db2 set schema bpmuser
db2 "select guid,substr(default_value,1,29) as default_value,last_modified from lsw_env_var where name='restEndpoint'"

You should see one entry where default value is equals to https://localhost

Update this entry:
db2 "update LSW_ENV_VAR set default_value='https://<Central-Server-3-IP>' where name = 'restEndpoint' and default_value='https://localhost'&quot;

Double-check that the update was successful:

db2 "select guid,substr(default_value,1,29) as default_value,last_modified from lsw_env_var where name='restEndpoint'"

db2 commit
db2 connect reset

Stop/Start the BPM admin server on Central-Server-4
/opt/ibm/BPM/v8.5/profiles/Node1Profile/bin/stopServer.sh SingleClusterMember1
/opt/ibm/BPM/v8.5/profiles/Node1Profile/bin/startServer.sh SingleClusterMember1


22) Apply fix for APAR ZZ00256

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

If your SCO server OS locale is non-English, please perform the following steps to return the correct status of openstack services to SAAM.

On each of the SCO servers (Central-servers, region-servers and KVM compute nodes), find the file 'openstack-servicectrl.sh' in '/home/saam/' or '/home/ <yourmechid>/root/'.

(1) make a backup of openstack-servicectrl.sh

(2) edit 'openstack-servicectrl.sh',

Modify the line
CTRL_CMD="/sbin/service $SERVICE_ID"

to
CTRL_CMD="/etc/init.d/${SERVICE_ID}"
LANG=C

(3) check the output of openstack-servicectrl.sh and ensure it can get the correct status of openstack services.

For example, on central-server 2 run:

$ LANG=C /etc/init.d/openstack-keystone status
keystone (pid 2817) is running...

$ ./openstack-servicectrl.sh openstack-keystone status
root: openstack-servicectrl.sh (28621): openstack-keystone status monitor detected openstack-keystone online.


23) Apply fix for APAR SE58688 - KVM nodes going offline

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

KVM compute nodes are going offline and can't deply any new patterns and VMs.
This is a bug of openstack https://bugs.launchpad.net/oslo-incubator/+bug/1211338.

On each kvm region server and compute node, backup the file /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py
Then change it as below.

find the words:
{"type": "Direct"}

modify it to:
{"type": "direct"}


24) Apply fix for ZZ00242 on Central Server 1

Note: This fix is already included in iFix3 (2.3.0.1-CSI-ISCO-IF0003). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0003 or a higher IFIX version to your SCO environment.

(1) backup file config_network.sh in $your_sco_install_media/installer/scripts/

(2) Update config_network.sh with the one in folder 2.3.0.1-CSI-ISCO-IF0003/ZZ00242


25) (Optional) Apply fix for APAR SE59801

Note: This fix is already included in iFix5 (2.3.0.1-CSI-ISCO-IF0005). Ignore this part if you have already applied the fix as part of the installation of 2.3.0.1-CSI-ISCO-IF0005 to your SCO environment.

Note: This fix can only be applied to the vmware region server which attempts to connect to a vCenter with resource pools defined with similar names but with a difference in upper/lower case. Ignore this fix if your region server don't have the above issue.

On the vmware region server:

(1) check smartcloud version: rpm -qa |grep smartcloud

If your smartcoud is 'smartcloud-2013.1-1.1.3.ibm.xxxxxxx.noarch.rpm', please use fix
rpm -Uvh smartcloud-2013.1-1.1.3.ibm.201407310025.noarch.rpm

If your smartcoud is 'smartcloud-2013.1-1.1.4.ibm.xxxxxxx.noarch.rpm', please use fix
rpm -Uvh smartcloud-2013.1-1.1.4.ibm.201407310043.noarch.rpm

(2) Modify the configure file

step 1. Ensure there is no VMs under resource pools before enable the setting in step 2. Otherwise, OpenStack will fail to manage those vms after applying this fix.

step 2. Add the following line into /etc/nova/smartcloud.conf after install the ifix:
filter_out_cloud_resource_pool=True

(3) restart openstack-smartcloud service and ensure it is up and running

service openstack-smartcloud restart
service openstack-smartcloud status


26) Apply fix for APAR ZZ00277, ZZ00293 (When using LDAP user filter, authentication fails ...)

Note: This fix is applied on Central Server 2.

(1) Make a backup copy of file /usr/lib/python2.6/site-packages/keystone/middleware/ldapauth.py

eg scp -p root@cs-2:/usr/lib/python2.6/site-packages/keystone/middleware/ldapauth.py <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/

(2) Update ldapauth.py on Central-Server-2

eg scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/ldapauth.py root@cs-2:/usr/lib/python2.6/site-packages/keystone/middleware/ldapauth.py

(3) Change ldapauth.py permissions to 644 on Central-Server-2

eg chmod 644 /usr/lib/python2.6/site-packages/keystone/middleware/ldapauth.py


27) Apply fix for APAR IT05168 (After deploying SCO 2.3 IFIX5 seeing the hypervisor cannot be reached in all cloud groups)

Note: This fix is applied on Central Server 2.

(1) Unzip keystone_memcache.zip

eg unzip /data/2.3.0.1-CSI-ISCO-IF0006/installfiles/keystone_memcache.zip

(2) Make backup copies of the following files on Central Server 2

keystone/auth/controllers.py keystone/auth/token_factory.py keystone/common/logging.py keystone/token/controllers.py keystone/token/core.py keystone/token/backends/memcache.py

eg scp -p root@cs-2:/usr/lib/python2.6/site-packages/keystone/auth/controllers.py <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/
scp -p root@cs-2:/usr/lib/python2.6/site-packages/keystone/auth/token_factory.py <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/
scp -p root@cs-2:/usr/lib/python2.6/site-packages/keystone/common/logging.py <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/
scp -p root@cs-2:/usr/lib/python2.6/site-packages/keystone/token/controllers.py <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/
scp -p root@cs-2:/usr/lib/python2.6/site-packages/keystone/token/core.py <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/
scp -p root@cs-2:/usr/lib/python2.6/site-packages/keystone/token/backends/memcache.py <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/

(3) Update the following files on Central Server 2

keystone/auth/controllers.py keystone/auth/token_factory.py keystone/common/logging.py keystone/token/controllers.py keystone/token/core.py keystone/token/backends/memcache.py

eg scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/keystone/auth/controllers.py root@cs-2:/usr/lib/python2.6/site-packages/keystone/auth/
scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/keystone/auth/token_factory.py root@cs-2:/usr/lib/python2.6/site-packages/keystone/auth/
scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/keystone/common/logging.py root@cs-2:/usr/lib/python2.6/site-packages/keystone/common/
scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/keystone/token/controllers.py root@cs-2:/usr/lib/python2.6/site-packages/keystone/token/
scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/keystone/token/core.py root@cs-2:/usr/lib/python2.6/site-packages/keystone/token/
scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/keystone/token/backends/memcache.py root@cs-2:/usr/lib/python2.6/site-packages/keystone/token/backends/

(4) Change file permission on Central Server 2

eg cd /usr/lib/python2.6/site-packages
chmod a+r keystone/auth/controllers.py keystone/auth/token_factory.py keystone/common/logging.py keystone/token/controllers.py keystone/token/core.py keystone/token/backends/memcache.py

(5) Restart openstack-keystone service

service openstack-keystone restart

(6) Check that openstack-keystone is up and running

service openstack-keystone status


27) Apply fix for APAR ZZ00282 (Amazon EC2 flavor list does not match the PCG Flavor list for Amazon EC2)

Note: This fix is applied on Central Server 2.

(1) Make a backup copy of HybridCSB-API.war on Central Server 2

eg scp -p root@cs-2:/opt/ibm/pcg/lib/HybridCSB-API.war <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/

(2) Update HybridCSB-API.war on Central-Server-2

eg scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/HybridCSB-API.war root@cs-2:/opt/ibm/pcg/lib/HybridCSB-API.war

(3) Change HybridCSB-API.war permissions to 750 on Central-Server-2

eg chmod 750 /opt/ibm/pcg/lib/HybridCSB-API.war

(4) Restart the Public Cloud Gateway and IAAS server on Central Server 2:

service pcg restart
service openstack-iaasgateway restart


28) Apply fix for APAR ZZ00298 (Security: Scan UI found Parameter Value Overflow - instance.list.expanded)

Note: This fix is applied on Central Server 3.

(1) Make a backup copy of config.ini on Central Server 3

eg scp -p root@cs-3:/opt/ibm/ccs/scui/etc/config.ini <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/

(2) Copy n3.app_1.0.0.20141021-1433.jar and n3.orchestrator.app_1.0.0.20141021-1433.jar to Central-Server-3

eg scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/n3.app_1.0.0.20141021-1433.jar root@cs-3:/opt/ibm/ccs/scui/lib/
scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/n3.orchestrator.app_1.0.0.20141021-1433.jar root@cs-3:/opt/ibm/ccs/scui/lib/

(3) Update config.ini on Central-Server-3

Edit /opt/ibm/ccs/scui/etc/config.ini and replace:

n3.app_1.0.0.<version>.jar with n3.app_1.0.0.20141021-1433.jar
n3.orchestrator.app_1.0.0.<version>.jar with n3.orchestrator.app_1.0.0.20141021-1433.jar

(4) Restart SCUI on Central Server 3

service scui restart


29) Apply fix for APAR ZZ00295 (Requests aren't shown again on IWD after SCO restart)

Note: This fix is applied on Central Server 3.

(1) Make a backup copy of plugin.com.ibm.orchestrator.rest-1.0.1.1.jar on Central Server 3

eg scp -p root@cs-3:/drouter/ramdisk2/mnt/raid-volume/raid0/usr/servers/kernelservices/plugins/bundles/com.ibm.orchestrator.task/1.0.1.1/plugin.com.ibm.orchestrator.rest-1.0.1.1.jar \
<cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/

(2) Update plugin.com.ibm.orchestrator.rest-1.0.1.1.jar on Central-Server-3

eg scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/plugin.com.ibm.orchestrator.rest-1.0.1.1.jar \
root@cs-3:/drouter/ramdisk2/mnt/raid-volume/raid0/usr/servers/kernelservices/plugins/bundles/com.ibm.orchestrator.task/1.0.1.1/

(3) Restart IWD services on Central Server 3

service iwd restart


30) Apply fix for APAR ZZ00292 (Exposed Workflows in BPM REST causing 2min respnse)

Note: This fix is applied on Central Server 3 and Central Server 4.

(1) Install BPM fixes using installation manager on Central Server 4

(1.1) Stop BPM server on Central Server 4

Stop BPM using the SCOrchestrator script on Central Server 1:
/iaas/scorchestrator/SCOrchestrator.py --stop -p bpm

(1.2) Download additional BPM fix for APAR JR51814

Note: This is an optional step for applying fix for APAR ZZ00292.

Login to "IBM Support: Fix Central" (FC) and download interim fix "8.5.0.0-WS-BPM-IFJR51814" for APAR JR51814 for BPM 8.5.0.0 if officially available from FC

Please select "IBM Business Process Manager Standard" as product selector, "8.5.0.0" as installed version and "Linux 64-bit,x86_64" as platform.
Select "Browse for fixes" and from the list of fixes select/download interim fix: 8.5.0.0-WS-BPM-IFJR51814.

When downloaded, copy 8.5.0.0-WS-BPM-IFJR51814.zip to the Central Server 1, in the ifix directory <your_dir>/2.3.0.1-CSI-ISCO-IF0006/installfiles.

(1.3) Transfer BPM fixes to Central Server 4

eg scp scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/8.5.0.0-WS-BPM-IFJR47778.zip root@cs-4:root@cs4:/tmp
scp scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/8.5.0.0-WS-BPM-IFJR47937.zip root@cs-4:root@cs4:/tmp
scp scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/8.5.0.0-WS-BPM-IFJR48541.zip root@cs-4:root@cs4:/tmp
scp scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/8.5.0.0-WS-BPM-IFJR48570.zip root@cs-4:root@cs4:/tmp
scp scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/8.5.0.0-WS-BPM-IFJR48704.zip root@cs-4:root@cs4:/tmp
scp scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/8.5.0.0-WS-BPM-IFJR49864.zip root@cs-4:root@cs4:/tmp
scp scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/8.5.0.0-WS-BPM-IFJR51596.zip root@cs-4:root@cs4:/tmp
scp scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/8.5.0.0-WS-BPM-IFJR51814.zip root@cs-4:root@cs4:/tmp

(1.3) Unzip BPM fixes, prepare intallation manager repository on Central Server 4

eg cd /tmp
mkdir /tmp/JR47778
cd /tmp/JR47778
unzip ../8.5.0.0-WS-BPM-IFJR47778.zip

Repeat this sequence for each BPM fix

(1.4) Install BPM fixes using installation manager on Central Server 4

eg cd /opt/IBM/InstallationManager/eclipse/tools
./imcl install 8.5.0.0-WS-BPM-IFJR47778 -repositories /tmp/JR47778 -installationDirectory /opt/ibm/BPM/v8.5
./imcl install 8.5.0.0-WS-BPM-IFJR48541 -repositories /tmp/JR48541 -installationDirectory /opt/ibm/BPM/v8.5
./imcl install 8.5.0.0-WS-BPM-IFJR48704 -repositories /tmp/JR48704 -installationDirectory /opt/ibm/BPM/v8.5
./imcl install 8.5.0.0-WS-BPM-IFJR49864 -repositories /tmp/JR49864 -installationDirectory /opt/ibm/BPM/v8.5
./imcl install 8.5.0.0-WS-BPM-IFJR47937 -repositories /tmp/JR47937 -installationDirectory /opt/ibm/BPM/v8.5
./imcl install 8.5.0.0-WS-BPM-IFJR48570 -repositories /tmp/JR48570 -installationDirectory /opt/ibm/BPM/v8.5
./imcl install 8.5.0.0-WS-BPM-IFJR51596 -repositories /tmp/JR51596 -installationDirectory /opt/ibm/BPM/v8.5
./imcl install 8.5.0.0-WS-BPM-IFJR51596 -repositories /tmp/JR51814 -installationDirectory /opt/ibm/BPM/v8.5

(1.1) Start BPM server on Central Server 4

Start BPM using the SCOrchestrator script on Central Server 1:
/iaas/scorchestrator/SCOrchestrator.py --start -p bpm

(2) Update plugin.com.ibm.orchestrator.BPMInvoker-1.0.1.1.jar on Central Server 3

(2.1) Make a backup copy of plugin.com.ibm.orchestrator.BPMInvoker-1.0.1.1.jar on Central Server 3

eg scp -p root@cs-3:/drouter/ramdisk2/mnt/raid-volume/raid0/usr/servers/kernelservices/plugins/bundles/com.ibm.orchestrator.BPMInvoker/1.0.1.1/plugin.com.ibm.orchestrator.BPMInvoker-1.0.1.1.jar \
<cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/

(2.2) Update plugin.com.ibm.orchestrator.BPMInvoker-1.0.1.1.jar on Central-Server-3

eg scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/plugin.com.ibm.orchestrator.BPMInvoker-1.0.1.1.jar \
root@cs-3:/drouter/ramdisk2/mnt/raid-volume/raid0/usr/servers/kernelservices/plugins/bundles/com.ibm.orchestrator.BPMInvoker/1.0.1.1/

(2.3) Restart IWD services on Central Server 3

service iwd restart


31) Apply fix for APAR SE60379 (SCO is provisioning VM's to local disk datastores)

Note: This fix is applied on all VMware Region Servers.

(1) Copy package 3.1.0.4-IBM-SCE-IF003-201410100432.zip to the folder $your_folder on VMware region server

(2) Extract the fix package on VMware Region Server

cd $your_folder
unzip 3.1.0.4-IBM-SCE-IF003-201410100432.zip

(3) Ensure SCE service is up and running with command service sce status, if not running start it with command service sce start

(4) Backup configure file on VMware Region Server

cp --preserve /opt/ibm/SCE31/program/skc.ini $your_folder/

(5) Install SCE ifix on VMware Region Server

Login into SmartCloud Entry console running
telnet localhost 7777

You will see a prompt like
osgi>

In this console type the following commands:

osgi> showrepos

Metadata repositories:

Artifacts repositories:

repositories:

Artifacts repositories:

file:/C:/Users/IBM_ADMIN/.eclipse/207580638/p2/org.eclipse.

equinox.p2.core/cache/

If the repository that is storing the extracted files is not available, use the addrepo command to add that repository.

osgi> addrepo file:<the absolute path when you unpacked the zip file>

osgi> SmartCloud Entry update repository added

Install the updates by using the installupdates command.

osgi> installupdates

osgi> SmartCloud Entry updates to install:

com.ibm.cfs.product 3.1.0.4-201407162230 ==> com.ibm.cfs.product 3.1.0.4-201410100430

SmartCloud Entry update done

When the update is complete, activate the changes by using the close command to end the OSGi session, then restarting SmartCloud Entry.

osgi> close

(6) Restore the original copy of /opt/ibm/SCE31/program/skc.ini on VMware Region Server

cp --preserve $your_folder/skc.ini /opt/ibm/SCE31/program/

(7) Modify file vmware.properties under path /root/.SCE31 on VMware Region Server
If vmware.properties does not exist please manually create one

Add below four parameters to vmware.properties and fill in the correct regular expression to filter all your vmware datastore

# Property name for enabling the following clone template properties (true or false, default is false)
com.ibm.cfs.cloud.vmware.enable.clone.template.properties=true

# Optional list of datastore and datastore cluster names to exclude when looking for an available datastore or
# datastore cluster. This is a comma separated list of datastore and datastore cluster names. This is optional.
com.ibm.cfs.cloud.vmware.datastore.exclude.list=<This should be a correct regular expression or blank>

# Optional list of datastore and datastore cluster names to include when looking for an available datastore or
# datastore cluster. This is a comma separated list of runningdatastore and datastore cluster names. This is optional.
com.ibm.cfs.cloud.vmware.datastore.include.list=<This should be a correct regular expression or blank>

# Optional sets the search method (text or regex) for the following properties.
# com.ibm.cfs.cloud.vmware.datastore.include.list and com.ibm.cfs.cloud.vmware.datastore.exclude.list
# 'text' mode is exact text matching and 'regex' is regular expression matching.
# This is an all or nothing setting for values in each lists
com.ibm.cfs.cloud.vmware.datastore.list.type=regex

(8) Restart SmartCloud Entry service on VMware region server

service sce restart


32) Apply fix for defect 140719 (Running nova-cloud-modify with changed VC password results in duplication of hypervisors)

Note: This fix is applied on all VMware Region Servers.

(1) Make a backup copy of nova-cloud-modify on VMware Region Server

eg scp -p root@vmwr:/opt/ibm/openstack/iaas/smartcloud/bin/nova-cloud-modify <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/

(2) Update nova-cloud-modify on VMware Region Server

eg scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/nova-cloud-modify root@vmwr:/opt/ibm/openstack/iaas/smartcloud/bin/


33) Apply fix for APAR SE60719 (Openstack nova synchronization powers off vms automatically)

Note: This fix is applied on all Region Servers and VMware Region Servers respectively.

(1) Install openstack-nova fix on all Region Servers

(1.1) Transfer openstack-nova fix rtc-190961-nova-283ad24da-ifix-el6.tgz to Region Server

eg scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/rtc-190961-nova-283ad24da-ifix-el6.tgz root@region:/tmp

(1.2) Untar openstack-nova fix rtc-190961-nova-283ad24da-ifix-el6.tgz on Region Server

eg cd /tmp
tar zxvf rtc-190961-nova-283ad24da-ifix-el6.tgz

The following RPMs will be extracted from the tgz:

openstack-nova-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-doc-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-api-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-network-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-cells-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-objectstore-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-cert-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-scheduler-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-common-2013.1.5.1-201411241107.ibm.22.noarch.rpm
python-nova-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-compute-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-conductor-2013.1.5.1-201411241107.ibm.22.noarch.rpm
openstack-nova-console-2013.1.5.1-201411241107.ibm.22.noarch.rpm

(1.3) Stop applicable nova services (to stop each service do: service <service name> stop) on Region Server

openstack-nova-api openstack-nova-objectstore openstack-nova-network openstack-nova-volume
openstack-nova-scheduler openstack-nova-cert openstack-nova-console openstack-nova-consoleauth

Make sure each service has been stopped using: service <service name> status

(1.4) Install all RPMs by means of "yum install *.rpm" (type "y" when required) on Region Server

(1.5) Check that all RPMs were successfully installed on Region Server using

rpm -qa |grep 201411241107

openstack-nova-cells-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-2013.1.5.1-201411241107.ibm.22.noarch
python-nova-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-network-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-common-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-compute-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-objectstore-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-api-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-conductor-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-scheduler-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-console-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-cert-2013.1.5.1-201411241107.ibm.22.noarch
openstack-nova-doc-2013.1.5.1-201411241107.ibm.22.noarch

(1.6) Start the nova services stopped under (1.3) (to start each service do: service <service name> start) on Region Server

Make sure each service has been started using: service <service name> status

(2) Install smartcloud fix smartcloud-2013.1-1.1.4.ibm.201412010336.noarch.rpm on all VMware Region Servers

(2.1) Transfer smartcloud fix smartcloud-2013.1-1.1.4.ibm.201412010336.noarch.rpm to VMware Region Server

eg scp -p <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/installfiles/smartcloud-2013.1-1.1.4.ibm.201412010336.noarch.rpm root@vmwr:/tmp

(2.2) Stop the nova and the smartcloud services (to stop each service do: service <service name> stop) on VMware Region Server

openstack-nova-api openstack-nova-objectstore openstack-nova-network openstack-nova-volume
openstack-nova-scheduler openstack-nova-cert openstack-nova-console openstack-nova-consoleauth
openstack-smartcloud

Make sure each service has been stopped using: service <service name> status

(2.3) Install smartcloud RPM package by means of "yum install smartcloud fix smartcloud-2013.1-1.1.4.ibm.201412010336.noarch.rpm (type "y" when required) on VMware Region Server

Make sure smartcloud RPM package was installed successfully

(2.4) Make backup copies of /etc/nova/nova.conf and /etc/nova/smartcloud.conf on VMware Region Server

eg scp -p root@vmwr:/etc/nova/nova.conf <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/
scp -p root@vmwr:/etc/nova/smartcloud.conf <cs-1>:/data/2.3.0.1-CSI-ISCO-IF0006/backups/

(2.5) Add the following property to /etc/nova/nova.conf

sync_power_state_interval = -1

eg openstack-config --set /etc/nova/nova.conf DEFAULT sync_power_state_interval -1

(2.6) Add the following property to /etc/nova/smartcloud.conf

auto_sync_data_on_start = False

eg openstack-config --set /etc/nova/smartcloud.conf DEFAULT auto_sync_data_on_start False

(2.7) Start the nova services stopped under (2.2) (to start each service do: service <service name> start) on VMware Region Server

Make sure each service has been started using: service <service name> status