Objective: Primary objective of this document is to make upgrade install  success from 5.2.X to 5.2.6.3  and run without errors.

Steps

1.Copy of 5.2. from old server to new server

2.Clone DB of 5.2 with different user name/password and Schema.

3.Change below parameters in sandbox.cfg which is pointing to new server parameters

DB_PASS=
DB_SCHEMA_OWNER=
DB_DATA=
ORA_PASS=
ORA_DATA=
DB_HOST=
ORA_HOST=
INSTALL_IP=
EXT_HOST_ADDR=
YANTRA_DB_USER=
YANTRA_DB_PASS=
YANTRA_DB_DATA=
YANTRA_DB_HOST=
DB_USER=
ORA_USER=

4.run setupfiles

5.start SI and make sure the SI52X is running successfully.

6.Stop SI

7.Upgrade JDK as per https://www.ibm.com/support/knowledgecenter/SS3JSW_5.2.0/com.ibm.help.sb2bi_install_upgrade_526.doc/_shared_files/UpgradeJDK_Windows_UNIX.html

8.Start installing SI5263

a.if IIM is used point the SI524X directory,JDK1.7 and JCE,system passphrase

b.if response file is used make sure SI524X directory,JDK1.7 and JCE is provided and where as system passphrase should be encrypted.

9.Installation is successful

10.Start SI where as SI throwing an exception as below.

java.io.IOException: java.io.IOException: Invalid keystore format
 at com.sterlingcommerce.security.keystoreprovider.SCIKeyStore$FormattedInputStream.load(SCIKeyStore.java:447)
 at com.sterlingcommerce.security.keystoreprovider.SCIKeyStore$FormattedInputStream.access$000(SCIKeyStore.java:409)
 at com.sterlingcommerce.security.keystoreprovider.SCIKeyStore.engineLoad(SCIKeyStore.java:97)
 at java.security.KeyStore.load(KeyStore.java:1238)
 at com.sterlingcommerce.security.kcapi.PrivateKeyInfo.getCertificate(PrivateKeyInfo.java:2533)
 at com.sterlingcommerce.woodstock.ops.server.PassPhrase.startup(PassPhrase.java:360)
 at com.sterlingcommerce.woodstock.ops.server.PassPhrase.main(PassPhrase.java:132)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
 at java.lang.reflect.Method.invoke(Method.java:620)
 at com.sterlingcommerce.woodstock.noapp.NoAppLoader.main(NoAppLoader.java:125)
[2017-08-04 08:20:44.639] ERRORDTL [1501849244639]java.io.IOException: java.io.IOException: Invalid keystore format
 at com.sterlingcommerce.security.keystoreprovider.SCIKeyStore$FormattedInputStream.load(SCIKeyStore.java:447)
 at com.sterlingcommerce.security.keystoreprovider.SCIKeyStore$FormattedInputStream.access$000(SCIKeyStore.java:409)
 at com.sterlingcommerce.security.keystoreprovider.SCIKeyStore.engineLoad(SCIKeyStore.java:97)
 at java.security.KeyStore.load(KeyStore.java:1238)
 at com.sterlingcommerce.security.kcapi.PrivateKeyInfo.getCertificate(PrivateKeyInfo.java:2533)
 at com.sterlingcommerce.woodstock.ops.server.PassPhrase.startup(PassPhrase.java:360)
 at com.sterlingcommerce.woodstock.ops.server.PassPhrase.main(PassPhrase.java:132)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
 at java.lang.reflect.Method.invoke(Method.java:620)
 at com.sterlingcommerce.woodstock.noapp.NoAppLoader.main(NoAppLoader.java:125)

Reason: As per the InstallService.log, Core_License.xml file is missing in the  path where sandbox.cfg has entry,It will be find error as below

Processing post service install tasks...Running...platform_ifcbase.scripts.PostServiceInstall
Licensefile to be added in ifcbase post install=installables\Core_License.xml
File or directory \si-install\Core_License.xml not found. Exiting

11.Copied Core_License.xml  in the path and installation is success.

Is that path correct? Is Core_License.xml present in the specified path out on the file system? 
If not, you'll have to fix it by updating the entry and pointing to 
Core_License.xml located in the <si-install>/properties/licensefiles 
directory.

12.Start SI and validate the version