Weekly Tips from DB2 Experts: Common techniques to troubleshoot DB2 installation/upgrade/migration problems.

Body

Are you stuck with DB2 installation/upgrade/migration?

I am listing down few common check points to start with:

1. Please look for any db2 related processes running on the system. You may run 'ps -elf | grep -i db2' on unix or look for db2* processes in task manager on windows and see what db2 processes are running. You may want to shutdown/kill the process if it is not required.
2. Please make sure *all* db2 filesystems are mounted and accessible.
3. Please make sure you have enough free space under /opt, /tmp and db2 instance's home directory. (min 500 MB is needed though I recommend up to 2 GB free space under each filesystem.)
   3a. Please make sure the system tablespace and db2 transactional logs filesystem have enough space as well.
4. If there is remote database cataloged under an instance, that may prevent you from successful instance upgrade. Please uncatalog such database entries, do upgrade and then re-catalog it.
5. Please make sure that following daemon processes/utilities/resources are shutdown before installation / upgradation / migration.
a. DB2 Fault Monitor ($DB2DIR/bin/db2fm -i <inst_name> –D)
b. DB2 License Daemon ("db2licd –end" for each instance)
c. DAS
d. IPC resources needs to be cleaned (ipclean)
e. Shared Library Clean up (applicable to AIX and command is "slibclean" as root user)
6. Run db2ckupgrade and see if all database upgrades are possible.
7. Run db2prereqchek to see if your system meets the prerequisites.


Even after doing all these checks if any of the install/upgrade/migrate commands fails/hangs, please collect the debug output as instructed below.

For installFixPack and  db2setup : Please add '-l log.out -t trace.out' to your original command.
For db2icrt, db2iupgrade, db2idrop, db2imigr : Please add '-d' option to turn on the debug mode.

The debug option creates debug files in /tmp . Please look for newly created db2* or *log* or *trc* files. These are the trace output files.
If it is a pureScale environment, please look for *GPFS*, *RSCT*, *TSA* files as well in /tmp .

Now look for words like 'Error', 'Failed', 'Not found' etc in trace output files.

In one of the case, I found this:

2586   || FilesetUtility::scanSystem 2 -DATA-  , STRING = /opt/IBM/db2/V9.1/.metadata
2587   || FilesetUtility::scanSystem 3 -DATA-  , STRING = Opendir failed.

So it is obvious that installFixpack was failing due to missing and inaccessible /opt/IBM/db2/V9.1/.metadata  . We need to see why it was inaccessble.

Suppose one of the command like (installFixPack, db2icrt etc) hangs forever and does not return to the prompt, then look for the last few lines in trace output file.

For eg:

33228  ||||||||||/ 1245630800 InstallProcess::InstallProcess ENTRY Wed Oct  8 21:05:46 2014 --  ,
33229  ||||||||||| 1245630800 InstallProcess::InstallProcess 10 -DATA-  , STRING = /opt/ibm/db2/V10.5/instance/db2icrt_local -a server -d  -s ese -u db2fenc1 db2inst1
33230  ||||||||||| 1245630800 InstallProcess::InstallProcess 10 -DATA-  , STRING = /opt/ibm/db2/V10.5/instance/db2icrt_local -a server -d  -s ese -u db2fenc1 db2inst1
33231  ||||||||||\ 1245630800 InstallProcess::InstallProcess EXIT Wed Oct  8 21:05:46 2014 --  , INT = 0
33232  ||||||||||/ 1245630800 InstallProcess::execute ENTRY Wed Oct  8 21:05:46 2014 --  ,
33233  |||||||||||/ 1245630800 InstallProcess::getHostname ENTRY Wed Oct  8 21:05:46 2014 --  ,
33234  |||||||||||\ 1245630800 InstallProcess::getHostname EXIT Wed Oct  8 21:05:46 2014 --  , INT = 0
33235  |||||||||||/ 1245630800 InstallProcess::getHostname ENTRY Wed Oct  8 21:05:46 2014 --  ,
33236  |||||||||||\ 1245630800 InstallProcess::getHostname EXIT Wed Oct  8 21:05:46 2014 --  , INT = 0
33237  |||||||||||/ 1245630800 InstallProcess::executeLocal ENTRY Wed Oct  8 21:05:46 2014 --  ,
33238  |||||||||||| 1245630800 InstallProcess::executeLocal 5 -DATA-  , STRING = /tmp/db2iocvQmaw
33239  |||||||||||| 1245630800 InstallProcess::executeLocal 10 -DATA-  , STRING = /tmp/db2ieJWOY1A
33240  ||||||||||||/ 1245630800 iGetDB2InstallPath ENTRY Wed Oct  8 21:05:46 2014 --  ,
33241  |||||||||||||/ 1245630800 FilesetUtility::getDB2InstallPath ENTRY Wed Oct  8 21:05:46 2014 --  ,
33242  |||||||||||||\ 1245630800 FilesetUtility::getDB2InstallPath EXIT Wed Oct  8 21:05:46 2014 --  , INT = 0
33243  ||||||||||||| 1245630800 iGetDB2InstallPath 40 -DATA-  , STRING = /opt/ibm/db2/V10.5
33244  ||||||||||||\ 1245630800 iGetDB2InstallPath EXIT Wed Oct  8 21:05:46 2014 --  , INT = 0
33245  ||||||||||||/ 1245630800 InstallProcess:setWorkingDir ENTRY Wed Oct  8 21:05:46 2014 --  ,
33246  ||||||||||||\ 1245630800 InstallProcess:setWorkingDir EXIT Wed Oct  8 21:05:46 2014 --  , INT = 0
33247  ||||||||||||/ 1245630800 URE::isURERunnerRoot ENTRY Wed Oct  8 21:05:46 2014 --  ,
33248  |||||||||||||/ 1245630800 URE::getURERunnerUid ENTRY Wed Oct  8 21:05:46 2014 --  ,
33249  ||||||||||||||/ 1245630800 IUser::getUID ENTRY Wed Oct  8 21:05:46 2014 --  ,
33250  |||||||||||||||/ 1245630800 UserManager::getUserID ENTRY Wed Oct  8 21:05:46 2014 --  ,
33251  |||||||||||||||\ 1245630800 UserManager::getUserID EXIT Wed Oct  8 21:05:46 2014 --  , INT = 0
33252  ||||||||||||||\ 1245630800 IUser::getUID EXIT Wed Oct  8 21:05:46 2014 --  , INT = 0
33253  |||||||||||||\ 1245630800 URE::getURERunnerUid EXIT Wed Oct  8 21:05:46 2014 --  , INT = 0
33254  ||||||||||||\ 1245630800 URE::isURERunnerRoot EXIT Wed Oct  8 21:05:46 2014 --  , UINT = 1
33255  |||||||||||| 1245630800 InstallProcess::executeLocal 40 -DATA-  , STRING = cd /opt/ibm/db2/V10.5/;
33256  |||||||||||| 1245630800 InstallProcess::executeLocal 90 -DATA-  , STRING = cd /opt/ibm/db2/V10.5/;/opt/ibm/db2/V10.5/instance/db2icrt_local -a server -d  -s ese -u db2fenc1 db2inst1 1> /tmp/db2iocvQmaw 2> /tmp/db2ieJWOY1A

So we are stuck at "db2icrt_local". I checked for files /tmp/db2iocvQmaw and /tmp/db2ieJWOY1A, both were empty.

When we ran "/opt/ibm/db2/V10.5/instance/db2icrt_local -a server -d  -s ese -u db2fenc1 db2inst1" from command line as root user, it hung as well. So we have atleast narrowed down the issue from big 'db2icrt' command to one single file - db2icrt_local .

Next step is to see what kind of file it is . If it is binary file, we can't do much. Fortunately it was a shell script.

Hence next step is to enable debug mode for shell script and rerun the command as follows:

export setopts=-x
/opt/ibm/db2/V10.5/instance/db2icrt_local -a server -d  -s ese -u db2fenc1 db2inst1 >/tmp/script.trace 2>&1

Finally '/tmp/script.trace' revealed the fact that we are getting stuck while executing 'du' command. When we run simple 'du' command from command line, it hung on RedHat server. So it is outside DB2. We engaged system admin to look at it. Once they fixed, db2icrt succeeded.

These are the common techniques which may help you to become an expert in DB2 installation/upgrade/migration.

Cheers,
Yogesh.