Automatically starting ITNM server coupled with a local DB2 NCIM database after a server reboot

Body

When you have your IBM Tivoli Network Manager (ITNM) core server and DB2 installed on the same server, AND you want to set up your
ITNM server to automatically restart upon a reboot, there are a couple of configuration items that you will need to take.

For this situation, we're going to set all three (tip, nco, and ncp) to autostart.  We call this a
single server install.  As the root user, go to the NCHOME/precision/install/scripts directory.

Run the create_all_control.sh script and use the "auto_only" option

#source /opt/IBM/tivoli/netcool/env.sh
#cd /opt/IBM/tivoli/netcool/precision/install/scripts
#./create_all_control.sh -auto_only

This will create scripts "ncp", "nco", "tip", in "/etc/init.d", create symbolic
links to the rcX.d folders and sets the run levels for these services to 3 and 5.

Check to see if these were added to autostart by using chkconfig

#chkconfig --list|grep -E "nco|tip |ncp"
nco             0:off   1:off   2:off   3:on    4:off   5:on    6:off
ncp             0:off   1:off   2:off   3:on    4:off   5:on    6:off
tip             0:off   1:off   2:off   3:on    4:off   5:on    6:off

Make sure your current runlevel is one that is set to "on"

#who -r
run-level 5  2015-04-03 21:09                   last=S

After doing the above, you are all set with automatically starting the ITNM core services, the TIP
server, and the Objectserver in a single server installation.  


Easy right?


But what about DB2?  What you will soon find out is that if you don't have DB2 automatically starting at
boot time as well, then the ITNM core server will fail to start.  ITNM requires a backend database
to be running before it will completely initiate, so if it has not completely initialized before
ITNM attempts to start, or it doesn't come up at all, then you will see that the ITNM core services
are not fully functional upon restart.

#itnm_status
OMNIbus:
      nco_pad                RUNNING  PID=3451  NCO_PA
      nco_objserv            RUNNING  PID=3479  NCOMS
      nco_p_mttrapd          RUNNING  PID=3525  NCOMS
Network Manager:
  Domain:  NCOMS
      ncp_ctrl               RUNNING  PID=6306  NCOMS
      ncp_store              RUNNING  PID=6596  NCOMS
      ncp_class              RUNNING  PID=6597  NCOMS
      ncp_model              FAILED   PID=      NCOMS
      ncp_disco              WAITING  PID=      NCOMS
      ncp_d_helpserv         RUNNING  PID=6598  NCOMS
      ncp_config             RUNNING  PID=6599  NCOMS
      ncp_poller(default)    WAITING  PID=      NCOMS
      nco_p_ncpmonitor       RUNNING  PID=6600  NCOMS
      ncp_g_event            WAITING  PID=      NCOMS
      ncp_webtool            RUNNING  PID=6603  NCOMS
      ncp_virtualdomain      WAITING  PID=      NCOMS
Tivoli Integrated Portal:
      Server                 RUNNING  PID=4302


Model's log file will show that it cannot connect to the database:


ncp_model ( IBM Tivoli Network Manager Topology Model Engine )
Copyright (C) 1997 - 2010 By IBM Corporation.  All Rights Reserved.  See product license for details.

IBM Tivoli Network Manager Version 4.1.1 (Build 11) 64 bit created by ncpbuild at 13:14:25 Wed May 21 BST 2014

Sun Apr  5 23:46:09 2015  Warning: Cannot connect to database found in file CMdlApplication.cc at line 175 - : 08001 (Error -30081): [IBM][CLI Driver] SQL30081N  A communication error has been detected. Communication protocol being used: "TCP/IP".  Communication API being used: "SOCKETS".  Location where the error was detected: "9.53.113.167".  Communication function detecting the error: "connect".  Protocol specific error code(s): "111", "*", "*".  SQLSTATE=08001
(from CNcpDB2Dbh.cc line 166) dbId: NCIM
Sun Apr  5 23:46:09 2015  Termination: Exit function called found in file CMdlApplication.cc at line 177 - Error: Failed to connect to database

ncp_model is dead.


One way to avoid this is to modify the ncp script that was previously created when the create_all_control.sh script
is ran above.  Adding this section to the script right before it attempts to start ITNM will provide a
check for the db2 database and will force the itnm initialization to wait until db2 is up and running before it allows
the ITNM core services to be started:

while true;
    do i=`ps aux|grep [d]b2sysc|wc -l`;
    if [ $i -gt 0 ]; then break;
    else echo "DB2 is not running";
    fi;
    sleep 10;
done

You will want to place this immediately above the following line in the script:

su - $user -c "${ITNM_BIN_DIR}/ncp_control.sh $1"

Note that this solution is simple and does not take into account things such as multiple DB2 instances
on the server (it only checks for existence of db2sysc running before it starts) and does not
make any provision for remote DB2 as well.  It also requires that you add autostart options for DB2 to start upon reboot as well.

See the documentation for DB2 on how to set that up.


Once this has been configured what you can expect is that your ITNM core to wait until the DB2 instance
is up and running before it attempts to initiate the services.