Topic
10 replies Latest Post - ‏2013-05-30T14:17:12Z by techFanatic
techFanatic
techFanatic
32 Posts
ACCEPTED ANSWER

Pinned topic The TBSM Data Server is unavailable, Reconnecting

‏2013-05-07T13:00:02Z |

Hi All,
When we log in to TIP using tipadmin id, we are getting the following error:
"The TBSM Data Server is unavailable, Reconnecting"
We checked logs in $TIP_HOME/profiles/TBSMProfile/logs/server1/trace.log.
Contents:

7/13 12:14:29:705 SAST] 00000016 beans         1 com.micromuse.sla.beans.DynamicAwareServiceInstanceBean addChildInternal NOTE ^Tried to add circular reference for child: CellC_RNC_KZN parent is: CellC_3GRAN_KZN^n^
[5/7/13 12:14:29:710 SAST] 00000016 beans         1 com.micromuse.sla.beans.ServiceTypeStoreBean loadInstancesAndRelationships NOTE ^println^S^STSB: Completed loadingInstances
[5/7/13 12:14:29:710 SAST] 00000016 beans         1 com.micromuse.sla.beans.RootStoreBean initialize NOTE ^^S^Singleton initialized
[5/7/13 12:14:29:881 SAST] 00000016 SystemOut     O   Sourcing startup file: /tbsmdev/opt/IBM/tivoli/tbsm/../tbsm/etc/scriptedAPIStartup.bsh

[5/7/13 12:14:30:751 SAST] 00000016 SystemOut     O   Welcome to the RAD Shell.

[5/7/13 12:14:30:752 SAST] 00000016 SystemOut     O   Enter help(); for help and exit(); to exit.

[5/7/13 12:14:30:965 SAST] 00000016 statemodel    1 com.micromuse.sla.statemodel.DeletedEventCheckerThread DeletedEventCheckerThread NOTE ^error^S^Cannot load policy Main to clear deleted events.
[5/7/13 12:14:30:965 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService startRunning NOTE ^println^S^RADService: saveStatusLocally is false
[5/7/13 12:14:30:965 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService startService NOTE ^println^S^RADService: collectMetricHistory is true
[5/7/13 12:14:30:965 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService startRunning NOTE ^println^S^starting TBSMMetricHistoryService
[5/7/13 12:14:30:966 SAST] 00000016 metrichistory 1 com.micromuse.sla.metrichistory.TBSMMetricHistoryService TBSMMetricHistoryService NOTE ^out^S^TBSMMetricHistoryService: in constructor
[5/7/13 12:14:32:051 SAST] 00000016 metrichistory 1 com.micromuse.sla.metrichistory.MetricHistoryPurge MetricHistoryPurge NOTE ^out^S^MetricHistoryPurge: in constructor
[5/7/13 12:14:32:052 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService startService NOTE ^println^S^RADService: updateMetricMarker is false
[5/7/13 12:14:32:052 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService resyncStateModelFromObjectServer NOTE ^println^S^Resyncing statemodel from ObjectServer
[5/7/13 12:14:32:052 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService resyncStateModelFromObjectServer NOTE ^println^S^RADService calling discoverObjectServerSchema()
[5/7/13 12:14:32:052 SAST] 00000016 SystemOut     O   RADService: about to discoverObjectServerSchema
[5/7/13 12:14:32:052 SAST] 0000001a metrichistory 1 com.micromuse.sla.metrichistory.MetricHistoryPurge nextPurgeTime NOTE ^^S^nextPurgeTime: 20130508 03:00:32:052 +0200
[5/7/13 12:14:32:052 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService resyncStateModelFromObjectServer ENTER^^T^Policy for discovering object server schema does not exist.  Looking for: DiscoverSchema.
                                 com.micromuse.response.common.RespException: Policy for discovering object server schema does not exist.  Looking for: DiscoverSchema.
 at com.micromuse.sla.impact.RADService.discoverObjectServerSchema(RADService.java:1025)
 at com.micromuse.sla.impact.RADService.resyncStateModelFromObjectServer(RADService.java:778)
 at com.micromuse.sla.impact.RADService.startRunning(RADService.java:508)
 at com.micromuse.response.server.ServiceManager.doAutoStartup(ServiceManager.java:307)
 at com.micromuse.response.broker.cluster.ClusterMember.convertRoleToPrimary(ClusterMember.java:1127)
 at com.micromuse.response.broker.cluster.ClusterMember.startMemberAsPartOfCluster(ClusterMember.java:485)
 at com.micromuse.response.broker.cluster.ClusterMember.<init>(ClusterMember.java:389)
 at com.micromuse.response.broker.cluster.ClusterMember.getClusterMember(ClusterMember.java:325)
 at com.micromuse.response.server.ResponseServer.<init>(ResponseServer.java:112)
 at com.micromuse.response.server.Server.start(Server.java:238)
 at com.micromuse.response.server.management.ManagedImpactServerSupport$1.run(ManagedImpactServerSupport.java:291)

[5/7/13 12:14:32:053 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService resyncStateModelFromObjectServer NOTE ^println-catch^S^Error discovering ObjectServer schema
[5/7/13 12:14:32:641 SAST] 00000016 SystemOut     O   RADService: about to resyncFromObjectServer
[5/7/13 12:14:32:641 SAST] 00000016 statemodel    1 com.micromuse.sla.statemodel.RADStateModelResyncer resyncFromObjectServer NOTE ^println^S^RADService calling resyncStateModelFromObjectServer()
[5/7/13 12:14:32:641 SAST] 00000016 statemodel    1 com.micromuse.sla.statemodel.RADStateModelResyncer resyncFromObjectServer NOTE ^println^S^RADStateModelResyncer: deleting status events in OS matching filter: Class = 12000
[5/7/13 12:14:32:644 SAST] 00000016 statemodel    1 com.micromuse.sla.statemodel.RADStateModelResyncer resyncFromObjectServer NOTE ^println^S^RADStateModelResyncer: resyncing...
[5/7/13 12:14:32:644 SAST] 00000016 statemodel    1 com.micromuse.sla.statemodel.RADStateModelResyncer resyncFromObjectServer NOTE ^out^S^Resyncing RAD state from ObjectServer.
[5/7/13 12:14:32:670 SAST] 00000016 statemodel    1 com.micromuse.sla.statemodel.RADStateModelResyncer resyncFromObjectServer NOTE ^println^S^RADStateModelResyncer: number of events to resync is 4
[5/7/13 12:14:32:670 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService resyncStateModelFromObjectServer ENTER^^T^Policy for resyncer Main does not exist.
                                 com.micromuse.response.common.RespException: Policy for resyncer Main does not exist.
 at com.micromuse.sla.statemodel.RADStateModelResyncer.getResyncPolicy(RADStateModelResyncer.java:209)
 at com.micromuse.sla.statemodel.RADStateModelResyncer.resyncFromObjectServer(RADStateModelResyncer.java:104)
 at com.micromuse.sla.impact.RADService.resyncStateModelFromObjectServer(RADService.java:803)
 at com.micromuse.sla.impact.RADService.startRunning(RADService.java:508)
 at com.micromuse.response.server.ServiceManager.doAutoStartup(ServiceManager.java:307)
 at com.micromuse.response.broker.cluster.ClusterMember.convertRoleToPrimary(ClusterMember.java:1127)
 at com.micromuse.response.broker.cluster.ClusterMember.startMemberAsPartOfCluster(ClusterMember.java:485)
 at com.micromuse.response.broker.cluster.ClusterMember.<init>(ClusterMember.java:389)
 at com.micromuse.response.broker.cluster.ClusterMember.getClusterMember(ClusterMember.java:325)
 at com.micromuse.response.server.ResponseServer.<init>(ResponseServer.java:112)
 at com.micromuse.response.server.Server.start(Server.java:238)
 at com.micromuse.response.server.management.ManagedImpactServerSupport$1.run(ManagedImpactServerSupport.java:291)

[5/7/13 12:14:32:671 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService resyncStateModelFromObjectServer NOTE ^println-catch^S^Error resyncing state from ObjectServer
[5/7/13 12:14:32:671 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService startRunning ENTER^^S^Exception occurred while starting RADService.  Server is stopping.
[5/7/13 12:14:32:671 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService startRunning NOTE ^^S^Exception message: Error resyncing state from ObjectServer
[5/7/13 12:14:32:671 SAST] 00000016 impact        1 com.micromuse.sla.impact.RADService startRunning NOTE ^^T^Error resyncing state from ObjectServer
                                 com.micromuse.response.common.RespException: Error resyncing state from ObjectServer
 at com.micromuse.sla.impact.RADService.resyncStateModelFromObjectServer(RADService.java:807)
 at com.micromuse.sla.impact.RADService.startRunning(RADService.java:508)
 at com.micromuse.response.server.ServiceManager.doAutoStartup(ServiceManager.java:307)
 at com.micromuse.response.broker.cluster.ClusterMember.convertRoleToPrimary(ClusterMember.java:1127)
 at com.micromuse.response.broker.cluster.ClusterMember.startMemberAsPartOfCluster(ClusterMember.java:485)
 at com.micromuse.response.broker.cluster.ClusterMember.<init>(ClusterMember.java:389)
 at com.micromuse.response.broker.cluster.ClusterMember.getClusterMember(ClusterMember.java:325)
 at com.micromuse.response.server.ResponseServer.<init>(ResponseServer.java:112)
 at com.micromuse.response.server.Server.start(Server.java:238)
 at com.micromuse.response.server.management.ManagedImpactServerSupport$1.run(ManagedImpactServerSupport.java:291)

[5/7/13 12:14:32:709 SAST] 0000001b TCPChannel    I   TCPC0002I: TCP Channel TCP_1 has stopped listening on host *  (IPv6) port 17315.
[5/7/13 12:14:32:740 SAST] 0000001b TCPChannel    I   TCPC0002I: TCP Channel TCP_2 has stopped listening on host *  (IPv6) port 17310.
[5/7/13 12:14:32:752 SAST] 0000001b TCPChannel    I   TCPC0002I: TCP Channel TCP_3 has stopped listening on host *  (IPv6) port 17316.
[5/7/13 12:14:32:770 SAST] 0000001b TCPChannel    I   TCPC0002I: TCP Channel TCP_4 has stopped listening on host *  (IPv6) port 17311.
[5/7/13 12:14:32:783 SAST] 0000001b StartUpServic I   STUP0013I: The Startup Beans service has stopped.
[5/7/13 12:14:32:793 SAST] 0000001b ApplicationMg A   WSVR0217I: Stopping application: ibmasyncrsp
[5/7/13 12:14:32:832 SAST] 0000001b servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [ibmasyncrsp] [/IBM_WS_SYS_RESPONSESERVLET] [rspservlet]: Destroy successful.
[5/7/13 12:14:32:860 SAST] 0000001b ApplicationMg A   WSVR0220I: Application stopped: ibmasyncrsp
[5/7/13 12:14:32:871 SAST] 0000001b ApplicationMg A   WSVR0217I: Stopping application: isclite
[5/7/13 12:14:32:906 SAST] 0000001b servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [isclite] [/ibm/console] [EventInitializer]: Destroy successful.
[5/7/13 12:14:32:940 SAST] 0000001b ApplicationMg A   WSVR0220I: Application stopped: isclite
[5/7/13 12:14:32:949 SAST] 0000001b ApplicationMg A   WSVR0217I: Stopping application: query
[5/7/13 12:14:32:963 SAST] 0000001b EJBContainerI I   WSVR0041I: Stopping EJB jar: querybean.jar
[5/7/13 12:14:32:980 SAST] 0000001b EJBContainerI I   WSVR0059I: EJB jar stopped: querybean.jar
[5/7/13 12:14:32:993 SAST] 0000001b ApplicationMg A   WSVR0220I: Application stopped: query
[5/7/13 12:14:33:003 SAST] 0000001b ApplicationMg A   WSVR0217I: Stopping application: TBSM
[5/7/13 12:14:33:034 SAST] 0000001b http          I   JAXRPC.JAXRPCSERVLET.57: JAX-RPC servlet: destroy
[5/7/13 12:14:33:045 SAST] 0000001b servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [TBSM] [/TBSMCLUSTER_TBSM_jaxrpc] [JAXRPCEndpoint]: Destroy successful.
[5/7/13 12:14:33:093 SAST] 0000001b EJBContainerI I   WSVR0041I: Stopping EJB jar: startup-ejb.jar
[5/7/13 12:14:33:123 SAST] 0000001b EJBContainerI I   WSVR0059I: EJB jar stopped: startup-ejb.jar
[5/7/13 12:14:33:148 SAST] 0000001b ApplicationMg A   WSVR0220I: Application stopped: TBSM
[5/7/13 12:14:33:159 SAST] 0000001b ApplicationMg A   WSVR0217I: Stopping application: twamarker
[5/7/13 12:14:33:180 SAST] 0000001b servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [twamarker] [/markerWeb] [MarkerMaintenanceTask]: Destroy successful.
[5/7/13 12:14:33:190 SAST] 0000001b servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [twamarker] [/markerWeb] [com_ibm_tivoli_twa_marker_service_ProviderBindingImpl]: Destroy successful.
[5/7/13 12:14:33:202 SAST] 0000001b servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [twamarker] [/markerWeb] [com_ibm_tivoli_twa_marker_service_UpdateBindingImpl]: Destroy successful.
[5/7/13 12:14:33:220 SAST] 0000001b servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [twamarker] [/markerWeb] [com_ibm_tivoli_twa_marker_service_QueryBindingImpl]: Destroy successful.
[5/7/13 12:14:33:236 SAST] 0000001b ApplicationMg A   WSVR0220I: Application stopped: twamarker
[5/7/13 12:14:33:250 SAST] 0000001b ApplicationMg A   WSVR0217I: Stopping application: nameserver
[5/7/13 12:14:38:943 SAST] 00000017 impact        1 com.micromuse.sla.impact.RADService run NOTE ^println^S^TBSM Datasource connection is available
[5/7/13 12:15:08:987 SAST] 00000017 impact        1 com.micromuse.sla.impact.RADService run NOTE ^println^S^TBSM Datasource connection is available
[5/7/13 12:15:33:283 SAST] 0000001b servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [nameserver] [/nameserver] [services]: Destroy successful.
[5/7/13 12:15:33:304 SAST] 0000001b ApplicationMg A   WSVR0220I: Application stopped: nameserver
[5/7/13 12:15:33:324 SAST] 0000001b ApplicationMg A   WSVR0217I: Stopping application: ivtApp
[5/7/13 12:15:33:339 SAST] 0000001b EJBContainerI I   WSVR0041I: Stopping EJB jar: ivtEJB.jar
[5/7/13 12:15:33:352 SAST] 0000001b EJBContainerI I   WSVR0059I: EJB jar stopped: ivtEJB.jar
[5/7/13 12:15:33:378 SAST] 0000001b ApplicationMg A   WSVR0220I: Application stopped: ivtApp
[5/7/13 12:15:33:388 SAST] 0000001b ApplicationMg A   WSVR0217I: Stopping application: filetransferSecured
[5/7/13 12:15:33:402 SAST] 0000001b servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [filetransferSecured] [/FileTransfer] [transfer]: Destroy successful.
[5/7/13 12:15:33:422 SAST] 0000001b ApplicationMg A   WSVR0220I: Application stopped: filetransferSecured
[5/7/13 12:15:33:438 SAST] 0000001b SchedulerServ I   SCHD0040I: The Scheduler Service is stopping.
[5/7/13 12:15:33:448 SAST] 0000001b SchedulerServ I   SCHD0002I: The Scheduler Service has stopped.
[5/7/13 12:15:33:468 SAST] 0000001b AppProfileCom I   ACIN0009I: The application profiling service is stopping.
[5/7/13 12:15:33:478 SAST] 0000001b ActivitySessi I   WACS0049I: The ActivitySession service is stopping.
[5/7/13 12:15:33:492 SAST] 0000001b ObjectPoolSer I   OBPL0011I: The Object Pool service is stopping.
[5/7/13 12:15:33:509 SAST] 0000001b distSecurityC I   securityServiceStarted is false
[5/7/13 12:15:33:521 SAST] 0000001b CGBridgeSubsc I   CWRCB0104I: The core group bridge service has stopped the subscription router.
[5/7/13 12:15:33:532 SAST] 0000001b CGBridgeServi I   CWRCB0103I: The core group bridge service has stopped.
[5/7/13 12:15:33:570 SAST] 0000001b TCPChannel    I   TCPC0002I: TCP Channel TCPInboundChannel_ipcc.Default_IPC_Connector_Name has stopped listening on host localhost  (IPv4: 127.0.0.1) port 17314.
[5/7/13 12:15:33:614 SAST] 0000001b FailureScopeC A   WTRN0105I: The transaction service has shutdown successfully with no transactions requiring recovery.
[5/7/13 12:15:33:640 SAST] 0000001b ServerCollabo A   WSVR0024I: Server server1 stopped

 


We checked in /tbsmdev/opt/IBM/tivoli/tbsm/policy
There is policy named TBSM_DiscoverSchema.ipl.
Please suggest some solution
Thanks

 

 

 

  • randybrown
    randybrown
    54 Posts
    ACCEPTED ANSWER

    Re: The TBSM Data Server is unavailable, Reconnecting

    ‏2013-05-07T14:55:15Z  in response to techFanatic

    Hello,

    The exception in the log suggests that a file on the TBSM data server is unavailable or cannot be read. The policy file that is needed is installed with TBSM and is  found in the TBSM_HOME/policy directory and has the name TBSM_DiscoverSchema.ipl.

    Please check that this file is available and has read access defined.

    Thanks...

     

    Randy Brown

    • techFanatic
      techFanatic
      32 Posts
      ACCEPTED ANSWER

      Re: The TBSM Data Server is unavailable, Reconnecting

      ‏2013-05-24T07:09:55Z  in response to randybrown

      Actually was trying to fine tune TBSM which caused deletion of certain crucial system db2 log files.Now TBSM is down and data server isn't connecting.I cannot go to radshell and cant export my services created too.Is there a way to restore TBSM back to working condition?

      • randybrown
        randybrown
        54 Posts
        ACCEPTED ANSWER

        Re: The TBSM Data Server is unavailable, Reconnecting

        ‏2013-05-29T12:45:16Z  in response to techFanatic

        Hi,

        The TBSM data server will not be able to start up or continue running if a working connection to the DB2 database is not available. This means the radshell utility will not be available, as it must connect to the data server.

        My best recommendation would be to contact TBSM support to see what options might exist. There are utilities to reset the TBSM database schema, or even create a new database, but these result in the loss of your data. The support team would be the best resource to explore these options.

        Let me know if I can provide more information...

         

        Randy Brown

        • techFanatic
          techFanatic
          32 Posts
          ACCEPTED ANSWER

          Re: The TBSM Data Server is unavailable, Reconnecting

          ‏2013-05-30T09:54:37Z  in response to randybrown

          Hi Randy,


          I was able to restore TBSM back but I cant view the status/time of my services i.e the green symbol and neither the tree structure,Im afraid that the default Relationships View has been deleted during the process of restoration and I cant add a new view in the Service Viewer section.I receive an error as attached below.Need help here.I referred http://pic.dhe.ibm.com/infocenter/tivihelp/v3r1/index.jsp?topic=%2Fcom.ibm.tivoli.itbsm.doc_6.1%2FServiceConfigurationGuide%2Fbsmu_vdft_create_vdefs.html

          Regards,

          techFanatic

          Attachments

          Updated on 2013-05-30T09:58:44Z at 2013-05-30T09:58:44Z by techFanatic
          • randybrown
            randybrown
            54 Posts
            ACCEPTED ANSWER

            Re: The TBSM Data Server is unavailable, Reconnecting

            ‏2013-05-30T12:04:07Z  in response to techFanatic

            Hi,

            When you restored TBSM, I assume you mean the DB2 database was somehow recovered. It sounds like you may have recreated the schema, wiping out the services and templates. Is that correct?

            TBSM also has configuration files that it stores in its database during the install, so depending on how you restored the database, it may be missing these files, which would cause definite problems for the console in trying to show the service tree or create view definitions.

            If that is the case, you can run a utility called TBSMInitDB, which is found in TBSM_HOME/tools/bin. This will load the required configuration files and templates/services that are normally loaded by TBSM during the install.

            I hope this will help...

             

            Randy Brown

            • techFanatic
              techFanatic
              32 Posts
              ACCEPTED ANSWER

              Re: The TBSM Data Server is unavailable, Reconnecting

              ‏2013-05-30T13:18:34Z  in response to randybrown

              Hi Randy,

              I had exported the services present using radshell and imported them too.I found this technote http://www-01.ibm.com/support/docview.wss?uid=swg24032245 Now i need to know what part of this needs to be done so that I do not lose my data and services that I had earlier imported.My datasources havent changed and neither is there any folder as TBSM_HOME/tools/bin present .Is there any alternate solution to this...?

              Regards,

              techFanatic.

              Updated on 2013-05-30T13:20:18Z at 2013-05-30T13:20:18Z by techFanatic
              • randybrown
                randybrown
                54 Posts
                ACCEPTED ANSWER

                Re: The TBSM Data Server is unavailable, Reconnecting

                ‏2013-05-30T14:13:11Z  in response to techFanatic

                Hi,

                This technote describes the utility I referenced, which was added with Fixpack 1 of 6.1. So if you have not installed the fixpack, then you would not have this directory.

                The TBSMUpdateDatasource utility is used primarily to make it easy to update a password when it changes on DB2. It also handles more extreme changes, like moving the database to a new host, changing the database name, or changing the DB2 port.

                Sounds like you only need to run TBSMInitDB. This will install a set of templates that TBSM provides out of the box, and more importantly for your scenario it will add the configuration files required by TBSM to view the service tree and to access view definitions. It will not affect the data already in the database, meaning your existing services and templates. Of course it is always good to have a backup when a database is involved.

                So while I would recommend installing the fixpack, you can also just download the utilities from this technote and follow the instructions for TBSMInitDB.

                Let me know if you need more information...

                 

                Randy Brown

                 

                 

                • techFanatic
                  techFanatic
                  32 Posts
                  ACCEPTED ANSWER

                  Re: The TBSM Data Server is unavailable, Reconnecting

                  ‏2013-05-30T14:17:12Z  in response to randybrown

                  Hi Randy,

                  I copied the contents attached in the technote and ran TBSMInitDB and everything has worked back to normal.Thanks for the constant help.

                  Regards,
                  techFanatic

  • BestFortune
    BestFortune
    2 Posts
    ACCEPTED ANSWER

    Re: The TBSM Data Server is unavailable, Reconnecting

    ‏2013-05-28T06:36:15Z  in response to techFanatic

    Hi,

    I am getting the same error(Duplicate or Resync Error...) and my TBSM data server is showing as unavailable. Can anyone please suggest me how to resolve the issue??

    Thanks in advance

     

    • BestFortune
      BestFortune
      2 Posts
      ACCEPTED ANSWER

      Re: The TBSM Data Server is unavailable, Reconnecting

      ‏2013-05-28T14:10:10Z  in response to BestFortune

      ok. the  issue got resolved by importing the tbsm schema in the object server.

      Thanks