Topic
10 replies Latest Post - ‏2012-03-07T16:58:29Z by Sammy1984
Sammy1984
Sammy1984
15 Posts
ACCEPTED ANSWER

Pinned topic Impact 5.1.1 event enrichment service is stopping

‏2012-02-22T12:49:03Z |
Hello Everyone,

The below are my environment details:

Netcool Impact 5.1.1
RHEL 5.5 64-bit

I have used Event Enrichment wizard to generate event enrichment policy. Every thing was working fine until last 15 days. From last 15 days i am observing that event enrichment is not happening even though the event enrichment services were running. When I restart event enrichment services it works fine for some time and then again it stops.

When I checked the log file, I found the following error:

[2/21/12 21:37:56:166 GMT+05:30] 0000001d SystemErr     R Exception in thread 
"WebContainer : 1" java.lang.RuntimeException: java.lang.OutOfMemoryError [2/21/12 21:37:56:167 GMT+05:30] 0000001d SystemErr     R   at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:801) [2/21/12 21:37:56:167 GMT+05:30] 0000001d SystemErr     R      at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:881) [2/21/12 21:37:56:168 GMT+05:30] 0000001d SystemErr     R       at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497) [2/21/12 21:37:56:168 GMT+05:30] 0000001d SystemErr     R Caused by: java.lang.OutOfMemoryError at java.lang.Throwable.fillInStackTrace(Native Method) at java.lang.Throwable.<init>(Throwable.java:56) at java.lang.Throwable.<init>(Throwable.java:67) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:206) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:751) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:881) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)


I have increased by following the below process

1) Stopped Impact server 2) Changed to NCHOME/eWAS/profiles/ImpactProfile/bin directory 3) Executed below set of commands ./wsadmin.sh  -conntype none -lang jython WASX7357I: By request, 

this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode. WASX7031I: For help, enter: 
"print Help.help()" wsadmin>jvm = AdminConfig.list(
"JavaVirtualMachine").split(
"\r\n") [0] wsadmin>AdminConfig.modify(jvm, 
'[[initialHeapSize 512]]') 
'' wsadmin>AdminConfig.modify(jvm, 
'[[maximumHeapSize 2048]]') 
'' wsadmin>attr=[] wsadmin>attr.append([[
'name',
'Xms'], [
'value',
'512']]) wsadmin>attr.append([[
'name',
'Xmx'], [
'value',
'2048']]) wsadmin>AdminConfig.modify(jvm, [[
'systemProperties',[]]]) 
'' wsadmin>AdminConfig.modify(jvm, [[
'systemProperties',attr]]) 
'' wsadmin>AdminConfig.save() 
'' wsadmin>exit


Increasing Java memory did not resolve my problem.

I am attaching log files and screenshots along with this post for the reference. Any help is really appreciated.

Thank you.

Attachments

Updated on 2012-03-07T16:58:29Z at 2012-03-07T16:58:29Z by Sammy1984
  • sunjit
    sunjit
    126 Posts
    ACCEPTED ANSWER

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-02-22T14:35:04Z  in response to Sammy1984
    You can enable Memory Monitoring in the Self Monitoring Service to check the memory usage. If your Impact install is 64 bit, you can set the max memory limit (Xmx) much higher than 2048MB.

    Regards,
    Sunjit Tara
    • Sammy1984
      Sammy1984
      15 Posts
      ACCEPTED ANSWER

      Re: Impact 5.1.1 event enrichment service is stopping

      ‏2012-02-22T18:39:56Z  in response to sunjit
      Hi Sunjit,

      Thank you for going through my post.

      Well its not just about Java memory, but I am also able to see below error in some of the logs.
      
      21 Feb 2012 21:38:24,244: Executed error-handler. 21 Feb 2012 21:38:24,281: An exception occurred 
      
      while executing Policy test_policy. Please examine server.log 
      
      for more details or check the Policy. The exception was: Unhandled Exception: java.lang.NullPointerException in policy: test_policy at line: 95.
      


      I am getting below error in netcool-error.log file under $NCHOME/logs directory.
      
      21 Feb 2012 21:35:49,627 ERROR [UddiClusterBootStrapper] Error pinging the nameserver DynamicBindingException- LABEL: MALFORMED_RESPONSE DESC: DynamicBindingException: nsservices server response [||] is malformed. at com.micromuse.common.nameserver.DynamicBinding.processResponse(DynamicBinding.java:2596) at com.micromuse.common.nameserver.DynamicBinding.getDesignatedBinding(DynamicBinding.java:1398) at com.micromuse.response.broker.cluster.UddiClusterNameserverHelper.getPrimaryInfo(UddiClusterNameserverHelper.java:284) at com.micromuse.response.broker.cluster.UddiClusterBootStrapper.pingNameserver(UddiClusterBootStrapper.java:381) at com.micromuse.response.broker.cluster.ClusterMember$NameserverTransBlock.doit(ClusterMember.java:1680) at com.micromuse.common.util.TimedExecutor$CallableTransBlockAdapter.call(TimedExecutor.java:100) at com.micromuse.common.util.concurrent.NamedDelegatingCallable.call(NamedDelegatingCallable.java:62) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.micromuse.common.util.concurrent.TrackingExecutor$1.run(TrackingExecutor.java:98) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690) at java.lang.Thread.run(Thread.java:810) 21 Feb 2012 21:35:49,627 ERROR [ClusterMember] Unable to contact the nameserver 21 Feb 2012 21:35:49,630 ERROR [UddiClusterBootStrapper] Error pinging the nameserver DynamicBindingException- LABEL: MALFORMED_RESPONSE DESC: DynamicBindingException: nsservices server response [||] is malformed. at com.micromuse.common.nameserver.DynamicBinding.processResponse(DynamicBinding.java:2596) at com.micromuse.common.nameserver.DynamicBinding.getDesignatedBinding(DynamicBinding.java:1398) at com.micromuse.response.broker.cluster.UddiClusterNameserverHelper.getPrimaryInfo(UddiClusterNameserverHelper.java:284) at com.micromuse.response.broker.cluster.UddiClusterBootStrapper.pingNameserver(UddiClusterBootStrapper.java:381) at com.micromuse.response.broker.cluster.ClusterMember$NameserverTransBlock.doit(ClusterMember.java:1680) at com.micromuse.common.util.TimedExecutor$CallableTransBlockAdapter.call(TimedExecutor.java:100) at com.micromuse.common.util.concurrent.NamedDelegatingCallable.call(NamedDelegatingCallable.java:62) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.micromuse.common.util.concurrent.TrackingExecutor$1.run(TrackingExecutor.java:98) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690) at java.lang.Thread.run(Thread.java:810) 21 Feb 2012 21:35:49,632 ERROR [ClusterMember] Unable to contact the nameserver
      


      If time permits please go through the attached log files and help me out in fixing this issue.

      Awaiting your reply.
      Thank you.
      • sunjit
        sunjit
        126 Posts
        ACCEPTED ANSWER

        Re: Impact 5.1.1 event enrichment service is stopping

        ‏2012-02-22T21:52:48Z  in response to Sammy1984
        Those messages could be a result of low heap and/or system memory availability. When Impact doesn't have enough room (heap/memory) and is very close to reaching the limit, it is already in an "unstable" state and you can expect to see NullPointerException messages since it doesn't have enough room to create Objects.
        As mentioned earlier, you can enable Memory Monitoring to check both the heap and system memory availability. Also consider increasing the max heap limit if resources are available.
        • Sammy1984
          Sammy1984
          15 Posts
          ACCEPTED ANSWER

          Re: Impact 5.1.1 event enrichment service is stopping

          ‏2012-02-28T14:27:03Z  in response to sunjit
          Hi Sunjit,

          I have increased Xmx size to 3072M, but still facing the same issue. The below are the memory details of the Impact server.
          
          [netcool@UPCDCDINI01 ~]$ free -g total       used       free     shared    buffers     cached Mem:            19         16          3          0          0         11 -/+ buffers/cache:          4         15 Swap:           39          0         39   [netcool@UPCDCDINI01 ~]$ cat /proc/meminfo MemTotal:     20523264 kB MemFree:       3656152 kB Buffers:        670208 kB Cached:       11768200 kB SwapCached:          0 kB Active:        4752008 kB Inactive:     11580436 kB HighTotal:           0 kB HighFree:            0 kB LowTotal:     20523264 kB LowFree:       3656152 kB SwapTotal:    40957708 kB SwapFree:     40957708 kB Dirty:            3812 kB Writeback:           0 kB AnonPages:     3893876 kB Mapped:         110892 kB Slab:           437500 kB PageTables:      27384 kB NFS_Unstable:        0 kB Bounce:              0 kB CommitLimit:  51219340 kB Committed_AS:  5412588 kB VmallocTotal: 34359738367 kB VmallocUsed:    285848 kB VmallocChunk: 34359452247 kB HugePages_Total:     0 HugePages_Free:      0 HugePages_Rsvd:      0 Hugepagesize:     2048 kB [netcool@UPCDCDINI01 ~]$
          


          I am using 64-bit Impact install on 64-bit RHEL 5.5.

          Can you place suggest me how can I take this forward?
          Thank you.
          • sunjit
            sunjit
            126 Posts
            ACCEPTED ANSWER

            Re: Impact 5.1.1 event enrichment service is stopping

            ‏2012-02-28T16:10:18Z  in response to Sammy1984
            I'd recommend contacting support, they can analyze your configuration/logs and also provide a fix if a problem has been identified.

            Regards,
            Sunjit
            • t_bar
              t_bar
              3 Posts
              ACCEPTED ANSWER

              Re: Impact 5.1.1 event enrichment service is stopping

              ‏2012-02-28T19:10:31Z  in response to sunjit
              I liked the suggestion "..enable Memory Monitoring in the Self Monitoring Service".
              This is a good thing to do, but has not been done (?).

              What about this:
              21 Feb 2012 21:38:24,244: Executed error-handler.
              21 Feb 2012 21:38:24,281: An exception occurred while executing Policy test_policy. Please examine server.log for more details or check the Policy. The exception was: Unhandled Exception: java.lang.NullPointerException in policy: test_policy at line: 95.

              Has the policy been checked?
              Are there a lot of these NullPointerExceptions before the memory runs out?
              You could specify a test_policy specific log file, and log some debug info with log(), and add your own error handler etc.
              Maybe too early to contact support? Any evidence the fault lies with Impact?

              HTH
              • Sammy1984
                Sammy1984
                15 Posts
                ACCEPTED ANSWER

                Re: Impact 5.1.1 event enrichment service is stopping

                ‏2012-03-01T13:43:23Z  in response to t_bar
                Hello HTH,

                I have enabled Memory Monitoring in the Self Monitoring Service, but not sure how it will be helpful.

                test_policy was created long back for testing purpose. As this policy is no more in use, I have deleted it (on 29/02/2012). This policy no more exists in Impact.

                Can you please help me out in understanding how Memory Monitoring will be helpful?

                Awaiting your reply..

                Thank you.
                • Sammy1984
                  Sammy1984
                  15 Posts
                  ACCEPTED ANSWER

                  Re: Impact 5.1.1 event enrichment service is stopping

                  ‏2012-03-01T15:50:16Z  in response to Sammy1984
                  Hello,

                  I found the following in the selfmonitoring service log:
                  
                  01 Mar 2012 21:06:15,651:  Sending the status event: (Type=13, Summary=Severity :  Not Running , EventEnrichment2 :  QueueSize: 0 DeltaQueue: 0, EventEnrichment1 :  QueueSize: 0 DeltaQueue: 0, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616175, Identifier=Impact Queue Status 
                  
                  for NCI, Node=UPCDCDINI01, Severity=1, LastOccurrence=1330616175) 01 Mar 2012 21:06:23,003: Memory Status: Alert: Low System Memory! Impact
                  's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory 01 Mar 2012 21:06:23,003:  Sending the status event: (Type=13, Summary=Alert: Low System Memory! Impact
                  's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616183, Identifier=Impact Memory Status for NCI, Node=UPCDCDINI01, Severity=5, LastOccurrence=1330616183) 01 Mar 2012 21:07:15,663:  Queue Status: Severity :  Not Running , EventEnrichment2 :  QueueSize: 0 DeltaQueue: 0, EventEnrichment1 :  QueueSize: 0 DeltaQueue: 0 01 Mar 2012 21:07:15,663:  Sending the status event: (Type=13, Summary=Severity :  Not Running , EventEnrichment2 :  QueueSize: 0 DeltaQueue: 0, EventEnrichment1 :  QueueSize: 0 DeltaQueue: 0, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616235, Identifier=Impact Queue Status 
                  
                  for NCI, Node=UPCDCDINI01, Severity=1, LastOccurrence=1330616235) 01 Mar 2012 21:07:25,071: Memory Status: Alert: Low System Memory! Impact
                  's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory 01 Mar 2012 21:07:25,071:  Sending the status event: (Type=13, Summary=Alert: Low System Memory! Impact
                  's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616245, Identifier=Impact Memory Status for NCI, Node=UPCDCDINI01, Severity=5, LastOccurrence=1330616245) 01 Mar 2012 21:08:15,674:  Queue Status: Severity :  Not Running , EventEnrichment2 :  QueueSize: 0 DeltaQueue: 0, EventEnrichment1 :  QueueSize: 0 DeltaQueue: 0 01 Mar 2012 21:08:15,674:  Sending the status event: (Type=13, Summary=Severity :  Not Running , EventEnrichment2 :  QueueSize: 0 DeltaQueue: 0, EventEnrichment1 :  QueueSize: 0 DeltaQueue: 0, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616295, Identifier=Impact Queue Status 
                  
                  for NCI, Node=UPCDCDINI01, Severity=1, LastOccurrence=1330616295)
                  


                  But I have nearly 5GB of unused memory.
                  
                  [netcool@UPCDCDINI01 log]$ free -g total       used       free     shared    buffers     cached Mem:            19         13          6          0          0         11 -/+ buffers/cache:          1         17 Swap:           39          0         39 [netcool@UPCDCDINI01 log]$
                  


                  Kindly find the log files as an attachment along with this post for the reference.

                  Any suggestions?

                  Thank you.

                  Attachments

                  • t_bar
                    t_bar
                    3 Posts
                    ACCEPTED ANSWER

                    Re: Impact 5.1.1 event enrichment service is stopping

                    ‏2012-03-02T13:14:51Z  in response to Sammy1984
                    Can you confirm you have Impact 5.1.1 64 bit installed? Rather than 32 bit. Executing:
                    file /opt/IBM/tivoli/tipv2/java/jre/bin/java
                    might tell you if it is 32 or 64 bit..
                    Are you still seeing out of memory errors?
                    • Sammy1984
                      Sammy1984
                      15 Posts
                      ACCEPTED ANSWER

                      Re: Impact 5.1.1 event enrichment service is stopping

                      ‏2012-03-07T16:58:29Z  in response to t_bar
                      Hello Everyone,

                      I was able to fix this problem after installing FP1 and IF1 for Netcool Impact 5.1.1. Theses fixes can be downloaded from the below URL

                      http://www-01.ibm.com/support/docview.wss?uid=swg24026299
                      https://www-304.ibm.com/support/docview.wss?uid=swg24027470

                      Thank you all who responded to my post.