Topic
  • 10 replies
  • Latest Post - ‏2012-03-07T16:58:29Z by Sammy1984
Sammy1984
Sammy1984
15 Posts

Pinned topic Impact 5.1.1 event enrichment service is stopping

‏2012-02-22T12:49:03Z |
Hello Everyone,

The below are my environment details:

Netcool Impact 5.1.1
RHEL 5.5 64-bit

I have used Event Enrichment wizard to generate event enrichment policy. Every thing was working fine until last 15 days. From last 15 days i am observing that event enrichment is not happening even though the event enrichment services were running. When I restart event enrichment services it works fine for some time and then again it stops.

When I checked the log file, I found the following error:

[2/21/12 21:37:56:166 GMT+05:30] 0000001d SystemErr     R Exception in thread 
"WebContainer : 1" java.lang.RuntimeException: java.lang.OutOfMemoryError [2/21/12 21:37:56:167 GMT+05:30] 0000001d SystemErr     R   at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:801) [2/21/12 21:37:56:167 GMT+05:30] 0000001d SystemErr     R      at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:881) [2/21/12 21:37:56:168 GMT+05:30] 0000001d SystemErr     R       at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497) [2/21/12 21:37:56:168 GMT+05:30] 0000001d SystemErr     R Caused by: java.lang.OutOfMemoryError at java.lang.Throwable.fillInStackTrace(Native Method) at java.lang.Throwable.<init>(Throwable.java:56) at java.lang.Throwable.<init>(Throwable.java:67) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:206) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:751) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:881) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)


I have increased by following the below process

1) Stopped Impact server 2) Changed to NCHOME/eWAS/profiles/ImpactProfile/bin directory 3) Executed below set of commands ./wsadmin.sh  -conntype none -lang jython WASX7357I: By request, 

this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode. WASX7031I: For help, enter: 
"print Help.help()" wsadmin>jvm = AdminConfig.list(
"JavaVirtualMachine").split(
"\r\n") [0] wsadmin>AdminConfig.modify(jvm, 
'[[initialHeapSize 512]]') 
'' wsadmin>AdminConfig.modify(jvm, 
'[[maximumHeapSize 2048]]') 
'' wsadmin>attr=[] wsadmin>attr.append([[
'name',
'Xms'], [
'value',
'512']]) wsadmin>attr.append([[
'name',
'Xmx'], [
'value',
'2048']]) wsadmin>AdminConfig.modify(jvm, [[
'systemProperties',[]]]) 
'' wsadmin>AdminConfig.modify(jvm, [[
'systemProperties',attr]]) 
'' wsadmin>AdminConfig.save() 
'' wsadmin>exit


Increasing Java memory did not resolve my problem.

I am attaching log files and screenshots along with this post for the reference. Any help is really appreciated.

Thank you.

Attachments

Updated on 2012-03-07T16:58:29Z at 2012-03-07T16:58:29Z by Sammy1984
  • sunjit
    sunjit
    132 Posts

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-02-22T14:35:04Z  
    You can enable Memory Monitoring in the Self Monitoring Service to check the memory usage. If your Impact install is 64 bit, you can set the max memory limit (Xmx) much higher than 2048MB.

    Regards,
    Sunjit Tara
  • Sammy1984
    Sammy1984
    15 Posts

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-02-22T18:39:56Z  
    • sunjit
    • ‏2012-02-22T14:35:04Z
    You can enable Memory Monitoring in the Self Monitoring Service to check the memory usage. If your Impact install is 64 bit, you can set the max memory limit (Xmx) much higher than 2048MB.

    Regards,
    Sunjit Tara
    Hi Sunjit,

    Thank you for going through my post.

    Well its not just about Java memory, but I am also able to see below error in some of the logs.
    
    21 Feb 2012 21:38:24,244: Executed error-handler. 21 Feb 2012 21:38:24,281: An exception occurred 
    
    while executing Policy test_policy. Please examine server.log 
    
    for more details or check the Policy. The exception was: Unhandled Exception: java.lang.NullPointerException in policy: test_policy at line: 95.
    


    I am getting below error in netcool-error.log file under $NCHOME/logs directory.
    
    21 Feb 2012 21:35:49,627 ERROR [UddiClusterBootStrapper] Error pinging the nameserver DynamicBindingException- LABEL: MALFORMED_RESPONSE DESC: DynamicBindingException: nsservices server response [||] is malformed. at com.micromuse.common.nameserver.DynamicBinding.processResponse(DynamicBinding.java:2596) at com.micromuse.common.nameserver.DynamicBinding.getDesignatedBinding(DynamicBinding.java:1398) at com.micromuse.response.broker.cluster.UddiClusterNameserverHelper.getPrimaryInfo(UddiClusterNameserverHelper.java:284) at com.micromuse.response.broker.cluster.UddiClusterBootStrapper.pingNameserver(UddiClusterBootStrapper.java:381) at com.micromuse.response.broker.cluster.ClusterMember$NameserverTransBlock.doit(ClusterMember.java:1680) at com.micromuse.common.util.TimedExecutor$CallableTransBlockAdapter.call(TimedExecutor.java:100) at com.micromuse.common.util.concurrent.NamedDelegatingCallable.call(NamedDelegatingCallable.java:62) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.micromuse.common.util.concurrent.TrackingExecutor$1.run(TrackingExecutor.java:98) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690) at java.lang.Thread.run(Thread.java:810) 21 Feb 2012 21:35:49,627 ERROR [ClusterMember] Unable to contact the nameserver 21 Feb 2012 21:35:49,630 ERROR [UddiClusterBootStrapper] Error pinging the nameserver DynamicBindingException- LABEL: MALFORMED_RESPONSE DESC: DynamicBindingException: nsservices server response [||] is malformed. at com.micromuse.common.nameserver.DynamicBinding.processResponse(DynamicBinding.java:2596) at com.micromuse.common.nameserver.DynamicBinding.getDesignatedBinding(DynamicBinding.java:1398) at com.micromuse.response.broker.cluster.UddiClusterNameserverHelper.getPrimaryInfo(UddiClusterNameserverHelper.java:284) at com.micromuse.response.broker.cluster.UddiClusterBootStrapper.pingNameserver(UddiClusterBootStrapper.java:381) at com.micromuse.response.broker.cluster.ClusterMember$NameserverTransBlock.doit(ClusterMember.java:1680) at com.micromuse.common.util.TimedExecutor$CallableTransBlockAdapter.call(TimedExecutor.java:100) at com.micromuse.common.util.concurrent.NamedDelegatingCallable.call(NamedDelegatingCallable.java:62) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.micromuse.common.util.concurrent.TrackingExecutor$1.run(TrackingExecutor.java:98) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690) at java.lang.Thread.run(Thread.java:810) 21 Feb 2012 21:35:49,632 ERROR [ClusterMember] Unable to contact the nameserver
    


    If time permits please go through the attached log files and help me out in fixing this issue.

    Awaiting your reply.
    Thank you.
  • sunjit
    sunjit
    132 Posts

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-02-22T21:52:48Z  
    • Sammy1984
    • ‏2012-02-22T18:39:56Z
    Hi Sunjit,

    Thank you for going through my post.

    Well its not just about Java memory, but I am also able to see below error in some of the logs.
    <pre class="jive-pre"> 21 Feb 2012 21:38:24,244: Executed error-handler. 21 Feb 2012 21:38:24,281: An exception occurred while executing Policy test_policy. Please examine server.log for more details or check the Policy. The exception was: Unhandled Exception: java.lang.NullPointerException in policy: test_policy at line: 95. </pre>

    I am getting below error in netcool-error.log file under $NCHOME/logs directory.
    <pre class="jive-pre"> 21 Feb 2012 21:35:49,627 ERROR [UddiClusterBootStrapper] Error pinging the nameserver DynamicBindingException- LABEL: MALFORMED_RESPONSE DESC: DynamicBindingException: nsservices server response [||] is malformed. at com.micromuse.common.nameserver.DynamicBinding.processResponse(DynamicBinding.java:2596) at com.micromuse.common.nameserver.DynamicBinding.getDesignatedBinding(DynamicBinding.java:1398) at com.micromuse.response.broker.cluster.UddiClusterNameserverHelper.getPrimaryInfo(UddiClusterNameserverHelper.java:284) at com.micromuse.response.broker.cluster.UddiClusterBootStrapper.pingNameserver(UddiClusterBootStrapper.java:381) at com.micromuse.response.broker.cluster.ClusterMember$NameserverTransBlock.doit(ClusterMember.java:1680) at com.micromuse.common.util.TimedExecutor$CallableTransBlockAdapter.call(TimedExecutor.java:100) at com.micromuse.common.util.concurrent.NamedDelegatingCallable.call(NamedDelegatingCallable.java:62) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.micromuse.common.util.concurrent.TrackingExecutor$1.run(TrackingExecutor.java:98) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690) at java.lang.Thread.run(Thread.java:810) 21 Feb 2012 21:35:49,627 ERROR [ClusterMember] Unable to contact the nameserver 21 Feb 2012 21:35:49,630 ERROR [UddiClusterBootStrapper] Error pinging the nameserver DynamicBindingException- LABEL: MALFORMED_RESPONSE DESC: DynamicBindingException: nsservices server response [||] is malformed. at com.micromuse.common.nameserver.DynamicBinding.processResponse(DynamicBinding.java:2596) at com.micromuse.common.nameserver.DynamicBinding.getDesignatedBinding(DynamicBinding.java:1398) at com.micromuse.response.broker.cluster.UddiClusterNameserverHelper.getPrimaryInfo(UddiClusterNameserverHelper.java:284) at com.micromuse.response.broker.cluster.UddiClusterBootStrapper.pingNameserver(UddiClusterBootStrapper.java:381) at com.micromuse.response.broker.cluster.ClusterMember$NameserverTransBlock.doit(ClusterMember.java:1680) at com.micromuse.common.util.TimedExecutor$CallableTransBlockAdapter.call(TimedExecutor.java:100) at com.micromuse.common.util.concurrent.NamedDelegatingCallable.call(NamedDelegatingCallable.java:62) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.micromuse.common.util.concurrent.TrackingExecutor$1.run(TrackingExecutor.java:98) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690) at java.lang.Thread.run(Thread.java:810) 21 Feb 2012 21:35:49,632 ERROR [ClusterMember] Unable to contact the nameserver </pre>

    If time permits please go through the attached log files and help me out in fixing this issue.

    Awaiting your reply.
    Thank you.
    Those messages could be a result of low heap and/or system memory availability. When Impact doesn't have enough room (heap/memory) and is very close to reaching the limit, it is already in an "unstable" state and you can expect to see NullPointerException messages since it doesn't have enough room to create Objects.
    As mentioned earlier, you can enable Memory Monitoring to check both the heap and system memory availability. Also consider increasing the max heap limit if resources are available.
  • Sammy1984
    Sammy1984
    15 Posts

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-02-28T14:27:03Z  
    • sunjit
    • ‏2012-02-22T21:52:48Z
    Those messages could be a result of low heap and/or system memory availability. When Impact doesn't have enough room (heap/memory) and is very close to reaching the limit, it is already in an "unstable" state and you can expect to see NullPointerException messages since it doesn't have enough room to create Objects.
    As mentioned earlier, you can enable Memory Monitoring to check both the heap and system memory availability. Also consider increasing the max heap limit if resources are available.
    Hi Sunjit,

    I have increased Xmx size to 3072M, but still facing the same issue. The below are the memory details of the Impact server.
    
    [netcool@UPCDCDINI01 ~]$ free -g total       used       free     shared    buffers     cached Mem:            19         16          3          0          0         11 -/+ buffers/cache:          4         15 Swap:           39          0         39   [netcool@UPCDCDINI01 ~]$ cat /proc/meminfo MemTotal:     20523264 kB MemFree:       3656152 kB Buffers:        670208 kB Cached:       11768200 kB SwapCached:          0 kB Active:        4752008 kB Inactive:     11580436 kB HighTotal:           0 kB HighFree:            0 kB LowTotal:     20523264 kB LowFree:       3656152 kB SwapTotal:    40957708 kB SwapFree:     40957708 kB Dirty:            3812 kB Writeback:           0 kB AnonPages:     3893876 kB Mapped:         110892 kB Slab:           437500 kB PageTables:      27384 kB NFS_Unstable:        0 kB Bounce:              0 kB CommitLimit:  51219340 kB Committed_AS:  5412588 kB VmallocTotal: 34359738367 kB VmallocUsed:    285848 kB VmallocChunk: 34359452247 kB HugePages_Total:     0 HugePages_Free:      0 HugePages_Rsvd:      0 Hugepagesize:     2048 kB [netcool@UPCDCDINI01 ~]$
    


    I am using 64-bit Impact install on 64-bit RHEL 5.5.

    Can you place suggest me how can I take this forward?
    Thank you.
  • sunjit
    sunjit
    132 Posts

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-02-28T16:10:18Z  
    • Sammy1984
    • ‏2012-02-28T14:27:03Z
    Hi Sunjit,

    I have increased Xmx size to 3072M, but still facing the same issue. The below are the memory details of the Impact server.
    <pre class="jive-pre"> [netcool@UPCDCDINI01 ~]$ free -g total used free shared buffers cached Mem: 19 16 3 0 0 11 -/+ buffers/cache: 4 15 Swap: 39 0 39 [netcool@UPCDCDINI01 ~]$ cat /proc/meminfo MemTotal: 20523264 kB MemFree: 3656152 kB Buffers: 670208 kB Cached: 11768200 kB SwapCached: 0 kB Active: 4752008 kB Inactive: 11580436 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 20523264 kB LowFree: 3656152 kB SwapTotal: 40957708 kB SwapFree: 40957708 kB Dirty: 3812 kB Writeback: 0 kB AnonPages: 3893876 kB Mapped: 110892 kB Slab: 437500 kB PageTables: 27384 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 51219340 kB Committed_AS: 5412588 kB VmallocTotal: 34359738367 kB VmallocUsed: 285848 kB VmallocChunk: 34359452247 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kB [netcool@UPCDCDINI01 ~]$ </pre>

    I am using 64-bit Impact install on 64-bit RHEL 5.5.

    Can you place suggest me how can I take this forward?
    Thank you.
    I'd recommend contacting support, they can analyze your configuration/logs and also provide a fix if a problem has been identified.

    Regards,
    Sunjit
  • t_bar
    t_bar
    3 Posts

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-02-28T19:10:31Z  
    • sunjit
    • ‏2012-02-28T16:10:18Z
    I'd recommend contacting support, they can analyze your configuration/logs and also provide a fix if a problem has been identified.

    Regards,
    Sunjit
    I liked the suggestion "..enable Memory Monitoring in the Self Monitoring Service".
    This is a good thing to do, but has not been done (?).

    What about this:
    21 Feb 2012 21:38:24,244: Executed error-handler.
    21 Feb 2012 21:38:24,281: An exception occurred while executing Policy test_policy. Please examine server.log for more details or check the Policy. The exception was: Unhandled Exception: java.lang.NullPointerException in policy: test_policy at line: 95.

    Has the policy been checked?
    Are there a lot of these NullPointerExceptions before the memory runs out?
    You could specify a test_policy specific log file, and log some debug info with log(), and add your own error handler etc.
    Maybe too early to contact support? Any evidence the fault lies with Impact?

    HTH
  • Sammy1984
    Sammy1984
    15 Posts

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-03-01T13:43:23Z  
    • t_bar
    • ‏2012-02-28T19:10:31Z
    I liked the suggestion "..enable Memory Monitoring in the Self Monitoring Service".
    This is a good thing to do, but has not been done (?).

    What about this:
    21 Feb 2012 21:38:24,244: Executed error-handler.
    21 Feb 2012 21:38:24,281: An exception occurred while executing Policy test_policy. Please examine server.log for more details or check the Policy. The exception was: Unhandled Exception: java.lang.NullPointerException in policy: test_policy at line: 95.

    Has the policy been checked?
    Are there a lot of these NullPointerExceptions before the memory runs out?
    You could specify a test_policy specific log file, and log some debug info with log(), and add your own error handler etc.
    Maybe too early to contact support? Any evidence the fault lies with Impact?

    HTH
    Hello HTH,

    I have enabled Memory Monitoring in the Self Monitoring Service, but not sure how it will be helpful.

    test_policy was created long back for testing purpose. As this policy is no more in use, I have deleted it (on 29/02/2012). This policy no more exists in Impact.

    Can you please help me out in understanding how Memory Monitoring will be helpful?

    Awaiting your reply..

    Thank you.
  • Sammy1984
    Sammy1984
    15 Posts

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-03-01T15:50:16Z  
    • Sammy1984
    • ‏2012-03-01T13:43:23Z
    Hello HTH,

    I have enabled Memory Monitoring in the Self Monitoring Service, but not sure how it will be helpful.

    test_policy was created long back for testing purpose. As this policy is no more in use, I have deleted it (on 29/02/2012). This policy no more exists in Impact.

    Can you please help me out in understanding how Memory Monitoring will be helpful?

    Awaiting your reply..

    Thank you.
    Hello,

    I found the following in the selfmonitoring service log:
    
    01 Mar 2012 21:06:15,651:  Sending the status event: (Type=13, Summary=Severity :  Not Running , EventEnrichment2 :  QueueSize: 0 DeltaQueue: 0, EventEnrichment1 :  QueueSize: 0 DeltaQueue: 0, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616175, Identifier=Impact Queue Status 
    
    for NCI, Node=UPCDCDINI01, Severity=1, LastOccurrence=1330616175) 01 Mar 2012 21:06:23,003: Memory Status: Alert: Low System Memory! Impact
    's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory 01 Mar 2012 21:06:23,003:  Sending the status event: (Type=13, Summary=Alert: Low System Memory! Impact
    's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616183, Identifier=Impact Memory Status for NCI, Node=UPCDCDINI01, Severity=5, LastOccurrence=1330616183) 01 Mar 2012 21:07:15,663:  Queue Status: Severity :  Not Running , EventEnrichment2 :  QueueSize: 0 DeltaQueue: 0, EventEnrichment1 :  QueueSize: 0 DeltaQueue: 0 01 Mar 2012 21:07:15,663:  Sending the status event: (Type=13, Summary=Severity :  Not Running , EventEnrichment2 :  QueueSize: 0 DeltaQueue: 0, EventEnrichment1 :  QueueSize: 0 DeltaQueue: 0, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616235, Identifier=Impact Queue Status 
    
    for NCI, Node=UPCDCDINI01, Severity=1, LastOccurrence=1330616235) 01 Mar 2012 21:07:25,071: Memory Status: Alert: Low System Memory! Impact
    's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory 01 Mar 2012 21:07:25,071:  Sending the status event: (Type=13, Summary=Alert: Low System Memory! Impact
    's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616245, Identifier=Impact Memory Status for NCI, Node=UPCDCDINI01, Severity=5, LastOccurrence=1330616245) 01 Mar 2012 21:08:15,674:  Queue Status: Severity :  Not Running , EventEnrichment2 :  QueueSize: 0 DeltaQueue: 0, EventEnrichment1 :  QueueSize: 0 DeltaQueue: 0 01 Mar 2012 21:08:15,674:  Sending the status event: (Type=13, Summary=Severity :  Not Running , EventEnrichment2 :  QueueSize: 0 DeltaQueue: 0, EventEnrichment1 :  QueueSize: 0 DeltaQueue: 0, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616295, Identifier=Impact Queue Status 
    
    for NCI, Node=UPCDCDINI01, Severity=1, LastOccurrence=1330616295)
    


    But I have nearly 5GB of unused memory.
    
    [netcool@UPCDCDINI01 log]$ free -g total       used       free     shared    buffers     cached Mem:            19         13          6          0          0         11 -/+ buffers/cache:          1         17 Swap:           39          0         39 [netcool@UPCDCDINI01 log]$
    


    Kindly find the log files as an attachment along with this post for the reference.

    Any suggestions?

    Thank you.

    Attachments

  • t_bar
    t_bar
    3 Posts

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-03-02T13:14:51Z  
    • Sammy1984
    • ‏2012-03-01T15:50:16Z
    Hello,

    I found the following in the selfmonitoring service log:
    <pre class="jive-pre"> 01 Mar 2012 21:06:15,651: Sending the status event: (Type=13, Summary=Severity : Not Running , EventEnrichment2 : QueueSize: 0 DeltaQueue: 0, EventEnrichment1 : QueueSize: 0 DeltaQueue: 0, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616175, Identifier=Impact Queue Status for NCI, Node=UPCDCDINI01, Severity=1, LastOccurrence=1330616175) 01 Mar 2012 21:06:23,003: Memory Status: Alert: Low System Memory! Impact 's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory 01 Mar 2012 21:06:23,003: Sending the status event: (Type=13, Summary=Alert: Low System Memory! Impact 's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616183, Identifier=Impact Memory Status for NCI, Node=UPCDCDINI01, Severity=5, LastOccurrence=1330616183) 01 Mar 2012 21:07:15,663: Queue Status: Severity : Not Running , EventEnrichment2 : QueueSize: 0 DeltaQueue: 0, EventEnrichment1 : QueueSize: 0 DeltaQueue: 0 01 Mar 2012 21:07:15,663: Sending the status event: (Type=13, Summary=Severity : Not Running , EventEnrichment2 : QueueSize: 0 DeltaQueue: 0, EventEnrichment1 : QueueSize: 0 DeltaQueue: 0, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616235, Identifier=Impact Queue Status for NCI, Node=UPCDCDINI01, Severity=1, LastOccurrence=1330616235) 01 Mar 2012 21:07:25,071: Memory Status: Alert: Low System Memory! Impact 's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory 01 Mar 2012 21:07:25,071: Sending the status event: (Type=13, Summary=Alert: Low System Memory! Impact 's Heap using 512M out of 3072M, Free System Memory Available: 645M, Impact requires atleast: 3222M of memory, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616245, Identifier=Impact Memory Status for NCI, Node=UPCDCDINI01, Severity=5, LastOccurrence=1330616245) 01 Mar 2012 21:08:15,674: Queue Status: Severity : Not Running , EventEnrichment2 : QueueSize: 0 DeltaQueue: 0, EventEnrichment1 : QueueSize: 0 DeltaQueue: 0 01 Mar 2012 21:08:15,674: Sending the status event: (Type=13, Summary=Severity : Not Running , EventEnrichment2 : QueueSize: 0 DeltaQueue: 0, EventEnrichment1 : QueueSize: 0 DeltaQueue: 0, Class=10500, AlertGroup=ImpactStatus, NodeAlias=10.1.5.138, FirstOccurrence=1330616295, Identifier=Impact Queue Status for NCI, Node=UPCDCDINI01, Severity=1, LastOccurrence=1330616295) </pre>

    But I have nearly 5GB of unused memory.
    <pre class="jive-pre"> [netcool@UPCDCDINI01 log]$ free -g total used free shared buffers cached Mem: 19 13 6 0 0 11 -/+ buffers/cache: 1 17 Swap: 39 0 39 [netcool@UPCDCDINI01 log]$ </pre>

    Kindly find the log files as an attachment along with this post for the reference.

    Any suggestions?

    Thank you.
    Can you confirm you have Impact 5.1.1 64 bit installed? Rather than 32 bit. Executing:
    file /opt/IBM/tivoli/tipv2/java/jre/bin/java
    might tell you if it is 32 or 64 bit..
    Are you still seeing out of memory errors?
  • Sammy1984
    Sammy1984
    15 Posts

    Re: Impact 5.1.1 event enrichment service is stopping

    ‏2012-03-07T16:58:29Z  
    • t_bar
    • ‏2012-03-02T13:14:51Z
    Can you confirm you have Impact 5.1.1 64 bit installed? Rather than 32 bit. Executing:
    file /opt/IBM/tivoli/tipv2/java/jre/bin/java
    might tell you if it is 32 or 64 bit..
    Are you still seeing out of memory errors?
    Hello Everyone,

    I was able to fix this problem after installing FP1 and IF1 for Netcool Impact 5.1.1. Theses fixes can be downloaded from the below URL

    http://www-01.ibm.com/support/docview.wss?uid=swg24026299
    https://www-304.ibm.com/support/docview.wss?uid=swg24027470

    Thank you all who responded to my post.