Topic
  • 8 replies
  • Latest Post - ‏2013-04-01T14:51:48Z by RM03_Chris_Barlock
cfulton
cfulton
7 Posts

Pinned topic Abnormal memory usage?

‏2013-03-23T15:19:51Z |
I have installed Application Insight on a RHEL 6.4 VM. I also installed the agent on a clone of this OS image in another VM.

Both were working fine but I noticed that the agents restarted and that the memory usage was creeping up for a couple days prior to the restart of the agent. The server did the same thing and CPU usage was >90%. SO I rebooted both machines and I'm noticing the same trend. The attached PDF shows that the Memory usage is creeping up for the agent. This VM is doing no other work and there are no other components loaded.

Thoughts?
Updated on 2013-04-01T14:51:48Z at 2013-04-01T14:51:48Z by RM03_Chris_Barlock
  • RM03_Chris_Barlock
    RM03_Chris_Barlock
    43 Posts

    Re: Abnormal memory usage?

    ‏2013-03-23T15:59:58Z  
    Which build do you have installed? You can check by clicking About on the help menu in the left vertical navigator. We recently fixed a memory leak in the group dashboard (home) page and a problem that caused high CPU utilization.

    Chris
  • cfulton
    cfulton
    7 Posts

    Re: Abnormal memory usage?

    ‏2013-03-23T23:42:09Z  
    Which build do you have installed? You can check by clicking About on the help menu in the left vertical navigator. We recently fixed a memory leak in the group dashboard (home) page and a problem that caused high CPU utilization.

    Chris
    Here is what I have installed:

    Version: 1.1
    Build Identifier: 201302121633

    I think I downloaded it about 2-3 weeks ago so maybe that is the issue?

    Chris
  • RM03_Chris_Barlock
    RM03_Chris_Barlock
    43 Posts

    Re: Abnormal memory usage?

    ‏2013-03-24T00:27:48Z  
    You have the latest build we put online. We have plans to post a new build in about a week.

    One of my co-workers reminded me of some research he did that Linux will allow applications to consume memory because it is available, but can reclaim it when it needs it. (Memory management is very OS-specific and you pretty much have to be an expert to understand the intricacies. I'd personally claim a little better than a beginner...)

    Have you noticed poor response time on your fabric node, VM's running agents or your browser?

    Chris
  • cfulton
    cfulton
    7 Posts

    Re: Abnormal memory usage?

    ‏2013-03-25T12:54:36Z  
    You have the latest build we put online. We have plans to post a new build in about a week.

    One of my co-workers reminded me of some research he did that Linux will allow applications to consume memory because it is available, but can reclaim it when it needs it. (Memory management is very OS-specific and you pretty much have to be an expert to understand the intricacies. I'd personally claim a little better than a beginner...)

    Have you noticed poor response time on your fabric node, VM's running agents or your browser?

    Chris
    Chris

    I have not noticed any slow down on the system where I have just the agent, nor on the fabric node.

    I'm attaching 2 files:

    The first is another screen shot of the scai endpoint system that I took this morning. The time stamp for this is from
    3/22 @ 3:42PM - TO - 3/25 @ 8:47AM

    This system has nothing running on it other than the endpoint. I understand your comment about Linux dynamically managing memory and maybe that is what is happening. But I still think the memory usage looks odd that it keeps increasing over time, very slowly.

    The second is from the fabric node. Running only the fabric node and ep. Timestamp is from
    3/21 @ 10:37PM - TO - 3/25 @ 8:55AM

    What concerns me about this is 2 things. 1: The memory is increasing just like on the other system. 2: The far right of the graph where the memory usage is increasing again is just from the past 10 minutes where I've been on the system. So, it is increasing at a faster rate. Furthermore, even when I've used the system in the past and the memory usage increased -- it never seems to release any memory (i.e. memory usage never decreases) over time.
  • cfulton
    cfulton
    7 Posts

    Re: Abnormal memory usage?

    ‏2013-03-25T12:55:27Z  
    • cfulton
    • ‏2013-03-25T12:54:36Z
    Chris

    I have not noticed any slow down on the system where I have just the agent, nor on the fabric node.

    I'm attaching 2 files:

    The first is another screen shot of the scai endpoint system that I took this morning. The time stamp for this is from
    3/22 @ 3:42PM - TO - 3/25 @ 8:47AM

    This system has nothing running on it other than the endpoint. I understand your comment about Linux dynamically managing memory and maybe that is what is happening. But I still think the memory usage looks odd that it keeps increasing over time, very slowly.

    The second is from the fabric node. Running only the fabric node and ep. Timestamp is from
    3/21 @ 10:37PM - TO - 3/25 @ 8:55AM

    What concerns me about this is 2 things. 1: The memory is increasing just like on the other system. 2: The far right of the graph where the memory usage is increasing again is just from the past 10 minutes where I've been on the system. So, it is increasing at a faster rate. Furthermore, even when I've used the system in the past and the memory usage increased -- it never seems to release any memory (i.e. memory usage never decreases) over time.
    Here is the first file -- it didn't get uploaded on the previous message.
  • RM03_Chris_Barlock
    RM03_Chris_Barlock
    43 Posts

    Re: Abnormal memory usage?

    ‏2013-03-25T19:45:10Z  
    • cfulton
    • ‏2013-03-25T12:55:27Z
    Here is the first file -- it didn't get uploaded on the previous message.
    Our performance expert took a look and commented:

    I looked at this memory issue thread. It looks a lot like what I have seen with EC2 environment where I was trying to put 2 agents per very small VM (600MB RAM). I had better success with one agent per VM. I think the issue was limited resources. RHEL 6 system requirement is 2GB RAM minimum.

    Chris
  • cfulton
    cfulton
    7 Posts

    Re: Abnormal memory usage?

    ‏2013-03-25T20:13:06Z  
    Our performance expert took a look and commented:

    I looked at this memory issue thread. It looks a lot like what I have seen with EC2 environment where I was trying to put 2 agents per very small VM (600MB RAM). I had better success with one agent per VM. I think the issue was limited resources. RHEL 6 system requirement is 2GB RAM minimum.

    Chris
    Both systems are RHEL 6 with 2 virtual CPU's and 2GB of RAM.

    There is only one agent on these boxes. I'm not sure that I understand the comment about RHEL requiring 2GB ram minimum and how that would cause an agent to grow its memory size over time.

    My guess is, based on the current trend, my sytems will run out of memory in the next several days. If that happens I will repost then. Otherwise, I wanted to make you aware of this and the potential issue that I see. I'm happy to provide any other testing or feedback that may be helpful.
  • RM03_Chris_Barlock
    RM03_Chris_Barlock
    43 Posts

    Re: Abnormal memory usage?

    ‏2013-04-01T14:51:48Z  
    • cfulton
    • ‏2013-03-25T20:13:06Z
    Both systems are RHEL 6 with 2 virtual CPU's and 2GB of RAM.

    There is only one agent on these boxes. I'm not sure that I understand the comment about RHEL requiring 2GB ram minimum and how that would cause an agent to grow its memory size over time.

    My guess is, based on the current trend, my sytems will run out of memory in the next several days. If that happens I will repost then. Otherwise, I wanted to make you aware of this and the potential issue that I see. I'm happy to provide any other testing or feedback that may be helpful.
    Chris:

    Did you predication come true? In any case, may I suggest you try the new beta that was posted yesterday? We have fixed a number of problems related to memory management since the beta you are using.

    Chris