IBM Support

High Memory usage after running trace command

Troubleshooting


Problem

Large amount of memory used by kernel trace is not released.

Symptom

Memory usage goes up after the usage of the trace command.
For example:
The below tracing command caused the kernel memory usage to jump from 3GB to 6.3GB in use.
Even after stopping the trace the memory usage was high.

# trace -anlp -r PURR -j 48F,106e,2FA,2FB,53B,53C,53D -C all -T50000000 -L100000000 -o /tmp/ibmsupt/testcase/trace/ktrace.raw

Cause

The kernel memory is not released after its unpinned because there is no kernel heap garbage collection.

Environment

AIX 6.1 TL9 SP8 (This may occur on different TL/SP).

Diagnosing The Problem

To check the high "kernel heap" usage run the following:
# perfpmr.sh -x memdetails.sh > memdetails.out
# grep "misc kernel" memdetails.out

Resolving The Problem

Run the vmstat before to see how much memory is used before:
# vmstat -v | grep computational

Reserve a one large_page of 16M and force to release it by running:
# vmo -o lgpg_size=16777216 - o lgpg_regions=1 ; vmo -o lgpg_regions=0
Setting lgpg_size to 16777216
Setting lgpg_regions to 1
Setting lgpg_regions to 0

To check the result of the released memory run the vmstat command after:
# vmstat -v | grep computational

In this specific example, memory usage reduced from 6.3GB to 4GB. The results may vary depending on the system configuration and activity.

[{"Product":{"code":"SWG10","label":"AIX"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"Process and memory management","Platform":[{"code":"PF002","label":"AIX"}],"Version":"6.1;Version Independent","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}},{"Product":{"code":"SWG10","label":"AIX"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
15 September 2021

UID

isg3T1025682