Data collection and Debugging Network memory related issues in AIX

Answer

The network memory related issues in AIX fall under one of these three categories :
I Double free, where we attempt to free an already freed memory. The second attempt to free the buffer causes system assert.
II Memory leak, where memory buffer is allocated but not freed due to a possible bug in the code. This leads to network memory depletion over a period of time.
III Memory corruption, where we either write beyond the allocated memory into the adjacent memory buffer or write into memory buffer that has been already freed by us.

The AIX network memory subsystem provides the following RAS features to help us debug these issues using a combination of error check level mechanism set using 'errctrl' command and certain network memory related 'no' options:

I Network memory policing for Double Free detection:

Double free issue is debugged by turning on network memory policing.

When enabled, this captures history of every allocation/free event for a network memory buffer. Ideally, the log history for a memory buffer should show an allocation record followed by a free record followed by another allocation record and so on. Presence of two consecutive free records for a memory buffer indicates an attempt to do double free.

In order to turn on allocation/free event logging, set the error check level to elevated (=5) or higher as :
# errctrl –c netmalloc errchecklevel=5

This is a run-time attribute and does not need a reboot. It internally sets the ‘no’ option ‘net_malloc_police’ to a non-zero value that enables logging.

To retain the setting across system reboot, use '-P' option with the errctrl command. However, on certain AIX releases, 'net_malloc_police' value may default to 0 across reboot and hence needs to be explicitly set after reboot:
# errctrl -c netmalloc errchecklevel=5 -P
# bosboot -aD
# reboot -q
After the system boots up, check if net_malloc_police is 0. If it is, we need to turn on 'net_malloc_police' as:
# no -Fo net_malloc_police=65536
If this is not done, allocation/free event will not be logged even though error check level is set to 5 or higher.

It is possible to track only specific buffer sizes or buffer types using the ‘no’ options ‘net_buf_size’ and ‘net_buf_type’ :
# no -Fo net_buf_size={all} or # no -Fo net_buf_size={64,512,4096}
# no -Fo net_buf_type={all} or # no -Fo net_buf_type={socket,pcb,mbuf}

Also, it is possible to track allocation/free events at normal error check level i.e. 3 itself without bumping up the error check level. If 'net_malloc_police' is non-zero, at normal error check level too, allocation/free event does get logged, but only a percentage of it. By default, only 10% of the allocation/free activity on random buffers gets logged at normal error check level. To increase logging percentage to 100%, we could configure ‘police_frequency’ as :
# errctrl –c netmalloc.police ‘police_frequency=1024’

When the system finally asserts as a result of double-free attempt, gather the dump. The kdb subcommand ‘netm -a <memory address>’ is used to display the allocation/freeing history related to a memory address. Two consecutive calls to net_free without an intermediate net_malloc call indicates a double free. By inspecting the stack trace along with the component trace, we could reconstruct the flow of events leading to the double-free of the buffer.

After data collection, remember to set the error check level back to normal i.e. a value of 3.

II Outstanding Memory Detection (OSTD) for Memory leak debugging:

Memory leak is debugged by turning on the OSTD feature of network memory subsystem.

When enabled, this captures and lists allocation information about all the outstanding network memory buffers that are still in use i.e. those that are allocated, but not yet freed. This information gives us a clue on who has allocated these buffers contributing to the memory leak.

To turn on OSTD, set the error check level to elevated (=5) or higher as :
# errctrl –c netmalloc errchecklevel=5

This is a run-time attribute and does not need a reboot.

To debug memory leak, it is suggested to additionally set the network option “extendednetstats” :
# no –ro extendednetstats=1 /* This option requires a reboot */
# reboot -q
With 'extendednetstats' enabled, 'netstat -m' output shows memory usage by type to give us a quick view of what kind of buffers are overly in use. Turning on this option is not absolutely essential though. It can be recommended to the customers only if they plan to reboot the system anyway. Otherwise, it is sufficient to turn on run-time OSTD feature alone.

Once the symptom of severe memory leak is seen on the system, such as socket allocation failure, initiate and gather the dump. The kdb subcommand 'netm -e' is used to display all the outstanding memory allocations that are not freed yet. These entries can be inspected to pick the identical buffer types that are allocated in large numbers. These are the suspect candidates contributing substantially to the memory leak. Running the kdb subcommand further on these buffer addresses as 'netm -e <buffer address>' should give a clue on the module that has allocated the buffer and hence is responsible for causing the leak.

Also, even if the error check level is set to a value less than 5, there is one condition under which the OSTD logging gets automatically enabled. When the total memory usage by the network memory subsystem exceeds 80% of ‘thewall’ 'no' option limit, then OSTD is automatically turned on. It is disabled again if the network memory usage falls below 75%.
Therefore, even if the error check level on the customer's system is set to normal level (i.e. 3) at the time they report the issue, but the memory leak happens to be still very much alive in the system, then request the customer to initiate and provide the dump immediately as it is possible that the OSTD feature is already turned on.

After data collection, remember to set the error check level back to normal i.e. a value of 3.

III Memory Overlay Detection System (MODS) for Memory corruption:

Memory corruption issues are debugged by turning on MODS as:
# errctrl –c netmalloc errchecklevel=7
or
# errctrl -c netmalloc errcheckdetail

Though MODS can be enabled at run-time as above, it's advisable to set it across reboot as follows. If set at run-time, there is a possibility that memory corruption that may happen with the already allocated buffers may not be caught:
# errctrl –c netmalloc errchecklevel=7 -P or # errctrl –c netmalloc errcheckdetail -P
# bosboot -aD
# reboot -q

Turning on MODS is sufficient if we are sure that the memory corruption is happening with buffers of size 4K or greater. If we are unsure, in addition to enabling MODS, it is required to turn on 'page promotion' for network memory buffers of size less than a page size (4K) as follows:
# no –o netm_page_promote=1
# no –o net_malloc_frag_mask={64,128,256,512,1024,2048}
This causes any memory allocation request for size 64, 128, 256, 512, 1024 and 2048 bytes to be converted into allocation request of page size i.e. 4K. We could selectively do page promotion for specific buffer sizes too, if we have knowledge of the size of buffer that is getting corrupted.

Also, note that 'netm_page_promote' option is set to 1 by default. However, unless 'net_malloc_frag_mask' is set, the system does not do page promotion.

Once the system asserts, gather the dump from the customer. As the system is made to halt right when the unauthorized access to memory is made, the stack trace from the dump should give an indication on who the culprit is.

After data collection, remember to set the error check level back to normal i.e. a value of 3.

Note:

1. To view the current error check level,
# errctrl -q | grep netmalloc

2. The errctrl setting is a run-time attribute. To retain the setting across system reboot, use '-P' option with the errctrl command:
# errctrl –c netmalloc errchecklevel=3 -P or # errctrl –c netmalloc errchecknormal -P
# bosboot -a

3. To restore the setting to normal error check level at run-time, once the data collection at a higher error-check level is complete,
# errctrl –c netmalloc errchecklevel=3

For this setting to stay across reboot, run the command with -P option followed by bosboot as mentioned above.

4. The ‘net_malloc_police’ 'no' option value stands for per-CPU logging buffer size, whose range is between 0 and 65536. Though it is user configurable, internally the value is limited by the number of CPUs as well as the available real memory on the system. If it is set to zero, it indicates that network memory policing is turned off.

5. Only if 'net_malloc_polic'e is non-zero does the allocation/free activity get logged. This is irrespective of the error check level that is configured on the system. So, ensure 'net_malloc_police' is set by running:
# no -Fa | grep net_malloc_police

6. The errctrl command internally sets 'net_malloc_police' to 65536 only when the run-time setting is done. Once the system is rebooted, net_malloc_police gets reset to zero irrespective of the errchecklevel set using -P option. Hence, in order to retain 'net_malloc_police' setting across reboot, it is required to turn on 'net_malloc_police':
# no -Fo net_malloc_police=65536

7. The following table shows the inter-dependency of net_malloc_police and errctrl error level setting:

net_malloc_police	error check level	Behavior
0 or non-zero	Minimal error check level or 1	No allocation/free event logging happens.
0	Normal error check level or 3	No allocation/free event logging happens.
non-zero	Normal error check level or 3	This logs 10% of net_malloc/net_free events for random buffers. To log a greater percentage of allocation/free events, you need to set the 'police_frequency' using errctrl command. A value of 1024 indicates 100% logging. So, to enable 50% logging, 'police_frequency' should be set to a value of 512.
0 or non-zero	Elevated error check level or 5	Irrespective of the value of net_malloc_police, when the errctrl command is run at run-time, it causes net_malloc_police to be set to a non-zero value based on the number of CPUs and the real memory on the system. This value stands for the per-CPU logging buffer. This configuration setting enables 100% logging of the allocation/free events. Additionally, OSTD feature is turned on at this error check level.
0 or non-zero	Detailed error check level or 7	Irrespective of the value of net_malloc_police, when the errctrl command is run at run-time, it causes net_malloc_police to be set to a non-zero value based on the number of CPUs and the real memory on the system. This value stands for the per-CPU logging buffer. This configuration setting enables 100% logging of the allocation/free events. At this error check level, OSTD and MODS are turned on.

8. The steps listed in this document applies to VIOS as well, in order to debug memory leak issues seen on VIOS.

SUPPORT:

If additional assistance is required after reviewing the content provided in this document, please follow the step-by-step instructions below to contact IBM to open a service request (PMR) for software under warranty or with an active and valid support contract. The technical support specialist assigned to your support call will confirm that you have thoroughly read the information in this article.

a. Contact IBM to open a support call (PMR):

- For electronic support, please visit the web page: https://www-947.ibm.com/support/servicerequest/newServiceRequest.action

- For telephone support, please visit the web page: http://www.ibm.com/planetwide

Please visit the IBM Support Portal web page for additional resources: https://www-947.ibm.com/support/entry/myportal/support

b. Provide a good description of your situation, issue, or question.

c. Upload any useful documentation and/or data to assist the support specialist to your support call (PMR):

Please visit this web page for instructions: https://www.secure.ecurep.ibm.com/app/upload

FEEDBACK:

Quality documentation is important to IBM and its customers. If you have feedback specific to this article, please send an detailed message to the email address:

aix_feedback@wwpdl.vnet.ibm.com

This email address is monitored for feedback purposes only. No support for any IBM products or services will be provided through this email. To receive support, please follow the step-by-step instructions in the above "SUPPORT" section.