Maxdata and 32 bit process memory models

Answer

Purpose of this technote

Customers can see processes drop core due to a segmentation violation, a SIGSEGV, or an illegal instruction, SIGILL, and will want to know how to address the problem for their applications. This technote is intended for 32 bit applications. It does not apply to 64 bit applications and using these commands in a 64 bit environment can limit memory rather than increase it.

AIX memory address models

Most AIX programs, such as ksh, ls, ps, and other legacy commands, are 32 bit applications. Unless specified otherwise using compiler options any applications that are buit will also default to 32 bit. A 32 bit application has 16 256MB segments available to it for address space and by default one of these is used for data and stack. A 32 bit application can be made to use a large or very large memory model. A large memory model allows up to 2GB to be used for data and a very large memory model allows up to 3.25GB to be used for data.

Further information about memory model architecture can be found below in the "Other documentation" section below.

Sample of a core dump

Segmentation fault in . at 0x8381fff0

0x8381fff0 (???) warning: Unable to access address 0x8381fff0 from core

(dbx) where

.() at 0x8381fff0

(dbx) corefile

Process Name: ./mtg/arc/emacs-24.3.C/src/bootstrap-emacs

Version: 430

Flags: FULL_CORE | CORE_VERSION_1 | UBLOCK_VALID | USTACK_VALID| LE_VALID

Signal: SEGV

Process Mode: 32 bit

In most cases using dbx on the core file where a memory issue has caused a segmentation violation will show the fauling instruction, or function, as .(). It is also possible that the core will show a SIGILL which indicates an illegal instruction was issued. The "corefile" command will show what type of signal was issued.

You should ensure that process ulimits are not set to low values by using the "proc rlimit" command from within dbx as low ulimits can cause segmentation violations. If memory ulimits are unlimited then the likely problem is with the applicaton's MAXDATA setting.

Checking MAXDATA for a binary

This is done using "dump -ov". For example:

$ dump -ov /bin/ksh

/bin/ksh:

***Object Module Header***

# Sections Symbol Ptr # Symbols Opt Hdr Len Flags

4 0x00000000 0 72 0x1007

Flags=( RELFLG EXEC LNNO DYNLOAD DEP_SYSTEM )

Timestamp = "Mar 14 16:05:17 2014"

Magic = 0x1df (32-bit XCOFF)

...

maxSTACK maxDATA SNbss magic modtype

0x00000000 0x00000000 0x0003 0x010b 1L

This output tells us that ksh is 32 bit XCOFF application and MAXDATA is set to 0. This means it is using one 256MB segment for data and stack.

Here is a different AIX command where MAXDATA is set in the application binary:

$ dump -ov /bin/ps

/bin/ps:

***Object Module Header***

# Sections Symbol Ptr # Symbols Opt Hdr Len Flags

4 0x00000000 0 72 0x1047

Flags=( RELFLG EXEC LNNO DYNLOAD DSA DEP_SYSTEM )

Timestamp = "Jun 03 12:16:12 2013"

Magic = 0x1df (32-bit XCOFF)

...

maxSTACK maxDATA SNbss magic modtype

0x00000000 0x40000000 0x0003 0x010b 1L

The ps command, like ksh, is also a 32 bit XCOFF application and here MAXDATA is set to use 4 segments, or 1GB of data and stack. Setting LDR_CNTRL=MAXDATA=0x<value>, as discussed below, will not affect this application.

Setting MAXDATA

As mentioned above, the default memory model is one where data and heap share the same 256MB segment. The heap pointer grows down to the data pointer and the data pointer grows up to the heap pointer. When they collide the program does down with a SEGV.

The way to address this is to change the memory model to allow the program to use more memory in its data space. There are two ways to do this:

1. export LDR_CNTRL=MAXDATA=<value>

where "value" is

0x00000000 -> 256MB

0x10000000 -> 256MB

0x20000000 -> 512MB

0x30000000 -> 768GB

0x40000000 -> 1GB

0x50000000 -> 1.25GB

0x60000000 -> 1.5GB

0x70000000 -> 1.75GB

0x80000000 -> 2GB

0x90000000 -> 2.25GB

0xA0000000 -> 2.5GB

0xB0000000 -> 2.75GB

0xC0000000 -> 3GB

0XD0000000 -> 3.25GB

For example,

export LDR_CNTRL=MAXDATA=0x40000000

will set the memory model size to 1GB for all 32 bit applications in the current shell unless the program has maxData set internally, in which case the environment variable is ignored.

Values above 2GB must use the DSA, or dynamic segment allocation parameter. An example of this is

export LDR_CNTRL=MAXDATA=0xA0000000@DSA

or

LDR_CNTRL=MAXDATA=0xA0000000@DSA a.out

Usually MAXDATA is set in a script that calls the application. If this is set in a login environment it will affect all 32 bit apps run in that login shell.

The other way to change this memory model is to set it on the binary itself using the ldedit command, as was perhaps done on the ps program shown above

ldedit -b maxdata:<value> program

where "value" is as given above and "program" is the name of the program you are adjusting. The caveat with this is, whenever you install a new version of the program, the ldedit settings will be gone. For this reason many people use the LDR_CNTRL environment variable to control the memory model.

Caveats

1. Do not try to set MAXDATA for a 64 bit application. This can limit the memory is uses

2. Setting MAXDATA generally in a shell environment can effect all 32 bit applications run out of that shell

3. Setting MAXDATA to a very high number can cause problems if the applicaiton uses shared memory as the more segments get reserved for data the fewer segments can be used to hold share memory.

Other documentation on memory models:

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.genprogc/lrg_prg_support.htm

Incorrect documentation about process memory size is found in this Tivoli document:

https://publib.boulder.ibm.com/tividd/td/ITAME/GC32-0846-00/en_US/HTML/am39_perftune11.htm