Out of memory during data sync with many administrators

You might encounter a memory issue during data synchronization when you have many administrators for one or more admin domains.

As an example, consider the following organization (container) structure:

Organization
	|--- Client Admin Domains -- Have 1000 Administrators
	|--- Client Admin Domain 1 -- Have 3 Administrators
	|--- Client Admin Domain 2 -- Have 3 Administrators
	|--- Client Admin Domain 3 -- Have 3 Administrators
	|--- ................... 
	|--- ...................
	|--- Client Admin Domain 50000 -- Have 3 Administrators

Administrators of the parent container are also administrators of child containers. So each child admin domain from 1 to 50000 have 1003 administrators. This administrator detail is stored in memory in a ‘HashMap’ during data sync process.

The data that is stored in ‘HashMap’ has the following format:

(Key - DistinguishedName of container, Value – HashSet of SystemUser DNs of the administrators)

Since there are around 50000 admin domains (child containers), each having approximately 1000 administrators, the number of HashSet objects are approximately 50000 and the total number of objects that are stored in the HashSet will be 50,000,000 (50,000 containers * 1000 administrator entries for each container). Thus the amount of memory that is used is large and results in out of memory.

Each entry in HashSet takes 32 bytes of memory. So the memory occupied by 1000 SystemUser DNs is approximately 32000 bytes = 31 KB. 31 KB of memory is required to store SystemUser information of each container. Since you have 50000 containers, total memory that is required is approximately 1600000000 bytes (50000 containers * 32000 bytes used for storing administrators details of each container) = 1.49 GB.

Tune the following two properties adminCacheSize and supervisorCacheSize in adhocreporting.properties file to limit the size of the HashMap.

The property, adminCacheSize, specifies the maximum number of admin domain containers whose administrator data is stored in the cache(HashMap).
The property, supervisorCacheSize, specifies the maximum number of OU/Location containers whose supervisor data is stored in the cache(HashMap).

You must have an idea about the number of administrators set for any container. Administrators of a parent container are also administrators of child containers.

Remember the following considerations:

Number of administrators of the parent container = Number of elements that are stored in the HashMap for parent container. Each element is an object of HashMap$Entry instance.
Memory that is used by HashMap$Entry instances to store administrator information for a single container = Number of administrators of a single container * 32 bytes
If you have x numbers of child containers for this particular parent container, then the number of HashMap$Entry instances are x * number of administrators of the parent container.
Memory used = x containers * number of administrators of the parent container * 32 bytes

Depending on the memory usage value that is described, the number of container details that you store in memory, must be decided. You decide this value, so that the memory that is used will not be high. In an ideal scenario, you can limit the memory that is occupied by admin domain cache to 256 MB.

Use the following formula to calculate the appropriate value for the adminCacheSize.

Value of adminCacheSize property = (max memory limit * 1024 * 1024) bytes / (average number of administrators for each container * 32 bytes)

For example:

If the described container structure scenario is considered, then the value of adminCacheSize property will be calculated as follows:

Value of adminCacheSize = (256*1024*1024) / (1000*32) = 8388.608 = 8000 approximately (rounded figure).

The adminCacheSize value can be set to 8000, so that the memory that is used is 256000000 bytes (8000 containers * 32000 bytes used to store administrators of each container) = 244 MB approximately.

This process limits the amount of memory that is used by the admin domain admin cache to 256 MB.

If you do not have such large number of administrators, the default value of -1 is appropriate. This default setting stores all the administrator details in memory. If your environment has a large number of administrators in the parent container and also have many child containers for the parent container, then you must confirm whether you are facing memory issues during data synchronization. The next step is to analyze the heap dumps and if the leak suspect is the HashMap storing HashSet objects, then configure the values of adminCacheSize and supervisorCacheSize by performing the calculation as explained above.

Note: A similar calculation is also applicable for the property, supervisorCacheSize. Supervisor is applicable only for OU and locations. Admin Domains and Business Partner Organizations do not have supervisors. This should be considered while deciding the number of containers present in the Organization structure.