Device Reloads, Backtrace Header Includes Configuration Object Information

Resolving The Problem

When the appliance must reload unrelated to user intervention (such as shutdown reboot, shutdown reload) a backtrace will typically be recorded. In certain cases, such as throttle events, a backtrace may not be recorded.

• Memory Throttler Events will contain just this message: Throttler forced system reload. Reason : Low Memory.Tue Jun 8 12:00:00 EST 2018 - in which you will need to review the system logs and track back in time before the throttler message in the logs, eg:

[0x804000a0][system][notice] throttle(Throttler): tid(1423): Not permitting connection. Reason(throttling state) : Low Memory

Note: The 'Low Memory' log message above can also be 'Normal', however we are still likely moving in and out of the throttle state. 'Terminate' means we have been unable to recover as per the Throttle Settings and must reload.

• Low XML Names / Prefixes / Namespaces / URIs / JSON Keys will contain a message such as: Throttler forced a system reload. Reason : Low XML Names Prefixes

Low XML Names Prefixes - Prefixes tied to a URI, eg in an xml element <ab:element/>, ab is the prefix.
Low XML Names URIs - URIs are namespaces, eg. xmlns:soap="http://www.w3.org/2003/05/soap-envelope/"
Low XML Names Local Names - XML elements, <element/> <element1/> etc. Note: JSON Keys currently throws the Local Names error.

• Regarding events when the throttler does not trigger, a reload may record a header inside the trace of Component=xyz, Domain=xyz, Config=xyz.

The Component field is the object type (CLI equivalent). For example sql-source for a SQL Source Object or mpgw for a Multi Protocol Gateway.

The Domain field is the domain related to the configuration object specified in the next field.

The Config field represents the configuration object name on DataPower. It will reside in the domain specified by the previous field.

When such information is recorded it can possibly pinpoint a root cause object related to the reload. A majority of the reloads will relate to user configured objects which will likely be outside the 'default' domain. This is not always the case, but as a temporary workaround, the object could be disabled (unless it is a required configuration object such as one residing in the default domain).

Note: In cases where the device reloads due to a watchdog, there may be multiple traces recorded. Typically most will be generated within the same second or a small window (roughly 5-10 seconds at most). In these situations, the fields above may not have any true relevance as the traces are generated on the spot and are more a representation of box activity and not necessarily what caused the device to reload. This will commonly include a few traces related to required configuration objects in the default domain, however, these are likely always running tasks and not related.

If you need further assistance in understanding why an appliance has reloaded please contact DataPower Support. Please make sure to include the backtrace information discovered as well as any error report generated from the appliance.