Automation Agent Statuses

SA z/OS defines different automation agent statuses that it applies to automated resources. The automation agent status of a resource is determined from a combination of its application monitor status, its desired status, recent history, and intended action.

These statuses are used on the automation agents to control the automation. They are translated to observed statuses (see Mapping the Automation Agent Statuses to the Automation Manager Observed Status) and then sent to the automation manager.

On the automation agent you can use the DISPSTAT and DISPINFO commands to see the automation statuses.

Table 1 gives a brief description of each automation agent status.

Table 1. Automation Agent Statuses
Value	Meaning
ABENDING	The application is undergoing a recoverable abnormal end. This status is entered when a TERMINATION message with the ABEND=YES attribute is received. It remains in ABENDING until its final termination message is received. If the application is defined to Automatic Restart Manager it may be posted to EXTSTART when Automatic Restart Manager attempts to recover it. If the application has not exceeded its critical threshold, SA z/OS posts the application to RESTART and attempts to restart it. If it has exceeded its critical threshold it is put into BROKEN status when it has been cleaned up, and SA z/OS does not attempt to start it. If it is slow to leave the system after its FINAL message is received it may go to ZOMBIE status. If the application is undergoing a normal shutdown by SA z/OS no further shutdown commands are issued. The shutdown may be resumed by using the INGREQ command with a stronger type.
ACTIVE	The application is running, but is not yet ready for work. An application can be put into ACTIVE status in response to the following conditions: SA z/OS has received an ACTIVE message for the application. The SA z/OS startup checker has run for the application when its automation status was STARTED. The startup checker found the application monitor status for the application to be ACTIVE. During routine status checking the application monitor status for the application was found to be ACTIVE when its automation status indicated it should be otherwise. SA z/OS attempted to start the application, but found that its application monitor status was ACTIVE. SA z/OS found that the Automatic Restart Manager status for the application is STARTING, but has not received any messages concerning the application status. SA z/OS checked an attempt by Automatic Restart Manager to restart the application and found that its Automatic Restart Manager status is UNREGISTERED, but the application monitor status for the application is ACTIVE. An application remains in ACTIVE status until its UP message is received. If the application is starting or restarting, an SA z/OS monitoring check for the application is done after its start delay time and number of start cycles. If the application is not found and if the critical threshold is not reached, SA z/OS will attempt to restart the application. If it is found, but is not yet UP, it is put into STARTED2 status.
AUTODOWN	The application is shut down. SA z/OS may restart it to comply with an operator request. If the shutdown specified that the application was to be restarted, it is put into RESTART status when the shutdown is complete. SA z/OS has shut the application down. When SA z/OS initialized, the operator replied NOSTART to the AOF603D WTOR. Any applications that would have been put into the DOWN status during initial status determination have instead been put into the AUTODOWN automation status. You can use the SETSTATE command to change the application status to either RESTART or CTLDOWN. SA z/OS may attempt to restart an application in AUTODOWN status when SA z/OS is reloaded, or when an operator requests SA z/OS to start one of the application descendents. In both cases, the application goes to RESTART status.
AUTOTERM	SA z/OS is in the process of shutting the application down. The shutdown is in response to a INGREQ REQ=STOP command. This status persists until SA z/OS is sure that the application has been cleaned up. Many things may happen to an application that is being shut down. If the shutdown is successful, the application is placed in either AUTODOWN or CTLDOWN status. If the shutdown specified that the application should be restarted it goes through AUTODOWN to RESTART status. If the application abnormally ends while it is being shut down it may go into either ABENDING or BREAKING. A normal shutdown will stop processing an application that abends, but other shutdowns will continue. If the shutdown runs out of commands to issue, the application is placed into STUCK status. If it has problems shutting down, the application may be placed into ZOMBIE status.
BREAKING	The application is undergoing a nonrecoverable abend; that is, it has received a termination message specifying BREAK=YES. If the application is undergoing a normal shutdown by SA z/OS no further shutdown commands are issued. The shutdown may be resumed by using the INGREQ REQ=STOP command. This status persists until SA z/OS receives its final termination message and is sure that the application has been cleaned up. If the termination experiences difficulties, the application may be posted to ZOMBIE status.
BROKEN	The application has suffered a nonrecoverable abend. SA z/OS will not restart it. An application can be put into BROKEN status in response to the following conditions: The application has suffered a nonrecoverable abend, indicated by the reception of a TERMINATION message with the BREAK=YES attribute. The application has suffered sufficient recoverable abends to exceed its critical threshold. Issuing the application's prestart or startup command resulted in a non-zero return code. This status is preserved across a recycle of SA z/OS or a re-IPL of the processor, unless the application does not have its Restart after IPL option set to NOSTART.
CTLDOWN	The application is shut down and SA z/OS is not allowed to restart it. An operator asked SA z/OS to shut the application down and not to restart it until authorized to do so by an operator. An operator used a SETSTATE command to tell SA z/OS that an application should not be restarted until an operator authorizes SA z/OS to do so. You can use the SETSTATE command to change an application status from CTLDOWN to RESTART or AUTODOWN, in which case SA z/OS will attempt to restart it.
DOWN	The application has not been started during the lifetime of this SA z/OS. The DOWN status is set only during initial status determination and is possible only if the application monitor status is INACTIVE. The automation status of the application when SA z/OS was last shut down on this system is used in the following manner to determine if it is to be placed into the DOWN status. The previous automation status was not one of STOPPED, CTLDOWN or BROKEN. The application does not have its Restart after IPL option set to NOSTART, the previous state was BROKEN, CTLDOWN or STOPPED, and this is the first time the agent has been started since the system was IPLed.
ENDED	This status is used for transient applications only, and indicates that the job for the application has finished and left the system without any errors. Any start-dependent resources for the application will be started as though it were a normal z/OS subsystem that was UP. If the transient application can be rerun, you can use the SETSTATE command to restart it. If the transient application cannot be rerun, it will remain in ENDED status.
ENDING	A transient application is in the process of terminating. A transient application goes to ENDING status when a termination message is received for it, and it is not being shut down by SA z/OS. This status shows that the application is terminating, but that this is expected. A transient application may also go to ENDING if an operator is shutting it down outside SA z/OS control, or if it has abnormally ended, but the abend messages are being treated as normal termination messages. The application remains in ENDING status until either: An abend message is received that will put it into either ABENDING or BREAKING status. If the application abends then either SA z/OS or Automatic Restart Manager can restart it. The application final termination message is received, at which point the RESTARTOPT for the application is checked. If it is ALWAYS then the application is put into RESTART status and SA z/OS will attempt to restart it. If it is anything else, the application goes to ENDED status. It is assumed that if a transient application ends normally then it will deregister from Automatic Restart Manager. If it is slow to clear the system after its FINAL message is received it may go to ZOMBIE status. If an ACTIVE or UP message is received for the application, its automation status is changed to either ACTIVE or UP, as appropriate.
EXTSTART	SA z/OS has determined that the application is being started or restarted by an agent external to SA z/OS. In situations where SA z/OS is able to identify the external agent (such as Automatic Restart Manager), it takes appropriate steps to monitor that agent's actions and, if necessary, step in to assist it. An application can be put into EXTSTART status in response to the following conditions: SA z/OS is unable to identify the external agent. Automatic Restart Manager is in the process of restarting the application.
FALLBACK	The application is not running on the primary system where it should run, but this status has been encountered for this application on one of its secondary systems. It should be active on another system. If the other system fails, the application can fall back to this system, where it could possibly be restarted. However, SA z/OS, will not perform the restart on the fallback system, but this may be done by an operator request. This is implemented to leave the decision of restarting the application on the fallback system to the operator. An application can be put into FALLBACK status in response to the following conditions: It is defined with a secondary association on this system. An operator has used the SETSTATE command to put the application into the MOVED status. If this is one of the secondary systems for the application it will go to FALLBACK instead.
HALFDOWN	SA z/OS was in the process of shutting the application down, but the stop request was canceled while it was in progress. The application shutdown did not complete (for example, ASCBs may still be active). You may sometimes find that some, but not all, of the shutdown commands have been issued. To recover an application from HALFDOWN status you must determine where it is in its shutdown process and complete the shutdown manually. Applications go into HALFDOWN only when you cancel a stop request. Alternatively, you can use SETSTATE to put the application back into UP or RUNNING status.
HALTED	The application is still running, but something has happened that may have severely impacted its capabilities. SA z/OS has received a HALT message for the application. SA z/OS has detected that the application represents a JES2 application that is running short of spool space. The Automatic Restart Manager status for the application is ELSEWHERE, but SA z/OS found its application monitor status to be either STARTING or ACTIVE. An application is taken out of HALTED status if its UP message is received. Also, operators may use the SETSTATE command to put the application into UP or RUNNING status.
MOVED	The application is not running: this is one of its primary systems: it should be active on another system. An application can be put into MOVED status in response to the following conditions: An operator has used the SETSTATE command to put the application into the MOVED status. This is possible only on a primary system. The startup detected that Automatic Restart Manager has started the subsystem on another system. A subsystem will remain in the MOVED status until it is restarted on the primary system by an external agent, such as an operator.
RESTART	The application is ready to be started. It has been previously active in the system. An application can be put into RESTART status in response to the following conditions: The application abended and, after checking thresholds, SA z/OS is allowed to restart it. SA z/OS has shut the application down in response to an operator request and is now preparing to restart it. An operator has used the INGREQ REQ=START command to ask SA z/OS to restart the application. SA z/OS checked an attempt by Automatic Restart Manager to restart the application and found that its Automatic Restart Manager status is UNREGISTERED and the application monitor status for the application is INACTIVE. This implies that the attempt by Automatic Restart Manager to restart the application timed out while the application was in RESTARTING status. SA z/OS changes the automation status of the application to RESTART and attempts to start the application itself. During restart processing, the application RESTART automation flag is checked. If it is turned on, the application start commands are issued and the application is put into STARTED status. If the RESTART automation flag is off, the application remains in RESTART status and the startup monitor cycle initiates the startup process each time it runs.
RUNNING	This status is equivalent to UP, but is used for transient applications. It indicates that the UP message has been received for the transient application, or an operator has used the SETSTATE command to change the status of a transient application to UP. A transient application is one that SA z/OS expects to terminate on its own. When the job finishes the application goes through ENDING status to ENDED, at which point its descendants are started. Unlike the UP status, the descendants of a transient application are not started until it has terminated. A transient application should leave RUNNING status on its own. If it gets stuck, you should investigate it. You can use the INGREQ REQ=STOP command to put it into AUTODOWN status.
STARTED	The commands to start the application have been issued, but it has yet to start running. An application can be put into STARTED status in response to the following conditions: SA z/OS has issued, or will soon issue, the commands. When SA z/OS attempted to start the application it found that the application monitor status for the application was STARTING. During initial status determination, SA z/OS found that the application monitor status for the application was STARTING. SA z/OS checked an attempt by Automatic Restart Manager to restart the application and found that its Automatic Restart Manager status is UNREGISTERED, but the application monitor status for the application is STARTING. Note that the relevant automation flag, Initstart or Restart, must be on. The application startup commands as defined in the automation control file are issued after the application is placed in STARTING status. An application remains in STARTING status until either its ACTIVE or its UP message arrives. After the application start delay time, an SA z/OS monitoring check is issued for it. If it is not found, and if the critical threshold is not reached, SA z/OS will attempt to restart the application. If it is found, but is not yet UP, it is put into ACTIVE status.
STARTED2	The application has become active, but has not indicated that it is ready to accept work within its start delay and number of start cycles. An application can be put into STARTED2 status in response to the following conditions: The startup checker found that the application monitor status was still STARTING. The startup checker was called for an application whose automation status was ACTIVE. SA z/OS found the Automatic Restart Manager status for the application to be AVAILABLE-TO. SA z/OS checked an attempt by Automatic Restart Manager to restart the application and found that its Automatic Restart Manager status is one of AVAILABLE-TO, RESTARTING or RECOVERING. SA z/OS checked an attempt by Automatic Restart Manager to restart the application and could not find a better status to put it into. Its application monitor status is ACTIVE. This covers situations where the restart checker is unable to determine the Automatic Restart Manager status for the application, or its status is FAILED. An application remains in STARTED2 status until either its UP message arrives or an operator uses the SETSTATE command to change the application status to UP or RUNNING.
STOPPED	The application has been shut down by an external agent, such as an operator cancel. SA z/OS is not permitted to restart it and will not allow the Automatic Restart Manager to restart it. This status is preserved across a recycle of SA z/OS or a re-IPL of the processor unless the application has its Restart after IPL Option set to START or NOSTART. An application remains in STOPPED status until an operator uses the SETSTATE command to change its status to either RESTART or CTLDOWN. An application may also leave STOPPED status if it is restarted outside the control of SA z/OS. In this case, it goes to either the ACTIVE or UP status, depending on which message is received.
STOPPING	The application is being shut down by an external agent. This status is entered if a TERMINATION message is received but SA z/OS is not in the process of shutting the application down. It may also indicate that the application is abending, but the abend messages issued are being treated as normal termination messages by automation. An application that is STOPPING remains in that status until either an abend message is received that puts it into ABENDING or BREAKING status, or the final termination message for the application is received, at which point the application RESTART option is checked. If RESTART is ALWAYS, the application is put into RESTART status and SA z/OS attempts to restart it. Otherwise, the application goes to STOPPED status. If it is slow to clear the system after the final message is received, it may be placed into ZOMBIE status. If an ACTIVE or UP message is received for the application, its automation status is changed to either ACTIVE or UP, as appropriate. Automatic Restart Manager interaction with this status depends on a number of factors. If the application is deregistered then there is no interaction. If the application is registered then Automatic Restart Manager will attempt to restart it. If the application goes to STOPPED status before the SA z/OS or Automatic Restart Manager exit is invoked then SA z/OS will tell Automatic Restart Manager not to restart the application. If it does not, SA z/OS will tell Automatic Restart Manager that it can restart the application.
STUCK	An application can get STUCK when it is being shut down. This is because SA z/OS has run out of NORM, IMMED, or FORCE shutdown commands.
UP	The application has finished initializing and is ready for work. An application can be put into UP status in response to the following conditions: SA z/OS has received an UP message for the application. An operator has used the SETSTATE command to change the application automation status. In this case, SA z/OS assumes that the operator has ensured that the application is actually UP. During initial status determination the application monitor status was found to be ACTIVE. SA z/OS found the Automatic Restart Manager status for the application to be AVAILABLE. There are a number of ways for an application to leave the UP status, if: It is shut down with the INGREQ REQ=STOP command, it goes to AUTOTERM status It is shut down outside SA z/OS, it goes to STOPPING status It abends, it might go to STOPPING, ABENDING or BREAKING status It has problems, it may go to HALTED status The regular monitor cannot find it, it will call the TERMMSG generic routine The application abends, SA z/OS does not pick up the abend messages, and Automatic Restart Manager detects that the address space has ended, the application may go to EXTSTART.
ZOMBIE	When an application is leaving the system it can enter a ZOMBIE status. This indicates that the final termination message for the application has been received but that SA z/OS monitoring still finds the application. SA z/OS retries monitoring after a delay and the application is put into ZOMBIE status if this situation persists for more than eight retries. There are three ways that an application can enter a ZOMBIE status: If MVS is slow in clearing the application and the termination delay time is short. If there are two jobs with the same name in the system, one of which is the application. When either of them terminates and SA z/OS does not know the address space ID of the application or does not get the address space ID with the termination message, SA z/OS assumes that the application has stopped, but SA z/OS monitoring will find the other job. To change the status to UP, either manually shut down the other job, or use the SETSTATE command to change the application status back to UP. The job may have become stuck in the system after issuing its final message. From ZOMBIE status, if the application suffers an unrecoverable abend it will go into BREAKING status. Note: The Restart after IPL option of the customization dialog may override these resource statuses at SA z/OS IPL or recycle, resulting in SA z/OS starting the subsystem.

Figure 1 and Figure 2 indicate the relationships between the automation statuses. You can change the states illustrated here with the SETSTATE command (see Changing the Automation Agent Status).

Figure 1. Transitions for a Normal Subsystem

Figure 2. Transitions for an Abending Subsystem