Automatic restart management parameters for administrative data utility

The automatic restart management policy identifies elements and restart groups with restart parameter values that differ from the policy default values.

Your installation does not need to specify a policy if the default values are acceptable, or if the application overrides the defaults when coding the IXCARM macro. The default values (if any) are described under each parameter, along with the IXCARM macro specifications that can be used to override these defaults.

Note: The automatic restart management policy values are determined when an element is restarted, not when the batch job or started task registers with automatic restart management.

RESTART_ORDER

Specifies the order in which elements in the same restart group are to become ready after they are restarted. The RESTART_ORDER parameter applies to all restart groups. RESTART_ORDER is optional. The default values for RESTART_ORDER are:

LEVEL(0) - ELEMENT_TYPEs: SYSIRLM, SYSLVL0
LEVEL(1) - ELEMENT_TYPEs: SYSDB2, SYSIMS, SYSVTAM, SYSTCPIP, SYSLVL1
LEVEL(2) - ELEMENT_TYPEs: SYSCICS, SYSMQMGR, SYSMQCH, SYSKERB, SYSLVL2
LEVEL(3) - ELEMENT_TYPEs: SYSCB, SYSLVL3
Any elements that do not match a default or policy-specified ELEMENT_TYPE or a policy-specified ELEMENT_NAME.

LEVEL(level)

Specifies the level associated with elements that must be restarted in a particular order. The elements are restarted from the lowest level to the highest level. The LEVEL keyword can be specified multiple times.

The set of elements associated with a particular level is identified by the ELEMENT_NAME or ELEMENT_TYPE parameters. ELEMENT_NAME or ELEMENT_TYPE must be specified with each LEVEL specification. (You can specify both ELEMENT_NAME and ELEMENT_TYPE.)

The level must be a decimal number from 0 through 65535. The default is LEVEL(2). The IXCARM macro parameter that overrides the default specification is ELEMTYPE (which allows an element to give a name or type that is associated with another LEVEL).

ELEMENT_NAME(name-list)

Specifies the name of each element to be restarted at the level specified by LEVEL. The element name can contain the wildcard characters '?' or '*'. See Using wildcard characters for more information about wildcard characters.

The name-list can be one or more element names separated by commas.

IBM® reserves the use of element names starting with the characters A through I and "SYS" for IBM use.

ELEMENT_TYPE(type-list)

Specifies the element types that are to be restarted at the level specified by LEVEL. The element type is a name given to elements with similar characteristics. For example, PAYROLL might be the element type of multiple payroll-related programs (elements).

Type-list can be one or more element types separated by commas. You can specify the element type on the IXCARM macro by coding the parameter ELEMTYPE.

IBM reserves the use of element types starting with the characters A through I and "SYS" for IBM use.

RESTART_GROUP(*|name)

Identifies related elements that are to be restarted as a group if the system on which they are running fails. Not all elements in a given restart group need to be running on the same system, or running at all.

The restart group name can:

Be up to 16 characters long with no imbedded blanks
Be a combination of alphabetic characters, numbers, and $, #, and @
Not begin with a number
Be named DEFAULT to change the characteristics (target system, CSA requirements, and so on) of the default restart group
Not be repeated
Not contain wildcard characters (except for RESTART_GROUP(*))

The RESTART_GROUP parameter can be specified more than once in a policy.

However, RESTART_GROUP(*) is a special category and can be specified only once. A global substitution of the values specified here are applied to all restart groups that are not specified in the policy. For example, when the following is specified, only systems SYS1 and SYS3 can be the targets of a cross-system restart.

RESTART GROUP(*)
   TARGET_SYSTEM(SYS1,SYS3)

As another example, the following restart groups are specified in a policy:

RESTART_GROUP(*)
 TARGET_SYSTEM(SYS1,SYS3)
RESTART_GROUP(JIM)
 TARGET_SYSTEM(SYS2)
 ELEMENT(CICS1)
 ELEMENT(DB28)

If a system fails, the elements listed in the restart group named JIM would be restarted only on SYS2. All other elements would be restarted on SYS1 or SYS3.

For each restart group, you can associate the following optional parameters:

RESTART_GROUP
   TARGET_SYSTEM
   FREE_CSA
   RESTART_PACING

Within each restart group, you specify the element or elements that are members of the restart group. Each element specifies its own set of restart requirements.

RESTART_GROUP is a required parameter. All elements that are not put into a specific restart group by a policy are in the restart group named DEFAULT.

TARGET_SYSTEM(*|system-list)

Specifies the systems on which elements can be restarted in a cross-system restart. Specify this parameter when you do not want all systems to be the targets of a cross-system restart.

The system-list must be one or more system names separated by commas. The order of the systems in system-list is irrelevant.

TARGET_SYSTEM(*) indicates that any system that supports automatic restart management might be the target of a cross-system restart. When the sysplex is running with WLM in goal mode, automatic restart management will choose the system that has the greatest available CPU capacity and that has the required available CSA (if the FREE_CSA parameter was specified for this restart group).

TARGET_SYSTEM is optional. If specified, the name of the target system must be 1-8 characters. Valid characters are alphanumeric (A-Z) and (0-9) and national characters (@,#,$). The first character of the target system name can be an alphanumeric or national character. TARGET_SYSTEM(*) is the default.

FREE_CSA(below,above)

Specifies the minimum amount of CSA and ECSA that must be available on a system for it to be eligible for a cross-system restart of the elements in the associated restart group.

FREE_CSA is optional. The default is FREE_CSA(0,0).

Note: This parameter value will not disqualify a system as eligible for a cross-system restart if WLM is running in compatibility mode on that system.

below: Indicates the number of kilobytes (KB) of CSA that must be available on a target system for this restart group to be restarted. A value of 0 indicates that the amount of free CSA is not a requirement for eligible systems for this restart group.
The maximum decimal value that can be specified for this keyword is 16384. The default is 0.
above: Indicates the number of kilobytes (KB) of ECSA that must be available on a target system for this restart group to be restarted. A value of 0 indicates that the amount of free ECSA is not a requirement for eligible systems for this restart group.
The maximum decimal value that can be specified for this keyword is 2080767. The default is 0.

RESTART_PACING(delay)

Specifies the amount of time in seconds that automatic restart management should delay between the restart of each element in the restart group. This value provides the ability to stagger the restarts of the elements to minimize contention on a system.

The value for delay must be a decimal number from 0 to 21600 seconds (6 hours). This value applies only for cross-system restarts.

RESTART_PACING is optional. The default is RESTART_PACING(0).

ELEMENT(*|name)

Specifies a batch job or started task that can register as an element of automatic restart management. The wildcard characters '?' or '*' can be used. See Using wildcard characters for more information about wildcard characters.

In addition, automatic restart management provides two system symbols that relate to the policy syntax for automatic restarts.

IBM reserves the use of element names starting with the characters A through I and "SYS" for IBM use.

&SYSELEM.

Specifies that the element name from the previous ELEMENT statement should be substituted for this system symbol. Element names used for &SYSELEM. cannot exceed 8 characters in length. For example:

ELEMENT(KAREN??)
  RESTART_METHOD(SYSTERM,STC,'S &SYSELEM.')

If an element named KAREN1A needs to be restarted, MVS™ substitutes 'S KAREN1A' for the command text.

&SYSSUF.

Substitutes the suffix of an element name. The characters following the first wildcard character are considered the suffix.

For example, if element ELWOOD was being restarted and the best match from an installation policy was ELEMENT(EL*), then the suffix for this element would be WOOD. Similarly, when element PKASKOVICH matches the policy entry ELEMENT(PKAS??VI*), then the suffix for this element would be KOVICH.

The substitution text for &SYSSUF. cannot exceed 7 characters.

See Using system symbol substitution for general information about using system symbols.

You can specify ELEMENT(*) only when RESTART_GROUP(*) or RESTART_GROUP(DEFAULT) is specified. When ELEMENT(*) is specified with RESTART_GROUP(*), the element-related parameters apply to all elements in all restart groups (if these parameters do not have explicitly specified values for those restart groups). When ELEMENT(*) is specified with RESTART_GROUP(DEFAULT), the associated parameters apply to all elements in this restart group.

ELEMENT is a required parameter. For each element specified, you can associate the following optional parameters:

ELEMENT
   RESTART_ATTEMPTS
   CLEANUP_TIMEOUT
   RESTART_TIMEOUT
   READY_TIMEOUT
   TERMTYPE
   RESTART_METHOD

RESTART_ATTEMPTS(max-number,time-interval)

Specifies the maximum number of times that automatic restart management should attempt to restart the specified element within a given interval. This limit prevents automatic restart management from continually restarting an element that is recursively terminating.

When this threshold has been reached, no further restarts will be attempted and the system will deregister the element.

max-number: Must be a decimal value from 0 to 3. When RESTART_ATTEMPTS(0) is specified, no restarts will be attempted for this element. The default is 3.
time-interval: Must be a decimal number from 1 to 86400 seconds (24 hours). The default is 300.

RESTART_ATTEMPTS is optional, but if specified, must refer to a specific ELEMENT.

The default is RESTART_ATTEMPTS(3,300).

CLEANUP_TIMEOUT(time-interval)

Specifies the maximum amount of time, in seconds, that cross-system restart of an element is to be delayed waiting for member cleanup of a terminated system.

time-interval: Must be a decimal number from 120 (2 minutes) to 86400 seconds (24 hours).

When a system terminates, automatic restart management updates the state of elements owned by that terminated system to FAILED. When member cleanup for the terminated system completes, systems targeted to perform cross-system restart update the state of eligible elements to RESTARTING and perform cross-system restart processing.

When member cleanup for the terminated system does not complete within two (120 seconds) minutes, systems that are targeted to perform cross-system restart update the state of eligible elements to RESTARTING and proceed to perform cross-system restart processing by using the CLEANUP_TIMEOUT parameter to introduce additional delay time as necessary.

The potential amount of additional delay time is any amount of time specified by the CLEANUP_TIMEOUT parameter that is more than 2 minutes. When a time-interval greater than 120 is specified (or defaulted to), and additional delay is introduced to wait for member cleanup processing to complete, message IXC815I is issued to the system log to record the delay in performing restart processing.

If the member cleanup does not occur within this interval, automatic restart management will proceed with processing the element for cross-system restart. And, if CLEANUP_TIMEOUT(120) is not being used, the system issues message IXC815I to the system log to record the time out.

Note: Cross-system restart processing performs element restart processing in order of ascending numerical level within a particular restart group. Restart processing of elements with numerically higher levels will not occur until restart processing of elements with numerically lower levels completes. In other words, delaying an element for CLEANUP_TIMEOUT also delays restart processing of elements with numerically higher is optional.

CLEANUP_TIMEOUT is optional. The default is CLEANUP_TIMEOUT(300).

RESTART_TIMEOUT(time-interval)

Specifies the maximum amount of time, in seconds, that an element is expected to take to issue the IXCARM macro with REQUEST=REGISTER after automatic restart management has restarted it.

If the re-registration does not occur within this interval, automatic restart management issues message IXC803I to inform the operator of the situation and deregisters the element. Any elements that were waiting for this element to become ready will be released from their wait.

time-interval: Must be a decimal number from 1 to 86400 seconds (24 hours).

Note: Many factors can affect how long it will take for a job to restart, so this value should not be set to less than 60 seconds. If time-interval is too small, automatic restart management can deregister an element that would have re-registered successfully. When the subsequent re-registration is received, automatic restart management will treat it as an initial registration.

RESTART_TIMEOUT is optional. The default is RESTART_TIMEOUT(300). The IXCARM macro parameter that overrides the default specification is RESTARTTIMEOUT.

READY_TIMEOUT(time-interval)

Specifies the maximum amount of time, in seconds, to wait for an element being restarted to issue the IXCARM macro with the REQUEST=READY parameter. If this interval is exceeded, automatic restart management will release any elements that were waiting for this element to become ready. No message is issued for this condition and the element is not deregistered. (If a DISPLAY XCF,ARMSTATUS is done for this element, it will be in the available-to state.)

The value for time-interval must be a decimal number from 1 to 86400 seconds (24 hours).

READY_TIMEOUT is optional. The default is READY_TIMEOUT(300).

TERMTYPE(ALLTERM)

TERMTYPE(ELEMTERM)

Specifies under which conditions MVS should restart this element.

ALLTERM: Indicates that the element should be restarted for all unexpected failures as appropriate. The value specified on an IXCARM REQUEST=REGISTER request for the ELEMBIND keyword determines what types of failures are appropriate.
ELEMTERM: Indicates that the element should be restarted only if the element itself terminates. It should not be restarted if the system terminates.

TERMTYPE is optional. The default is TERMTYPE(ALLTERM). The IXCARM macro parameter that overrides the default specification is TERMTYPE.

When the element is an abstract resource (has a bind to the system on which it is registered), the following applies:

If the policy specifies TERMTYPE(ELEMTERM), the policy specification for TERMTYPE is ignored. The element will be restarted when the system fails. To suppress all restarts for an element, specify RESTART_ATTEMPTS(0).
If the policy specifies TERMTYPE(ALLTERM), the policy specification for TERMTYPE is honored. (The element will be restarted when the system fails.)

RESTART_METHOD(event,restart-type)

Specifies restart text that automatic restart management is to use to restart the element.

Specify one of the following for event:

ELEMTERM: Indicates that the persistent restart text is to be overridden by the JCL data set or the command text specified in restart-type only when the element itself terminates.
SYSTERM: Indicates that the persistent restart text is to be overridden by the JCL data set or the command text specified in restart-type only when the system the element was running on terminates.
BOTH: Indicates that the persistent restart text is to be overridden by the JCL data set or the command text specified in restart-type when either the system the element was running on terminates, or when the element itself terminates.

Specify one of the following for restart-type:

PERSIST

Indicates that MVS is to use the persistent restart text. The persistent restart text is either the JCL or the started task command that previously started the element.

JOB,'jcl-source'

Indicates that MVS is to restart the element as a batch job. 'jcl-source' is the name of the data set that contains the JCL for restarting the element. This data set name must be enclosed within single quotation marks. The data set must have the same data set characteristics (for instance, LRECL) as standard procedure libraries.

The data set name can:

Contain symbolic-substitution keywords (such as &SYSCLONE)
Be up to 72 characters in length (after symbolic-substitution keywords are resolved, the data set name cannot exceed 44 characters in length, and the member name cannot exceed 8 characters in length and must be enclosed in parentheses).

Note: Data set names are not validated when a policy is defined except as noted above. They are validated only when a restart is necessary for the element.

STC,'command-text'

Indicates that automatic restart management is to restart this element by issuing the command provided in 'command-text'. The command text:

Must be enclosed in single quotation marks
Can include symbolic substitution keywords
Length cannot exceed 126 characters
Can contain a single quotation mark(') but must be represented by specifying two single quotation marks('') in succession.
Can contain lowercase characters but will be changed to uppercase when the utility program runs. Your command text must not be dependent on lowercase characters.

Note: Command text strings are not validated when a policy is defined except as specified above. They are validated only when a restart is necessary for this element.

Note:

If the jcl-source or command-text will not fit on one line, it can be continued on another line by using single quotation marks to indicate segments. The quotation marks are not counted toward the input length. For example:
```
RESTART_METHOD(BOTH,STC,'S ELEMENT1,PARM1=ABC,'
                 'PARM2=XYZ')
```
is equivalent to specifying:
```
RESTART_METHOD(BOTH,STC,'S ELEMENT1,PARM1=ABC,PARM2=XYZ')
```
The RESTART_METHOD can be specified once for ELEMTERM and once for SYSTERM per element. (Specifying RESTART_METHOD more than once for an element when BOTH is specified results in an error.)
The RESTART_METHOD can be specified by the element restart exit, IXC_ELEM_RESTART, if one is coded, and by the STARTTXT parameter of the IXCARM macro. If more than one restart method is specified for an element, the order of priority, from highest to lowest, is:
1. Element restart exit
2. Installation-written policy (if RESTART_METHOD is specified)
3. STARTTXT parameter of IXCARM macro
4. Policy defaults
Blanks specified within quotation marks are significant.

RESTART_METHOD is optional. The default is RESTART_METHOD(BOTH, PERSIST). For started tasks and for elements that represent abstract resources, the IXCARM macro parameter that overrides the default specification is STARTTXT.

To specify the restart text for an element that has a bind to the system on which it is registered (an abstract resource) and restarts only on system termination, use one of the following:

RESTART_METHOD(SYSTERM,STC,'command-text')

where command-text is a command that causes the element to restart.

RESTART_METHOD(SYSTERM,JOB,'jcl-source')

where jcl-source is the name of the data set that contains the JCL for restarting the element.