SAzOS : Using INGTHRES and CHKTHRES to Handle Messaging THRESHOLDS

Answer

A customer wanted to be notified when they received a set number of IOS071I messages within a time period. You can define (dynamically with INGTHRES or permanently in the Policy DB) a MVSESA minor resource IOS071I and use the SAzOS thresholds to control actions based upon the number of occurrences in a number of minutes/seconds.

Here is an example of trapping the IOS071I message and setting up a SA threshold to send out a
message/email if you receive the message 5 times in 10 seconds (eg. for intrusion detection). This
example received some very good results and allowed for the flexibility of setting individual thresholds for secondary values like a UCB addresses (eg. UCB 41B8 shown below).

Here is the example used for the IOS071I messages :

1. Issue INGTHRES to define the thresholds for MVSESA.IOS071I (or define them via the PDB).
For example (PDB):

Entry Type : MVS Component PolicyDB Name : TIVPLEX_SAZOS
Entry Name : MVS_COMPONENTS Enterprise Name : TIVPLEX_SAZOS
Resource : MVSESA.IOS071I

Critical Number. . . . 5 (1 to 50)
Critical Interval. . . 00:05:00 (hh:mm:ss or hhmmss, 00:00:01 to
24:00:00)
Frequent Number. . . . 2 (1 to 50)
Frequent Interval. . . 00:10:00 (hh:mm:ss or hhmmss, 00:00:01 to
24:00:00)
Infrequent Number. . . 1 (1 to 50)
Infrequent Interval. . 00:30:00 (hh:mm:ss or hhmmss, 00:00:01 to
24:00:00)

INGTHRES (to view flag):
Cmd System Resource Critical Frequent Infrequent
--- -------- ---------------- ----------- ----------- -----------
sysn MVSESA.IOS071I 5 in 00:05 2 in 00:10 1 in 00:30

2. Create an error by issuing 'CHKTHRES MVSESA.IOS071I.41B8 MVSESA NEW'

Should result in:
AOF503I : RECOVERY FOR MINRES MVSESA.IOS071I.41B8 CONTINUING - 1 ERRORS
SINCE hhmmss ON ddmmyyyy - INFREQUENT ERROR THRESHOLD EXCEEDED

3. Review by issuing 'CHKTHRES MVSESA.IOS071I.41B8 MVSESA CHECK'

This will echo back the error status (1 error found);
AOF503I : RECOVERY FOR MINRES MVSESA.IOS071I.41B8 CONTINUING - 1 ERRORS
SINCE hhmmss ON ddmmyyyy - INFREQUENT ERROR THRESHOLD EXCEEDED

4. Repeat step (2) two more times to replicate additional occurences of the messages taking place for this UCB.

5. Once you have hit the CRIT threshold (5 in our case), you will see:

AOF501E : RECOVERY FOR MINRES MVSESA.IOS071I.41B8 HALTED - 5 ERRORS
SINCE hhmmss ON ddmmyyyy - CRITICAL ERROR THRESHOLD EXCEEDED

You can then trap on the AOF501E message to know when the critical threshold has been met.

You should be able to modify the above example for any message, eg. ABC1234I. The error is created by issuing something like the following:
'CHKTHRES MVSESA.ABC1234I.jobname MVSESA NEW'

Issue this command 5 times in 15 minutes to cause a critical threshold to be reached and to generate a AOF501E message. Trap the AOF501E message in the MAT to issue an alert or error message to your operators.

One key peice in our automation is to setup a trap in our automation table for the ABC1234I message and to run a rexx program to issue the actual CHKTHRES command to increment the current threshold.
For example :
IF MSGID='ABC1234I' & JOBNAME = SSID & TEXT = MESSAGE THEN
THEN EXEC (CMD('yourrexx 'SSID) ROUTE (ONE AUTO1))
DISPLAY(Y) NETLOG(Y);

Your local rexx "yourrexx" program can review the content of the message or use the passed JOBNAME, for example JOBA. If the JOBNAME is JOBA
it will issue the CHKTHRES command, such as ...
CHKTHRES MVSESA.ABC1234I.JOBA MVSESA NEW

Here the JOBNAME is the secondary value. At any time you can issue the CHECK parm to find out what the current count is, for example :
CHKTHRES MVSESA.ABC1234I.JOBB MVSESA CHECK