Troubleshooting an event rule that does not trigger the required action

You have created an event rule, but the required action is not triggered when the event condition is encountered.

Cause and solution:

The cause and subsequent solution might be any of a number of things. Use the following checklist and procedures to determine what has happened and resolve the problem. The checklist uses a test event which has the following characteristics:

<eventRule name="TEST1" ruleType="filter" isDraft="no">
		<description>A Rule that checks the sequence of events</description>
		<eventCondition name="fileCreated1" eventProvider="FileMonitor"
		eventType="FileCreated">
		<scope>
				C:\TEMP\FILE5.TXT ON CPU_MASTER
				</scope>
				<filteringPredicate>
				<attributeFilter name="FileName" operator="eq">
				<value>c:\temp\file5.txt</value>
		</attributeFilter>
		<attributeFilter name="Workstation" operator="eq">
		<value>CPU_MASTER</value>
		</attributeFilter>
		<attributeFilter name="SampleInterval" operator="eq">
		<value>60</value>
		</attributeFilter>
		</filteringPredicate>
		</eventCondition>
		<action actionProvider="TWSAction" actionType="sbj" responseType="onDetection">
		<scope>
				SBJ CPU_MASTER#JOB1 INTO CPU_MASTER#JOBS
				</scope>
				<parameter name="JobUseUniqueAlias">
				<value>true</value>
		</parameter>
		<parameter name="JobDefinitionWorkstationName">
		<value>CPU_MASTER</value>
		</parameter>
		<parameter name="JobDefinitionName">
		<value>JOB1</value>
		</parameter>
		</action>
		</eventRule>

The checklist is as follows:

Step 1: Is event management enabled?

Check if the event management feature is enabled (at installation it is enabled by default):

Run the following command:
```
optman ls
```
and look for the following entry:
```
enEventDrivenWorkloadAutomation / ed = YES
```
If the value is "YES", go to Step 2: Is the workstation enabled for event processing?.
Action: If the property is set to NO, run the command:
```
optman chg ed=YES
```
To effect the change, run:
```
JnextPlan –for 0000
```
Check that the event rule is now being processed correctly. If not, go to Step 2: Is the workstation enabled for event processing?.

Step 2: Is the workstation enabled for event processing?

Check that the workstation is enabled for event processing. By default the master domain manager and backup master domain manager are enabled for event processing, but the default value might have been changed. Perform the following steps:

View the localopts file on the master domain manager with a text editor or viewer, and check for the following entry:
```
can be event processor = yes
```
If the value is "yes", go to Step 3: Is the event processor installed, up and running, and correctly configured?.
Action: If the value is "no", set it to "yes". Save the localopts file and stop and start IBM Workload Scheduler. Check that the event rule is now being processed correctly. If not, go to Step 3: Is the event processor installed, up and running, and correctly configured?.

Step 3: Is the event processor installed, up and running, and correctly configured?

Start conman.

Issue the showcpus command:

%sc @!@

You will see the following output:

CPUID        RUN NODE        LIMIT FENCE DATE     TIME  STATE       METHOD   DOMAIN
CPU_MASTER    11 *WNT  MASTER    0     0 09/03/19 09:51    I JW MDEA         MASTERDM
FTA1          11  WNT  FTA       0     0                 LT                  MASTERDM

Check the STATE field for the presence of an M, a D, and an E (uppercase). In the example, the STATE field has a value of I JW MDEA, and the MDE is highlighted. If all are present, the event processor is installed, up and running, and correctly configured; go to Step 9: Is the SSM agent running (for rules with FileMonitor plug-in-related events only?).
Actions: If one or more of M, D, and E are not present, perform one or more of the following actions until they are all present:
The STATE field has neither an uppercase E nor a lowercase e

If there is neither an uppercase E nor a lowercase e, the event processor is not installed. The event processor is installed by default on the master domain manager and backup master domain manager. If you are working on either, then the installation did not complete correctly. Collect the log files in the following paths:

Windows operating systems

<TWA_home>\TWS\stdlist\logs

UNIX operating systems

TWA_DATA_DIR/stdlist/logs/

and contact customer support for assistance.

The STATE field has a lowercase e
If the STATE field has a lower case e, the event processor is installed but not running. Start the event processor using the conman startevtproc command, or the Dynamic Workload Console. If you use conman, for example, you will see the following output:
```
%startevtproc
AWSJCL528I The event processor has been started successfully.
```
The STATE field has no M
If the STATE field has no M, monman is not running. Start monman using the conman startmon command. You will see the following output:
```
%startmon
AWSBHU470I A startmon command was issued for CPU_MASTER.
```
The STATE field has no D

If the STATE field has no D, the current monitoring package configuration is not deployed. Go to Step 5: Has the rule been added to the monitoring configuration on the workstation?.
Rerun the showcpus command.
When the M, D, and E are all present, check that the event rule is now being processed correctly. If not, go to Step 9: Is the SSM agent running (for rules with FileMonitor plug-in-related events only?).

Step 4: Is the workstation definition present in the plan?

Start conman.
Issue the showcpus command:
```
%sc @!@
```
If the workstation definition is not included in the plan, add it by generating the production plan with the JnextPlan - for 00 command. If the workstation definition is included, go to Step 5: Has the rule been added to the monitoring configuration on the workstation?.

Step 5: Has the rule been added to the monitoring configuration on the workstation?

Check if the rule is present in the workstation monitoring configuration by running the conman showcpus command with the ;getmon argument:

%sc ;getmon
Monitoring configuration for CPU_MASTER:
********************************************
***  Package date : 2019/09/03 07:48 GMT ***
********************************************

TEST1::FileMonitor#FileCreated:C:\TEMP\FILE5.TXT ON CPU_MASTER; 
TEST1::TWSObjectsMonitor#JobSubmit:* # * . TEST*;

If the rule is present, go to Step 7: Has the new monitoring configuration been deployed to the workstation?.

Action: If the configuration does not contain the expected rule, go to Step 6: Is the rule active.

Step 6: Is the rule active

If the configuration does not contain the expected rule, check if it is active.

Check the rule status, using the composer list command or the Dynamic Workload Console. For example, if you use composer you will see the following output:

-list er=@

Event Rule Name                 Type       Draft  Status           Updated On  Locked By
------------------------------  ---------  -----  ---------------  ----------  ----------------
TEST1                          filter        N     active           	09/03/2019  -

If the rule is in active status go to Step 7: Has the new monitoring configuration been deployed to the workstation?.

Action: If the rule is in error status, activate the IBM Workload Scheduler trace, collect the log files located in

Windows operating systems

<TWA_home>\TWS\stdlist\logs

UNIX operating systems

TWA_DATA_DIR/stdlist/logs/

and contact customer support for assistance.

Step 7: Has the new monitoring configuration been deployed to the workstation?

If the rule is active, check if the new monitoring configuration has been deployed to the workstation.

The deployment of a new monitoring configuration can be checked in either of these ways:
- Check if the configuration is present in the following paths:
  
  On Windows systems
  
  <TWA_home>/TWS/monconf
  
  On UNIX systems
  
  <TWA_DATA_DIR>/monconf
- Check in the messages.log file located in
  
  On Windows systems
  
  <TWA_home>\stdlist\appserver\engineServer\logs
  
  On UNIX systems
  
  <TWA_DATA_DIR>/stdlist/appserver/engineServer/logs
  
  Look for the message:
```
[9/3/19 9:50:00:796 CEST] 00000020 sendEventReadyConfiguration(wsInPlanIds, zipsToDeploy)
                          AWSDPM001I The workstation "CPU_MASTER" has been notified about
                          a new available configuration.
```
  If the message is present for the workstation in question after the time when the rule was made available for deployment, then the new configuration has been deployed.
If the configuration has been deployed, go to Step 8: Has the deploy of the new monitoring configuration worked correctly?.
Action: If the configuration has not been deployed, deploy it with the conman deploy command:
```
%deploy
AWSBHU470I A deployconf command was issued for MASTER_CPU.
```
Check that the event rule is now being processed correctly. If not, go to Step 8: Has the deploy of the new monitoring configuration worked correctly?.

Step 8: Has the deploy of the new monitoring configuration worked correctly?

If the new monitoring configuration has been deployed, check that the deployment was successful:

Check in the <date>_TWSMERGE.log in the following paths:

Windows operating systems

<TWA_home>\TWS\stdlist\traces

UNIX operating systems

TWA_DATA_DIR/stdlist/traces/

and look for the most recent occurrence of these 2 messages:
```
09:51:57 03.09.2019|MONMAN:INFO:=== DEPLOY ===> CPU_MASTER has been notified 
                                of the availability of the new monitoring configuration.
09:51:57 03.09.2019|MONMAN:INFO:=== DEPLOY ===> The zip file d:\TWS\twsuser\monconf\deployconf.zip 
                                has been successfully downloaded.
```
If you find these messages, referring to the workstation in question, and occurring after the time when the rule was deployed, then the rule has been successfully deployed to the workstation: go to Step 9: Is the SSM agent running (for rules with FileMonitor plug-in-related events only?).
Actions: If you find messages that indicate an error, follow one of these actions:
Message indicates that the server could not be contacted or that the action has been resubmitted by monman
You find one of the following messages:
```
=== DEPLOY ===> ERROR contacting the server for receiving the zip file (rc=8)

=== DEPLOY ===> The deploy action has been automatically resubmitted by monman.
```
The application server could be down. Either wait for 5 minutes, or see Application server - starting and stopping.
If you need to change any aspect of the application server configuration, run JnextPlan –for 0000.

When you are certain that the application server is up, retry Step 8: Has the deploy of the new monitoring configuration worked correctly?.
Message indicates a problem with decoding or extracting the compressed file
You find one of the following messages:
```
=== DEPLOY ===> ERROR decoding the zip file temporarily downloaded in
	<TWA_home>\TWS\monconf

=== DEPLOY ===> ERROR unzipping the zip file <file_name>
```
Collect the log files and contact customer support for assistance.

Step 9: Is the SSM agent running (for rules with FileMonitor plug-in-related events only?)

If the rule has an event that uses the FileMonitor plug-in, check that the SSM Agent is running. Check in the log that when the conman startmon command was run (either when you ran it manually or when IBM Workload Scheduler started).
Search in the log for the following message:
```
11:13:56 03.09.2019|MONMAN:INFO:SSM Agent service successfully started
```
If it is present, or the rule does not use the FileMonitor plug-in, go to Step 6: Is the rule active.
Action: If the SSM Agent message is not present, collect the log files. Log files are located in
Windows operating systems
<TWA_home>\TWS\stdlist\logs
```
<TWA_home>/ssm
```
UNIX operating systems
TWA_DATA_DIR/stdlist/logs/
```
<TWA_DATA_DIR>/EDWA/ssm
```
and contact customer support for assistance.

Step 10: Have the events been received?

You know the rule has been deployed, but now you need to know if the event or events have been received.

Check in the SystemOut of the server to see if the event has been received. The output is different, depending on the type of event:

FileMonitorPlugIn event

Following is the output of a FileMonitorPlugIn event:

[9/3/19 9:55:05:078 CEST] 00000035 EventProcessor A com.ibm.tws.event.EventProcessorManager 
                          processEvent(IEvent) 
                          AWSEVP001I The following event has been received: 
                          event type = "FILECREATED"; event provider = "FileMonitor"; 
                          event scope = "c:\temp\file5.txt on CPU_MASTER". 
                          FILECREATED FileMonitor c:\temp\file5.txt on CPU_MASTER

If the event has been received, go to Step 11: Has the rule been performed?.

If the event has not been received check if it has been created by looking in the traps.log for the message that indicates that the event has been created:
```
.1.3.6.1.4.1.1977.47.1.1.4.25 OCTET STRING FileCreatedEvent event
```
Action: Whether the event has or has not been created, collect the information in
Windows operating systems
```
<TWA_home>/ssm
```
UNIX operating systems
```
<TWA_DATA_DIR>/EDWA/ssm
```
and contact customer support for assistance.

TWSObjectMonitorPlugIn event

Following is the output of a TWSObjectMonitorPlugIn event:

[9/3/19 12:28:38:843 CEST] 00000042 EventProcesso A com.ibm.tws.event.EventProcessorManager 
                           processEvent(IEvent) 
                           AWSEVP001I The following event has been received: event type = "JOBSUBMIT"; 
                           event provider = ""TWSObjectsMonitor""; event scope = "CPU_MASTER # JOBS . 
                           (CPU_MASTER #) TEST". JOBSUBMIT "TWSObjectsMonitor" CPU_MASTER # JOBS . 
                           (CPU_MASTER #) TEST

Action: If the event has not been received, collect the log data and contact customer support for assistance.

If the TWSObjectMonitorPlugIn event has been received, check in the same log that the EIF event has been sent. Following is the output of an EIF event:

12:27:18 03.09.19|MONMAN:INFO:Sending EIF Event: 
                 "JobSubmit;
                  TimeStamp="2019-09-03T12:26:00Z/";
                  EventProvider="TWSObjectsMonitor";
                  HostName="CPU_MASTER";
                  IPAddress="9.71.147.38";
                  PlanNumber="11";
                  Workstation="CPU_MASTER";
                  JobStreamWorkstation="CPU_MASTER";
                  JobStreamId="JOBS";
                  JobStreamName="JOBS";
                  JobStreamSchedTime="2019-09-03T12:26:00";
                  JobName="TEST";
                  Priority="10";
                  Monitored="false";
                  EstimatedDuration="0";
                  ActualDuration="0";
                  Status="Waiting";
                  InternalStatus="ADD";
                  Login="twsuser";END

If the EIF event has been sent, it might be cached in the following path:

On Windows operating systems

<TWA_home>/EIF

On UNIX operating systems

<TWA_DATA_DIR>/EIF
If the event is found there, check the communication with the agent and the server. If no communication problem is present wait until the event is sent.
The event might also be cached in the machine where the event processor is located. Check this in the following paths:

On Windows operating systems

<TWA_home>TWS\stdlist\appserver\engineServer\temp\TWS\EIFListener/eif.templ

On UNIX operating systems

<TWA_DATA_DIR>/stdlist/appserver/engineServer/temp/TWS/EIFListener/eif.templ

If the event is found there, check the communication with the agent and the server. If no communication problem is present wait until the event is sent.

Action: If the problem persists, collect the log data and contact customer support for assistance.

Step 11: Has the rule been performed?

You now know that the event has been received, but that the action has apparently not been performed.

Check in the SystemOut of the server to see if the rules have been performed. Look for messages like these:

[9/3/19 9:55:05:578 CEST] 00000035 ActionHelper  A com.ibm.tws.event.plugin.action.ActionHelper 
                          invokeAction(ActionContext,Map,EventRuleHeader) 
                          AWSAHL004I The rule "TEST1" has been triggered. TEST1
[9/3/19 9:55:05:625 CEST] 00000036 ActionHelper  A com.ibm.tws.event.plugin.action.ActionHelper 
                          AsynchAction::run() 
                          AWSAHL002I The action "sbj" for the plug-in "TWSAction" has been started. 
                          sbj TWSAction
[9/3/19 9:55:06:296 CEST] 00000036 ActionHelper  A com.ibm.tws.event.plugin.action.ActionHelper 
                          AsynchAction::run() 
                          AWSAHL003I The action "sbj" for the plug-in "TWSAction" has completed. 
                          sbj TWSAction

If the rule has been triggered and the action completed, go to Step 12: Is the problem in the visualization of the event?.

Action: If the action has not been completed collect the log data and contact customer support for assistance.

Step 12: Is the problem in the visualization of the event?

Action: If the event has been received, but you cannot see it, there might be a problem with the console you are using to view the event. See Troubleshooting Dynamic Workload Console problems.