IZ69780: DEADLOCKING ON CRONTASKS OCCURS ON LOW INTERVALS AND DBCONNECTION WATCHDOG SETTINGS

APAR status

• Closed as program error.

Error description

• ****Overview
  Client is running a number of Maximo crontasks and finding
  deadlock is occurring more frequently when some of the crontasks
  intervals are low (30sec).
  The problem seems to be related to dbconnection watchdog, but
  different to the issue 09-13851 already logged.
  
  
  ****Development reply after receiving GBS testing:
  
  It is indeed a different bug though very much related.
  Please enter an APAR for it.
  However, it is not easy to fix.
  Basically it is all because of DBconnectionwatchdog, so setting
  the DBConnectionWatchdog to WARN or ERROR should avoid the
  problem.
  
  
  ****Steps to replicate:
  
  1. Goto crontask setup
  2. Make a few crontask active and frequency interval to 30 sec.
  3. Lets these crontask run
  
  Result eventually Maximo should experience a deadlock occurring
  and the application stops functioning.
  The client generally had a few escalation,MEA related crontasks
  polling around 30 sec.
  
  ****Required
  
  Need to alter the logger to do with DBCONNECTIONWATCHDOG to
  either WARN or ERROR in the logging application.
  
  
  
  
  ****Client Environment:
  AIX 5.3 TL9
   Hostname: eamdap01 (development)
   Maximo Base Services 7.1.1.5 Build 20090627-0754 DB Build
  V7115-149
   WAS 6.1 (Java 1.5 SR6B)
  Oracle 10g on other Host
  
  
  ****GBS testing:
  
  1) Left all the crontasks as per the modified schedules (New
  schedules), and set the SQL and DBCONNECTION loggers to ERROR
   Result: No Dead Lock
  
  2) Left all the crontasks as per the modified schedules (New
  schedules), and set the SQL and DBCONNECTION loggers to DEBUG
   Result: No Dead Lock.
  
  3) Changed all the crontasks to have previous schedules (Set the
  crontask to run at very low intervals. Several crontasks at 30
  sec frequency.), and set the SQL and DBCONNECTION loggers to
  ERROR
   Result: No Dead Lock
  
  4) Changed all the crontasks to have previous schedules (Set the
  crontask to run at very low intervals. Several crontasks at 30
  sec frequency), and set the SQL and DBCONNECTION loggers to
  DEBUG
   Result: Dead Lock occurs and was able to consistently recreate
  the problem.
   The logs are similar to that of the logs that we provided
  earlier. So no more attaching the logs to the email.
  
  5) Changed all the crontasks schedules to run at a much lower
  frequency than test 4, and set the SQL and DBCONNECTION loggers
  to DEBUG / ERROR (tried both)
   Result: Dead Lock occurs and not even able to start the server.
   Not able to extract the logs, as the server is not started the
  processid cannot be verified. But has the maximo logs.
  
  
  ****Client information:
  
  Crontask/Escalation Instance name  Previous Frequency  Modified
  Frequency Expected Schedule for Day 1 of Production. Crontask /
  Escalation detail or purpose
  ESCBLTNEXP  24Hrs 24Hrs 24Hrs Escalation to delete expired
  bulletin messages
  HACTL_SR_TO_WO  30Sec 2Min 2Min Escalation to synchronize status
  between the workorder and its originator SR. This esclation has
  6 Escalation points
  ESCAUTOEXPIRERO  5min 4Hrs 4Hrs Esclation to cancel the RO, if
  no issue transa ction after 7 days of approval
  WOESC  30Sec 5Min 1Min This escalation has 3 escalation points.
  Handles transfer of workorder across departments. Also changes
  the status of workorder to inprogress when labor is assigned
  HACTL_WOCLOSE  5min 4Hrs 4Hrs Escalation to close workorder if
  status='COMP' and changedate <= sysdate-14
  HACTL_WOWFINITIATE  2min 5min 2min Escalation to initiate
  default workflow on the workorders created by SR
  HACTL_PMWO  2min 5min 5min Escalation to change the status of
  workorder to ACTIVE (WAPPR) when schedule is reached
  ESCAUTOCLOSERO  5min 4Hrs 4Hrs Escalation to change the status
  of RO to CLOSE
  ESCAUTOCLOSEPARTIALRO  2min 5Min 5Min Escalation to close the
  Partially issued RO
  WOLCSUPDATE  30Sec 30Sec 30Sec Escalation to indicate the kind
  of Operation for LCS interface
  HACTL_CANCELTASK  30Sec 2min 30Sec Escalation to cancel task
  workorder
  REORDERCRONTASK  24Hrs 24Hrs 24Hrs Reorder Crontask
  PMWOGenCronTask  1hr 24Hrs 24Hrs PMWOGen Crontask
  KPICrontask  24Hrs 24Hrs 24Hrs KPICrontask
  JMSSEQCONSUMER  30Sec 30Sec 30Sec 4 instances of crontask
  IFACETABLECONSUMER  30Sec 30Sec 30Sec 1 instance of crontask
  BBCron  1hr 1hr 1hr Updates Bulletin board cache
  REPORTLOCKRELEASE  1min 1min 1min Report Queue Lock Release Task
  REPORTUSAGECLEANUP  30min 30min 30min Report Usage Log Cleanup
  Task
  XMLFILECONSUMER  30Sec (8 instances at 30 Sec each) 30Sec(8
  instances at 30 Sec each) 30Sec(8 instances at 30 Sec each) XML
  File inbound processing through crontask, 6 instances of
  crontask each running at 30 Sec interval
  VMMSync  2min 2min 2min VMM Sync crontask
  LCS Crontask (For Inbound processing)  30Sec (2 instances at 30
  Sec each) 30Sec (2 instances at 30 Sec each) 30Sec (2 instances
  at 30 Sec each) This is a crontask used to poll on BPIP (BPIP is
  SUN MQServer, Here in hactl environment any application should
  poll on the bpip queues to get the message and send the message
  to bpip queues to send the message). This has 2 instances each
  running at 30 Sec interval
  LCS _Outbound (For Outbound processing)  30Sec(2 instances at 30
  Sec each) 30Sec(2 instances at 30 Sec each) 30Sec(2 instances at
  30 Sec each) Has 2 instances each running at 30 Sec interval
  HRIS_BPIPCron(For inbound processing)  30Sec 30Sec 30Sec 1
  instance of crontask
  LaborAssingment  15Sec 2min 2min This crontask sends messages to
  Pagers for the labor assigned on different workorder. This
  custom crontask calls the shell scipt on the AIX box. The email
  system in hactl environment is not able to process the message
  sent from Maximo. So have to call a shell script which executes
  the AIX email send function to send an email.
  PS_Outbound  30Sec 30Sec 30Sec This is a crontask used to poll
  on BPIP (BPIP is SUN MQServer, Here in hactl environment any
  application should poll on the bpip queues to get the message
  and send the message to bpip queues to send the message).
  PS_InBound  30Sec 30Sec 30Sec This is a crontask used to poll on
  BPIP (BPIP is SUN MQServer, Here in hactl environment any
  application should poll on the bpip queues to get the message
  and send the message to bpip queues to send the message).
  

Local fix

• none, not working. Switch off dbconnectionwatchdog
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All                                          *
  ****************************************************************
  * PROBLEM DESCRIPTION: When dbconnectionwatch dog is enabled,  *
  *                      deadlock may occur. The work around is  *
  *                      to disable db conneciton watch dog.     *
  *                                                              *
  ****************************************************************
  * RECOMMENDATION:                                              *
  *                                                              *
  *                                                              *
  *                                                              *
  ****************************************************************
  Deadlock when dbconnectionwatch dog is enabled.
  

Problem conclusion

• Code change
  
  
  
  
  
  The fix for this APAR is contained in the following maintenance
  package:
  
    | release\fix pack | REL 7.1.1.7 - BS
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• MAXIMO
  

Fix information

• Fixed component name

  MAXIMO SYS CONF

• Fixed component ID

  5724R46S1

Applicable component levels

• R710 PSY

     UP