PI44032: IN A WORKLIGHT CLUSTERED SETUP ,AN EVENTSOURCE MAY BE FIRED ON MORE THAN 1 SERVER.

IBM MobileFirst Platform Foundation Interim Fix README for 7.0
IBM MobileFirst Platform Foundation Interim Fix README for 6.3
IBM Worklight Interim Fix README for 6.2.0.1
IBM Worklight Interim Fix README for 6.1.0.2
IBM Worklight Interim Fix README for 6.0.0.2

APAR status

• Closed as program error.

Error description

• There is no special error in the logs, however, the polling
  method assumes uniqueness of its operation and might have
  strange applications results if run twice simulatneously. For
  example, a user may receive duplicate push notifications should
  that occur.
  Polling is used widely in Worklight for push notifications. For
  example, a user may write an adapter with the following JS code:
  WL.Server.createEventSource({
      name: 'Source2',
      poll: {
          interval: 40, // seconds
          onPoll: 'getNotificationFromBackend'
      }
  });
  
  
  function getNotificationFromBackend() {
      WL.Logger.info('sending notifications now');
     // will iterate here over a list of users and will send a
  notification to each user...
  }
  

Local fix

• Reduce the number of active Worklight servers to one.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Worklight servers which are configured as a cluster or farm  *
  * such that there are several Worklight servers sharing        *
  * identical configuration (database,Worklight WAR etc). If     *
  * only one server is used for Worklight this problem will not  *
  * occur.                                                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * A race condition may occur when two or more Worklight        *
  * servers are processing a Worklight event source. This race   *
  * condition was not handled well, and, as a result, the event  *
  * source may be fired simultaneously in more than one server.  *
  * This can have many side effects when using EventSource for   *
  * mass push notifications, including high CPU utilization and  *
  * significant Worklight database activity.                     *
  * The other side effect with push and event sources is that    *
  * many mobile users may receive multiple notifications         *
  * (depending upon the cluster size and the race condition)  on *
  * their mobile devices. Each time the user opens the push      *
  * notification, it will trigger a login process which will     *
  * probably add extra traffic to the Worklight servers (which   *
  * already may be in a high CPU utilization state).             *
  * An example for a Worklight adapter that may trigger this     *
  * race condition is the following adapter Java script code:    *
  *                                                              *
  * WL.Server.createEventSource({                                *
  *     name: 'Source2',                                         *
  *     poll: {                                                  *
  *         interval: 40, // seconds                             *
  *         onPoll: 'getUsersFromBackend'                        *
  *     }                                                        *
  * });                                                          *
  *                                                              *
  *                                                              *
  * function getUsersFromBackend() {                             *
  *     WL.Logger.info(' getUsersFromBackend');                  *
  *     // here the code will probably iterate through a list of *
  * users and will send each one a notification                  *
  *     // or some other long operation.                         *
  *     // if you notice the INFO log line in more than one      *
  * server : then you have hit the race condition.               *
  * }                                                            *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * -                                                            *
  ****************************************************************
  

Problem conclusion

• A cluster syncronization issue causing an event source method to
  be run simultaneously on multiple servers has been corrected.
  By design, an event source's poll methods are meant to be run
  only in one server within a cluster.
  

Temporary fix

• A temporary fix can be to shutdown all Worklight servers in the
  cluster except for one. However, a single server may not hold
  the standard traffic (depending on the total mobile network
  throughput.)
  In some cases the way to recover from the race condition is to
  reboot the Worklight servers.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  MFPF/WORKLIGHT

• Fixed component ID

  5725I4301

Applicable component levels

• R506 PSY

     UP