Use a DatabaseInput node to respond to events in a database. For example, the broker can keep an external system synchronized with a database by sending updates to the target system whenever data is changed in the database.
The database must record the fact that data has changed in an event store, which is typically a database table. The event store is not the same as the application data. The following diagram shows the interaction between the database, event store, and the broker.
The following diagram shows how the DatabaseInput node works.
When the process starts, ReadEvents checks the event store for new events, which are then used by BuildMessage to build the message. This message is propagated to the message flow and then EndEvent updates the event store to ensure that the event cannot be processed again. When all events have been processed, the broker calls ReadEvents to retrieve any events that have been added since the previous check. If the event store is empty, the broker waits until the polling interval has expired, and then calls ReadEvents again. To avoid contention, the check of the event store is single-threaded.
For each event that is read by ReadEvents, BuildMessage builds the message that is propagated to the message flow. Building the message typically uses the event data to look up data in the application table. The data from the application table is then used to construct the message. When BuildMessage ends, the message is automatically propagated to the message flow. When the message is propagated, the broker starts any downstream nodes that are required to process the message.
After the message has been propagated to the message flow, EndEvent updates the event store to ensure that the event that has just been processed cannot be processed again.
The detailed operation of ReadEvents, BuildMessage, and EndEvent is controlled by ESQL code. The DatabaseInput node contains an ESQL module with sample code and comments, which you must modify to suit your requirements. ReadEvents populates a NewEvents structure with event data that is read from the event table. Each event data row that is generated has a Key and a Usr field name; NewEvents.Event[].Key and NewEvents.Event[].Usr. The Key field contains a unique key for an event, and is used to prevent duplicate event processing. The Usr field is typically a subtree that contains user-defined data that is associated with an event. For information about modifying the ESQL, see Configuring a DatabaseInput node.
The processes that are completed by the DatabaseInput node are split across separate transactions. A new transaction is started when ReadEvents starts. When ReadEvents ends, this transaction is committed and new events are marked for processing. By committing this transaction, any locks put on the database by ESQL code that is run from ReadEvents are released. Then, for each event received by BuildMessage, a new transaction is started. This new transaction is committed after EndEvent finishes.
To scale a DatabaseInput node for many events, change the Additional instances property on the Instances tab from its default value of 0 to the number of instances that you require. If you are using additional instances, the database must be configured so that multiple applications can read different rows from the application table at the same time. ReadEvents always runs in single-threaded mode to avoid database contention, even if you use additional instances. To improve performance, ReadEvents can read multiple events each time it runs, and these events can be processed at the same time by multiple instances of BuildMessage. The event store must have a primary key, which ReadEvents uses to identify events that are currently being processed. You do not have to write the ESQL in ReadEvents to filter out events that are currently being processed by the message flow.
If 'Failure' is selected, the event is moved to the 'Failed' state and the transaction is rolled back. Failed events are detected and dealt with before dispatching any Ready events. The failed event is propagated to the Failure terminal, if wired, with an exception list.
For 'Short retry then failure', the message flow performs a number of retries equal to the Retry Threshold property, propagating the message down the Out terminal with an interval between each retry equal to the 'Short retry interval'. If the retry threshold is met without successful completion of the transaction, the message is then propagated to the Failure terminal.
For 'Short then long retry', the retry logic is the same as 'short retry then failure' up to the point that the retry threshold has been met. Then, instead of ending the retry attempts and propagating to the failure terminal, it will continue to retry indefinitely but now with a delay between each retry equal to the 'Long retry interval'.
The EndEvent operation is not automatically invoked if the transaction has been rolled back, or if the message has been propagated to the failure terminal. In these circumstances it is up to the user to remove or update the event in the event store if they wish to prevent it from being re-processed.
The EndEvent operation is automatically invoked if the transaction completes successfully, including when an exception has occurred downstream of the Database Input node but that exception has been handled successfully down a Catch terminal.