Implementing Guaranteed Event Storage in Optimize

Edit online

About this task

Guaranteed event storage functionality in Optimize uses a custom Monitor API (MAPI) configuration and batching techniques to deliver events and provide for enhanced notification of event delivery problems. When implemented appropriately, this functionality supports robust event delivery and, in the event of an event delivery problem, helps users correct problems before they cause significant data corruption. Also, it enables you to configure your system to persist cached event data to disk. Note that for most Optimize configurations, there is a trade-off between event delivery integrity and performance; there are several variables involved in this trade-off, and some experimentation may be necessary to achieve the desired balance for your system.

This feature includes two customer-configurable components that support guaranteed event storage.

  • Setting a MAPI flag to support synchronous event delivery to Analytic Engines.
  • Setting System Properties that determine whether system caches are configured for high event delivery integrity (persistence) or high performance.
Note: The Analytic Engine is designed to recover from Broker connectivity and database connectivity issues. There is no need to restart the Analytic Engine when a database and/or Broker connectivity issue arises. Once the connectivity issue is addressed, Analytic Engine will continue to process data as expected.

Configuring the MAPI Flag

Edit online

About this task

By default, events in Optimize are published by the Monitor API (MAPI) and handled asynchronously. This method provides optimal performance, but it means that users may not receive immediate notification of event publication errors. Using the flag described herein, you can configure the Optimize MAPI service to publish events synchronously. If an event delivery problem occurs in this configuration, errors are generated immediately.

The -Dsag.opt.mapi.sync.event.flag system property setting in the Integration Server startup file enables synchronous event delivery in all the components that use MAPI client mechanism running in the same Java Virtual Machine (JVM). It is important to remember that this flag is not limited to a single component; it can be set for any client JVM that uses the MAPI jar file.

Note that, depending on your system configuration, there may be multiple MAPI instances present in a system or JVM. In most cases, you can simply set the MAPI flag in the startup script for Integration Server or Infrastructure Data Collector or the custom build client application, etc.

To configure the MAPI flag on an Optimize system

Procedure

  1. Open the Integration Server startup file located in the Integration Server_directory/bin folder under your installation directory. On a Windows system, the file is named server.bat, while on a Unix system, it is named server.sh.
  2. Add the following line to the “run integration server” section of the startup file. 
    -Dsag.opt.mapi.sync.event.flag=y

    Here’s an example of the flag added to a generic startup file:

    SET JAVA_OPTS=-Dsun.lang.ClassLoader.allowArraySyntax=true 
-Dwatt.server.prepend.classes=%PREPEND_SYSTEM_CLASSPATH%;%APPEND_SYSTEM_CLA                      
SSPATH%;%PREPENDCLASSES% -Dwatt.server.append.classes=%APPENDCLASSES% 
-Dcom.webmethods.sc.auditing.Use80TableFormat=true 
-Djavax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom=com.sun. 
org.apache.xpath.internal.jaxp.XPathFactoryImpl 
-Dtangosol.coherence.shutdownhook=false -Dsag.opt.mapi.sync.event.flag=y 
%DEBUG_JAVA_OPTS% %9
  3. Save the file and close it.
  4. Repeat the preceding steps for any other applicable components in your system.

Disabling the MAPI Flag

Edit online

About this task

If you want to disable the MAPI flag, you must remove the event flag line from the configuration file.

Note that, depending on your system configuration, there may be multiple MAPI instances and flags present in a system or JVM. For most typical systems, there will be a MAPI flag in the startup script for Integration Server or Infrastructure Data Collector or the custom build client application, etc.

To disable the MAPI flag on an Optimize system

Procedure

  1. Open the Integration Server startup file located in the Integration Server_directory/bin folder under your installation directory. On a Windows system, the file is named server.bat, while on a Unix system, it is named server.sh.
  2. Locate and delete the following line from the “SET_JAVA_OPTS=-” section of the startup file. 
    -Dsag.opt.mapi.sync.event.flag=y
  3. Save the file and close it.
  4. Repeat the preceding steps for any other applicable components in your system.

Setting System Cache Properties

Edit online

About this task

This section explains how to set system cache properties to maximize event data integrity on an Optimize system. By default, the Optimize Analytic Engine caches event data in memory, which means that, in some cases, data can be lost if system components terminate or experience hardware failure. To provide enhanced event delivery integrity, this property enables you to configure an Optimize Analytic Engine to cache event data to disk so that event data is persisted even if a system component crashes or shuts down for some reason.

Note that disk caching can entail a significant performance penalty on some systems, especially those that process a large number of events; so making decision about caching options is often a matter of balancing performance vs. event integrity. When cache properties are configured for guaranteed event delivery, performance will be degraded to some degree, according to the specifics of your data and system configuration.

Basic cache configuration properties for the Analytic Engine are set using two parameters in the optimize\analysis\conf\wrapper.conf file. The first parameter specifies the desired level of event integrity. The second parameter specifies the local folder to use for persisting event data when it is cached to disk.

The two level settings available for the event caching parameter are as follows.
  • Normal - This is the default setting, and it is equivalent to having disc caching disabled. For practical purposes, there is no reason to explicitly set event caching to this level.
  • High - This setting saves event data to a specified disk location. However, event data is held in memory for some time, and if the Analytic terminates abnormally for some reason, that data could be lost. There is some performance degradation, though the exact amount depends on the specific system configuration and the amount of data being processed. The primary performance/event integrity trade-off occurs during Analytic Engine shutdown.

To enable disk caching on a system, you must add the appropriate lines to your wrapper.conf file to point to a valid folder on your system suitable for saving cache information, and you must specify the event guarantee level. The new caching information will take effect when you next start the system. If you implement disk caching, the persistence folder specified must have appropriate security configuration so that the user running the Analytic Engine has permission to write to the folder.

To set system cache properties:

Procedure

  1. Locate and open the Optimize_directory\analysis\conf\wrapper.conf file in a text editor.
  2. Based on your system configuration, add the appropriate line to indicate the event caching persistence folder and the desired level of event guarantee. The following examples are sample event caching lines for a system that uses C:\temp (Windows) and /var/log (Linux) as the persistence folder, and High as the event guarantee level:
    wrapper.java.additional.300= -Dsag.opt.persistence.dir="C:\temp" wrapper.java.additional.301= -Dsag.opt.event.guarantee=High
  3. Save the file and close it. Remember that you must restart your system for any changes to take effect.
    Note: To minimize the potential for data loss, you must exercise caution if you are switching a system between event caching levels. Before switching the event caching setting, shut off the date feed and allow data to be persisted to disk. Then, shutdown the Analytic Engine normally before you make a change to the event caching setting.
  4. You must restart your system for changes to take effect.

Disabling Disk Caching

Edit online

About this task

Use the following procedure to disable disk caching on an Optimize system.

To disable disk caching

  1. Locate and open the wrapper.conf file in a text editor.
  2. Locate the caching configuration line and delete it. See the previous procedure for information about this line in various system configurations.
  3. Save the file and close it.