OMNIbus ObjectServer updates without any probe downtime

Body

It is now possible to change the schema of a live ObjectServer without disrupting the probes that are connected to it. 

This capability is automatically supported in probes running on an installation of IBM Tivoli Netcool/OMNIbus 8.1 Fix Pack 8 or a later version. IBM Tivoli Netcool/OMNibus 8.1 was released as part of the Netcool Operations Insight V1.4.0.2 release.

Schema changes could involve:

1. Adding new columns
2. Deleting columns that have become obsolete.

For installations older than OMNIbus 8.1 Fix Pack 8: -

schema changes, especially option (ii), will adversely affect the probe event stream. It will be necessary to stop and restart all the probes so that they can adopt the new schema. The only schema change that might not affect the probe input adversely, would be to add a new column at the end of the schema.

For OMNIbus 8.1 Fix Pack 8 or later: -

probes will automatically adjust to the new schema without any alerts being lost. Any new columns will contain default values until the probes’ rules files have been updated and the probes have reread them.

What happens within the Netcool/OMNIbus system is the following:

• SQL could be inserted into the ObjectServer that alters the target event table (e.g. alerts.status)
• At the probe layer the following occurs:
  1. Probe receives an error return when attempting to insert an alert.
  2. Probe disconnects.
  3. Incoming event alerts are cached in a Store-and-Forward (SAF) file.
  4. Probe automatically reconnects to the ObjectServer and retrieves information about the current ObjectServer schema.
  5. SAF alerts are reformatted and sent to the ObjectServer.
  6. New alerts are now also correctly formatted for the new schema.
     • Default data type values are used for all new columns, including those that have been defined as ‘NODEFAULT’ in the schema.
     • Data for deleted columns are simply dropped even if the column is still referenced by the rules file.

Probe rules files may require updating to populate any new columns that were added to the schema.

Users are responsible for triggering the probes to reread altered rules. See Reloading multiple rules files as one method whereby this could be done. There is no event loss as any incoming alerts will be cached until the new rules have been processed.

Note that although the probes will now smoothly handle a dropped column, the other OMNIbus components such as the gateways, Web GUI, ObjectServer triggers etc., will all still have issues, if a column they are using, disappears. This makes some downtime of the system inevitable.

The advantage that probes have is that, while maintenance is being done on the ObjectServer and other components, they will simply go into store-and-forward (SAF) mode or failover to the backup ObjectServer. In both failover/failback and in resending the SAF records, the probes will automatically reformat the alerts according to the new ObjectServer schema and thus there should be no event loss. Probes can also handle the scenario where the backup server has a schema that differs from the primary schema. For example, the column order could differ between the primary and the backup.

This enhancement was implemented in response to user requests RFE 34711 and RFE 18860. Other RFEs in this area can be found in the DeveloperWorks RFE Community by searching for the Tivoli Netcool/OMNIbus product  in the Websphere brand.