Overview

The Guardium universal connector enables Guardium® Data Protection to get data potentially from the native activity logs of any data source without using S-TAPs. It includes support for various plug-in packages, requiring minimal configuration. You can easily develop plug-ins for other data sources and install them in Guardium.

The captured events embed messages of any type that is supported by the configured data source. That includes: information and administrative system logs (e.g., login logs, various data lake platform native plug-in related data), DDLs and DMLs, errors of varying subtypes, etc. The incoming events received by the universal connector can be configured to arrive either encrypted or as plain text.
Figure 1. Guardium Universal Connector architecture
Guardium Universal Connector architecture

The Guardium universal connector supports many platforms and connectivity options. It supports pull and push modes, multi-protocols, on-premises, and cloud platforms. For the data sources with pre-defined plug-ins, you configure Guardium to accept audit logs from the data source.

For data sources that do not have pre-defined plug-ins, you can customize the filtering and parsing components of audit trails and log formats. The open architecture enables reuse of prebuilt filters and parsers, and creation of shared library for the Guardium community.

The Guardium universal connector identifies and parses the received events, and converts them to a standard Guardium format. The output of the Guardium universal connector is forwarded to the Guardium sniffer on the collector, for policy and auditing enforcements. The Guardium policy, as usual, determines whether the activities are legitimate or not, when to alert, and the auditing level per activity.

The Guardium universal connector is scalable. It provides load-balancing and fail-over mechanisms among a deployment of universal connector instances, that either conform to Guardium Data Protection as a set of Guardium Collectors, or to Guardium Insights as a set of universal connector pods. The load-balancing mechanism distributes the events sent from the data source among a collection of universal connector instances installed on the Guardium endpoints (i.e., Guardium Data Protection collectors or Guardium Insights pods). For more information, see Enabling Load-Balancing and Fail-Over.

Connections to databases that are configured with the Guardium universal connector are handled the same as all other datasources in Guardium. You can apply policies, view reports, monitor connections, for example.

How Universal Connector works?

The universal connector is a Logstash pipeline comprised of a series of three plug-ins:

  1. Input plug-in. This plug-in ingests events. Depending on the type of plug-in, there are settings to either pull events from APIs or receive a push of events.

  2. Filter plug-in. This plug-in filters the events captured by the input plug-in. The filter plug-in parses, filters, and modifies event logs into a Guardium-digestible format.

  3. Output plug-in. This plug-in receives the formatted event logs from the filter plug-in and transmits them to IBM Guardium (either Guardium Data Protection or Guardium Insights).

Note: The output plug-in is presented here as an internal component of the universal connector pipeline and is not to be accessed or modified by the user.
Figure 2. Logstash pipeline
overview pipiline

Enabling audit log collection

There are a couple of flavors aimed at enabling audit log forwarding into Guardium for various data sources, comprised of either a cloud or on-premise data lake platform, of a database type that is supported by the Guardium sniffer:

  1. Utilize the out-of-the-box, pre-installed plug-in packages that require minimal configuration on the client's end by either plugging suited values into their respective template configuration files in the input and filter sections, or by adding a Ruby code subsection to the said filter section in case a more complex parsing method is necessary as a pre-processing stage to be executed prior to the execution of the respective filter plug-in. See each plug-in's user manual via Available Plug-ins.

  2. For data sources that are not yet supported, you can either upload an IBM-approved filter plug-in or develop your own and add it to our plug-in repository. You can also clone and modify the existing plug-ins as a template for your convenience (either in Ruby or Java). You can optionally either let the parsing operations be executed by your filter plug-in, or assign this task to the Guardium Sniffer by transferring the event to the output plug-in in a designated structure as part of the filter plug-in development, as instructed in the links in the Developers Guide.

Remember:
  1. The pre-defined and pre-installed plug-ins do not require any manual uploads or other such prerequisites on the user's end, as opposed to user-made plug-ins or other available Logstash plug-ins. You can simply use a ready-made template for plugging in values to the input and filter sections of their respective configuration files, expand these sections by using online pre-installed Logstash plug-ins, or write your own Ruby code parser using the Ruby filter plug-in as a pre-processing stage prior to executing the filter plug-ins.
  2. It is recommended to use one of the input plug-ins already in the repository and modify its config file input section. But if the input plug-ins already in the repository are insufficient for your needs, you can add a new one.
  3. You can choose to configure either pull or push methods via the messaging middleware service installed on the data lake platform that is used by the input plug-in. Messages can be received with pull or push delivery. In pull mode, the universal connector instance initiates requests to the remote service to retrieve messages. In push mode, the remote service initiates requests to the universal connector instance to deliver messages.
  4. The specific audit log types transmitted into the universal connector from the data source are configurable via the SQL instance settings installed on the data lake platform. This can vary depending on the installed data lake platform native plug-ins and the utilized messaging middleware service.
  5. For some data lake platforms, you can define inclusion and exclusion filters for the events routed to the universal connector to be ingested by the input plug-in. This can result in a more efficient filtering implemented either as part of the filter scope in the connector's configuration file, or in the developed filter plug-in.

Enabling load balancing and fail-over

When you use the built-in features of both Guardium Data Protection, you might inadvertently distribute the entire set of received events to each Guardium instance, which can result in duplicated and redundant event processing. To prevent this default behavior and ensure efficient operation, it is recommended to configure these mechanisms as part of the input scope in the configuration file of the installed connector. You can achieve this configuration through both pull (Pub/Sub,JDBC, SQS) and push (Filebeat) methods. It's important to note that when using the push method in Guardium Data Protection, configuring the entire set of collectors as part of the input scope is necessary. For more detailed information about each plug-in, please refer to the Supported data sources page.

Configure the Guardium universal connector end-to-end

Quick overview of the steps that are available to configure the Guardium universal connector end-to-end.
Important: You must have S-TAP Management Application role permission.
  1. Allocate Guardium collectors to receive the audit files.
  2. For the data source types supported by Guardium, do the following.
    1. Configure the native audit logs on the data source that Guardium can parse, and then configure the data shipper to forward the audit logs to the Guardium universal connector.
    2. Configure the Guardium universal connector to read the native audit logs. For more information, see Configuring a universal connector by using the legacy workflow topic.
      Note: If you are using secrets or sensitive information in the configuration, see Creating and managing secrets topic before you configure a new connector.
  3. For a data source that does not have off-the-shelf support by Guardium, follow the instructions that are detailed in upload a plug-in topic.
  4. Enable the universal collector feature on the designated Guardium collectors or the stand-alone system. For more information, see Enabling universal connector on collectors topic to enable the Guardium universal connector on collectors.

Limitations

Note: Limitations associated with specific data sources are described in the universal connector plug-in readme files for each datasource. See Supported data sources for more information.
  • When configuring universal connectors, only use port numbers higher than 5000. Use a new port for each future connection.
  • Use only the packages that are supplied by IBM. Do not use extra spaces in the title.
  • IPV6 support
    • S3 SQS and S3 Cloudwatch plug-ins are not supported on IPV6 Guardium systems.
    • The DynamoDB plug-in does not support IPV6.
  • Native MySQL plug-in
    • Do not send the database name to Guardium if the database commands are performed by using MySQL native client.
    • When connected with this plug-in, queries for non-existent tables are not logged to GDM_CONSTRUCT.
  • MongoDB plug-ins do not send the client source program to Guardium.
  • It is recommends not configure more than 10 universal connectors on a single Guardium collector.