Azure event hubs input plug-in

This is a Logstash input plug-in for the universal connector that pulls events from the Azure Event Hub. The events are then sent over to the corresponding filter plug-in, which, transforms the audit logs into a Guardium instance. The information is then sent over to Guardium. Guardium records include the accessor (the person who tried to access the data), the session, data, and exceptions. If there are no errors, the data contains details about the query "construct". The construct details the main action (verb) and collections (objects) involved.

Specifications

  • Tested versions: 1.4.3
  • Input plug-in developer: Elastic
  • To configure Azure event Hubs, see Azure PostgresSQL.
  • Supported Guardium versions:
    • Guardium Data Protection: 11.4 or later

Usage

This plug-in uses events from Azure Event Hubs, a highly scalable data streaming platform and event ingestion service. Event producers send events to the Azure Event Hub, and this plug-in use those events for use with Logstash.

Parameters

Table 1. Parameters
Parameters Input types Required Default Description
config_mode String (basic or advanced)   Basic The config_mode setting allows specifying configuration to either Basic configuration (default) or Advanced configuration
event_hub_connections Array Yes, when config_mode is basic  

The event_hub_connections setting allows specifying the list of connection strings that identifies the Event Hubs to be read. Connection strings include the EntityPath for the Event Hub.

The event_hub_connections option is defined per Event Hub. All other configuration options are shared among Event Hubs.
initial_position String, (beginning, end, or look_back) No Beginning

The initial_position setting allows specifying when first reading from an Event Hub, start from this position:

Valid options for start_position are:

  • beginning - reads all pre-existing events in the Event Hub (default)
  • end - does not read any pre-existing events in the Event Hub
  • look_back reads the end minus a number of seconds worth of pre-existing events. You control the number of seconds by using the initial_position_look_back option.
threads Number No 16 The threads setting allows setting the total number of threads that are used to process events. The value that you set here applies to all Event Hubs. Even with advanced configuration, this value is a global setting, and can’t be set per event hub.
decorate_events Boolean No $Default The decorate_events setting allows adding metadata about the Event Hub, including Event Hub name, consumer_group, processor_host, partition, offset, sequence, timestamp, and event_size.
consumer_group String No  

The consumer_group setting allows specifying the Consumer group that is used to read the Event Hub(s). Create a consumer group specifically for Logstash. Then ensure that all instances of Logstash use that consumer group so that they can work together properly.

Logstash Default config parameters

Other standard Logstash parameters are as follows:

  • add_field
  • type
  • tags
Example: 
input {
	azure_event_hubs 
		{
			config_mode => "basic"
			event_hub_connections => [<connection_string>]
			initial_position => "end"
			threads => 8
			decorate_events => true
			consumer_group => "$Default" 
			type => "azure_event_hub"
			add_field => {"enrollmentId" => <enrollmentId>}	
		}
}