Integrating the gateway with Microsoft Azure Event Hubs

The Gateway for Message Bus forwards events to Microsoft Azure Event Hubs using REST HTTP messages.

In this integration, the Gateway for Message Bus functions as the producer that publishes ObjectServer events to Microsoft Azure Event Hubs. Consumer applications may connect to Microsoft Azure Event Hubs using Kafka or AMQP to consume events forwarded by the gateway.

This integration uses the new Multi-Channel HTTP Transport and has the following features:

Support for custom headers in the REST messages for example the Authorization header which is used to specify the API key or token required for authentication.
Secret credentials ie. API key or token can be provided in a header file which can be encrypted.
To address token expiry, the header file will be read and refreshed periodically to read updated values. This enables the gateway to access updated tokens.
A maximum size can be defined for the message body to comply with the service endpoint restrictions.
Support for sending batched message events.

Configuring the Gateway for Message Bus

To configure the Gateway for Message Bus to integrate with Microsoft Azure Event Hub, use the following steps:

Edit the configuration in the gateway properties file $OMNIHOME/gates/xml/G_XML.props:
```
Gate.Java.Arguments         : '-Dcom.ibm.jsse2.overrideDefaultTLS=true'

Gate.XMLGateway.TransportFile       : '$OMNIHOME/java/conf/multiChannelHttpTransport.properties'
Gate.XMLGateway.TransportType       : 'MultiChannelHttp'
```
The Java argument com.ibm.jsse2.overrideDefaultTLS=true instructs the IBM Java to enable higher versions of TLS protocols. Refer to the following documentation for more details: https://www.ibm.com/docs/en/sdk-java-technology/8?topic=customization-matching-behavior-sslcontextgetinstancetls-oracle

Update the configuration in the $OMNIHOME/java/conf/multiChannelHttpTransport.properties file:

# Path to the Transport properties file
httpConnectionPropertiesFile=/opt/IBM/tivoli/netcool/omnibus/java/conf/azureMultiChannelHttpTransport.json

# Specifies the encryption method to be used
#ConfigCryptoAlg=AES_FIPS

# Path to the encryption key file
#ConfigKeyFile=/home/netcool/aesfips.key

# Defines the maximum size of the message body in bytes. This is intended to limit the size of batched message events to comply with the requirements of the service endpoint.
MaxMsgSize=0

If MaxMsgSize is set to 0, no restriction on the message body size is applied.

Apply the correct configuration in azureMultiChannelHttpTransport.json:
```
   "SEND":
   {
      "NEW_EVENT":
      {
          "uri":"https://mynamespace.servicebus.windows.net/myeventhub/messages",
          "method":"POST",
          "headers":"Authorization=++Authorization++,Content-Type=application/vnd.microsoft.servicebus.json”,
          "content":"{\"Body\":\"++message++\"}",
         "headerFile":"/opt/IBM/tivoli/netcool/omnibus/java/conf/azureHeader.props",
         "headerFileRefresh":"60",
          "requireSSL":"true",
         "bufferSize":"1000",
         "bufferFlushTime":"45",
         "batchHeader":"[",
         "batchSeparator":",",
         "batchFooter":"]"
      }
}
```
This example shows the configuration to send JSON batch events to Microsoft Azure Event Hubs.

uri specifies the target service endpoint for which the gateway is to post events. For Azure Event Hubs, the URI is in the following format:
```
https://{servicebusNamespace}.servicebus.windows.net/{eventHubPath}/messages 
```
headers specify the headers to be included in every request. Specific headers that have secret credentials or values that need to be updated after a period of time should be defined with a placeholder option:
```
Authorization=++Authorization++
```
The actual value is to be defined in the header file, which will be read and substituted into the placeholder. Other headers with static values or non-confidential information may be defined directly in the headers property.

headerFile specifies the full path to the header file.

content allows for modification of a JSON message into a custom format. For Microsoft Azure Event Hubs, this serves the modify the format of every JSON message to comply with the format for batch JSON message events. The event is modified to be a nested JSON as shown below (simplified for readability):
```
[
  {
    "Body": "{\"Identifier\" : \"TEST ident 6_CLOSED\"}"
  },
  {
    "Body": "{\"Identifier\" : \"TEST ident 7_CLOSED\"}"
  },
  {
    "Body": "{\"Identifier\" : \"TEST ident 8_CLOSED\"}"
  }
]
```
If content is not specified, no modification is performed on the JSON events.

headerFileRefresh specifies the interval in seconds for which the gateway will read and refresh the header file. To address token expiry, the header file contents should be updated before the time the token expires. Configure headerFileRefresh to occur before the time the token expires. If headerFileRefresh is set to 0, the header file will only be read once at startup.

requireSSL should be set to true if connecting to a HTTPS endpoint. For HTTP endpoint, set to false.

bufferSize specifies the size of the buffer that the gateway uses to store events before sending them to Azure Event Hubs. During event replication, events are gradually stored in the buffer until it reaches the size defined by bufferSize. When this size is reached the events will extracted from the buffer and sent to Azure Event Hubs. If bufferSize is set to 1, events will be sent out immediately to Azure Event Hubs.

bufferFlushTime specifies the interval in seconds that the gateway waits before flushing the events in the buffer. When this interval is reached, the gateway will extract all events in the buffer for transmission to Azure Event Hubs. If bufferFlushTime is set to 0, this functionality is disabled.

batchHeader, batchSeparator and batchFooter are properties to specify the format of batched message events. Specify the following values for JSON batch events:
```
         "batchHeader":"[",
         "batchSeparator":",",
         "batchFooter":"]"
```
For XML batch events, specify the following values:
```
         "batchHeader":"<?xml version=\"1.0\" encoding=\"UTF-8\"?><tns:netcoolEventList xmlns:tns=\"http://item.tivoli.ibm.com/omnibus/netcool\">",
         "batchSeparator":"",
         "batchFooter":"</tns:netcoolEventList>"
```
Edit azureHeader.props to set the correct values:
```
Authorization=<token>
```
For Microsoft Azure Event Hubs,

To authenticate access to Microsoft Azure Event Hubs, the Shared Access Signature (SAS) token should be provided in the Authorization header. Refer to Microsoft Azure Event Hubs REST API documentation for more information on generating the token.

The header file must be updated before the token expiry time. You can update the header file without restarting the gateway. Set the headerFileRefresh value so that the gateway will refresh the header file before the token expiry time is reached.
Start the gateway.
To optimize the replication of events to Azure Event Hubs, set higher values of bufferSize if the gateway is expected to process a larger number of events from the ObjectServer. This will result in more events being sent to Azure Event Hubs per each connection. It is preferred to minimize the number of requests sent to the Azure Event Hubs. If too many requests are sent to Azure Event Hubs within a short period of time, Azure Event Hubs might consider that the gateway is attempting to flood the service and reject the events. Also, consider specifying longer interval periods for bufferFlushTime to coordinate processing the events in the buffer.

Configuring for encryption

Optionally you can enhance the security of the header file using encryption. The following methods can be applied:

Encrypting the entire header file
Encrypting only the authorization header value

Setup

Configure the IBM JRE to work with AES FIPS:
See https://www.ibm.com/docs/en/netcoolomnibus/8.1?topic=tasks-configuring-jre-fips-1402
Generate a key file:
See https://www.ibm.com/docs/en/netcoolomnibus/8.1?topic=encryption-generating-key-in-key-file

Configure the MultiChannelHttpTransport.props file

# Path to the Transport properties file
httpConnectionPropertiesFile=/opt/IBM/tivoli/netcool/omnibus/java/conf/azureMultiChannelHttpTransport.json

# Specifies the encryption method to be used
ConfigCryptoAlg=AES_FIPS

# Path to the encryption key file
ConfigKeyFile=/home/netcool/aesfips.key

Currently only AES_FIPS is supported.

Apply one of the following encryption methods:
Encrypt the entire header file:
1. Populate the unencrypted header file /home/netcool/azureHeader.props with the token or API key:
```
Authorization=<SAS token>
```
2. Encrypt the azureHeader.props header file using the AES_FIPS algorithm and key file:
```
$OMNIHOME/bin/nco_aes_crypt -k /home/netcool/aesfips.key -c AES_FIPS -f /home/netcool/azureHeader.props -o $OMNIHOME/java/conf/azureHeader.props
```
  This creates the encrypted header file: $OMNIHOME/java/conf/azureHeader.props. The full path to this file is to be specified in the headerFile in the azureMultiChannelHttpTransport.json.
Encrypt only the Authorization header value:
1. Encrypt the token using the AES_FIPS algorithm and key file:
```
$OMNIHOME/bin/nco_aes_crypt -k /home/netcool/aesfips.key -c AES_FIPS <token>
```
  This creates the encrypted token string.
2. Copy the encrypted string into the header file $OMNIHOME/java/conf/azureHeader.props. The full path to this file is to be specified in the headerFile in the azureMultiChannelHttpTransport.json.
```
Authorization=<encrypted token string>
Anotherheader=someheadervalue
```

Configuring Microsoft Azure Event Hub

Create a Resource Group.
Create the Event Hubs namespace.
Create the Event Hub.
Create the Storage Account and put it under the Resource Group created in Step 1.

See the Microsoft Azure Event Hubs documentation for more details.