Connecting to an External Elasticsearch

Edit online

This section explains the changes that you must make in the config.properties file to enable API Gateway to communicate with the external Elasticsearch.

You can also connect to an external Elasticsearch through externalized configurations. For more details, see Externalizing Configurations.

Note: If you use an external Elasticsearch with same version as API Data Store, then you can use the Kibana or dashboard that is shipped with API Gateway, else they have to be configured separately. If you have configured Elasticsearch externally, then you have to configure Kibana externally. To know the compatible Kibana and Filebeat (Beats) versions for your Elasticsearch, see https://www.elastic.co.
When
Important: When you use an external Elasticsearch, you must use the Elasticsearch plugin mapper size. Without this plugin, API Gateway does not start. To install the mapper size plugin, use the command sudo bin/elasticsearch-plugin install mapper-size. If you have deployed a clustered environment and have configured external Elasticsearch on multiple nodes, you must install this plugin on all the nodes that use external Elasticsearch.

To connect to an external Elasticsearch

  1. Navigate to WmAPIGateway/config/resources/elasticsearch/config.properties

    The config.properties file contains all the properties and Elasticsearch configurations.

  2. Configure the following properties:
    Property and Description
    pg.gateway.elasticsearch.autostart

    This property specifies whether the Elasticsearch starts automatically. If an external Elasticsearch is configured it has to be manually started. This property needs to be set to false to avoid API Data Store starting automatically.

    Default value: true

    pg.gateway.elasticsearch.client.http.response.size

    This property specifies the response size, in MB, for API Gateway Elasticsearch client.

    Default value: 100

    pg.gateway.elasticsearch.config.location

    This property specifies the location of the config file if you want to read port details from some other Elasticsearch config file

    pg.gateway.elasticsearch.hosts

    Mandatory

    This property lists Elasticsearch hosts and ports. The values are comma separated.

    Default value: localhost:9240

    Note: Once a host is added to this property, this is the value that is used to connect to Elasticsearch and the host configured in gateway-es-store.xml is not considered.
    pg.gateway.elasticsearch.http.keepAlive

    Mandatory

    This property creates the persistent connection between client and server.

    Default value: true

    pg.gateway.elasticsearch.http.connectionTimeout

    Mandatory

    This property specifies the time, in milliseconds, after which the connection times out.

    Default value: 10000

    pg.gateway.elasticsearch.http.socketTimeout

    Mandatory

    This property specifies the wait time, in milliseconds, for a reply once the connection to Elasticsearch is established after which it times out.

    Default value: 30000

    pg.gateway.elasticsearch.http.maxRetryTimeout

    Mandatory

    This property specifies the wait time, in milliseconds, for retries after which it times out.

    Default value: 100000

    It is advisable to set max retry time for a request to (number of nodes * socketTimeOut )+connectionTimeout

    pg.gateway.elasticsearch.http.keepAlive.maxConnections

    Mandatory

    This property specifies the maximum number of persistent connections that can be established between an API Gateway and Elasticsearch cluster.

    Default value: 50

    pg.gateway.elasticsearch.http.keepAlive.maxConnectionsPerRoute

    Mandatory

    This property specifies the maximum number of persistent connections that can be established per HTTP route to an Elasticsearch server.

    Default value: 15

    pg.gateway.elasticsearch.http.username

    This property specifies the user name to connect to Elasticsearch using basic authentication.

    pg.gateway.elasticsearch.http.password

    This property specifies the password to connect to Elasticsearch using basic authentication.

    pg.gateway.elasticsearch.https.keystore.filepath

    This property specifies the Keystore file path for establishing HTTPS communication with Elasticsearch.

    pg.gateway.elasticsearch.https.truststore.filepath

    This property specifies the truststore file path for establishing HTTPS communication with Elasticsearch.

    pg.gateway.elasticsearch.https.keystore.password

    This property specifies the Keystore password for establishing HTTPS communication with Elasticsearch.

    pg.gateway.elasticsearch.https.keystore.alias

    This property specifies the Keystore alias for establishing HTTPS communication with Elasticsearch.

    pg.gateway.elasticsearch.https.truststore.password

    This property specifies the truststore password for establishing HTTPS communication with Elasticsearch.

    pg.gateway.elasticsearch.https.enabled

    This property specifies whether you want to enable or disable the HTTPS communication with Elasticsearch.

    Default value: false

    If this property is set to false none of the above properties related to HTTPS are respected.

    pg.gateway.elasticsearch.outbound.proxy.enabled

    This property specifies whether you want to enable or disable outbound proxy communication.

    Default value: true

    pg.gateway.elasticsearch.outbound.proxy.alias

    This property specifies the outbound proxy alias name used to connect to Elasticsearch.

    pg.gateway.elasticsearch.https.enforce.hostname.verification

    This property enforces the host name verification for SSL communication.

    Default value: false

    pg.gateway.elasticsearch.sniff.enable

    Mandatory

    This property enables sniffers to add the other nodes in an Elasticsearch cluster to the client so that the client can talk to all nodes.

    Default value: true

    This configuration must be set to false if you are changing the network when API Gateway or Elasticsearch is running.

    pg.gateway.elasticsearch.tenantId

    This property allows you to specify a tenant name of your choice. This value must be same across all nodes.

    The default value of this property is the Integration Server instance name. So, ensure that you provide same name for all Integration Server nodes in a cluster.

    If you modify this value, you must edit the value in all nodes and restart the API Gateway server for the change to take effect.

    pg.gateway.elasticsearch.sniff.timeInterval

    Mandatory

    This property enables adding the newly added Elasticsearch cluster nodes to existing REST client in a specified time interval in milliseconds.

    Default value: 60000

  3. Restart API Gateway for the HTTP client to take effect.
    Note: If hosts and ports are changed for Elasticsearch then you have to update the appropriate Elasticsearch configuration for Kibana separately and restart the Elasticsearch server as well as Kibana.

You can also externalize the Elasticsearch tenant ID and configuration by using a master configuration file. For details, see Externalizing Configurations.