Configuring high availability

Before you begin

  • Ensure that there are a minimum three nodes available with Developer Portal installation and Ignite in them.
  • Ensure that the required Load balancer is downloaded.
  • Open appropriate ports on each machine so that they can access all components on the other machines. Else, the cluster is not formed.
  • Ensure that the OpenSearch and Developer Portal services in all nodes are stopped.

About this task

This procedure describes in detail the setting up of the High Availability (HA) setup for Developer Portal.

Configuring OpenSearch

About this task

Developer Portal installation includes the deploying and configuring of OpenSearch that is used to store data. For clustering the OpenSearch instances, you must perform OpenSearch clustering. For more information, see https://opensearch.org/docs/latest/tuning-your-cluster/.

You must perform the following steps in all nodes of a cluster.

To configure OpenSearch

Procedure

  1. Open the opensearch.yml file from the location, OpenSearchInstallDir/config/.
    The following configuration is a sample of how the configuration appears initially.
    cluster.name: SAG_EventDataStore
    node.name: portalnode1
    path.logs: ElasticsearchInstallDir/logs
    network.host: 0.0.0.0
    http.port: 9240
    discovery.seed_hosts: ["localhost:9340"]
    transport.port: 9340
    path.repo: ['ElasticsearchInstallDir/archives']
    cluster.initial_master_nodes: "portalnode1"
    xpack.ml.enabled: false
    xpack.security.enabled: false
  2. Provide the name of the cluster in the cluster.name property.
    Nodes with same cluster names form a cluster. That is, if there are three nodes in the cluster, the value in the cluster.name property must be same across all three nodes. In other words, Elasticsearch forms a cluster with nodes that have the same cluster.name.
    For example,
    cluster.name: Dev_portal_cluster
  3. Provide the names of all participating nodes in the discovery.seed_hosts property in the following format.
    host_name:port_name
    For example,
    discovery.seed_hosts: ["node1:9340","node2:9340","node3":"9340"]
    Note: The value of the port must be same as specified in the transport.port property.
  4. Provide the names of the master-eligible nodes in the initial cluster in the cluster.initial_master_nodes property.
    You must provide the node.name value in the cluster.initial_master_nodes property.
    This parameter enables cluster bootstrapping when you start a cluster for the first time. This property must be defined on every master-eligible node in the cluster. This setting helps prevent split-brain, the existence of two masters in a single cluster. For example,
    cluster.initial_master_nodes:["portalnode1"]
  5. Provide the repository in which you want to save the OpenSearch backup snapshots in the path.repo property.
    It is important to have a location that is accessible to all the nodes. The location could be network file system, S3, or Azure in the clustered setup.
    Sample configuration
    cluster.name: Dev_portal_cluster
    node.name: portalnode1
    path.logs: /FS/fslocal/DPO1015/InternalDataStore/logs
    network.host: 0.0.0.0
    http.port: 9240
    discovery.seed_hosts:["portal_server1:9340","portal_server2:9340","portal_server3":"9340"]
    transport.port: 9340
    path.repo: ['/FS/fslocal/DPO1015/InternalDataStore/archives']
    cluster.initial_master_nodes: ["portalnode1","portalnode2","portalnode3"]
    xpack.ml.enabled: false
    xpack.security.enabled: false
    xpack.security.transport.ssl.enabled: false
    xpack.security.http.ssl.enabled: false
    action.destructive_requires_name: false
  6. Save the file.
    The specified OpenSearch nodes are clustered.

Configuring Ignite

About this task

Developer Portal uses the embedded Ignite-core library to setup a cluster, without having to start any external runtime.

In Ignite terminology:

  • a single Developer Portal with embedded ignite is an Ignite server node,
  • together they form an Ignite cluster,
  • and it is a stateless cluster as Developer Portal does not require persistence for its distributed caches.

Ignite cluster discovery

Each Ignite server node has to open a discovery port. Through this port, the nodes discover each other and form the cluster, and they detect the nodes that are added or removed from the cluster.

Each node in the cluster must be configured with a list of initial host names. These nodes will contact each other via the discovery port and form a cluster. Later more nodes can be added to the cluster, and even if their host names are not part of the initial host name list they can join the cluster by contacting one of the initial hosts, and then their host names will be communicated around the cluster.

Pre-requisites

Ensure that you have opened up the following ports:
  • 47500 - 47509. These ports enable the discovery of Ignite nodes.
  • 47100 - 47200. These ports enable communication between Ignite nodes. Ensure that these ports are accessible across other nodes.

    If you have blocked the access of the port 47100 by any another application in the system, you need to open up the complete series. You can just open up the 47100 port. For more info refer https://ignite.apache.org/docs/latest/clustering/network-configuration.

  • It is recommended that you do not open the following local ports when you system is behind firewall:
    • 10800. Enables communication between the thin client and the Ignite cluster.
    • 11211. Enables communication between the JDBC client and the Ignite database.
    • 47400. Enables node discovery via the UDP port.

Perform the following steps in all nodes of a cluster.

To configure Ignite

Procedure

  1. Open the dpo_wrapper.conf file from the location, SAGInstallDir/DeveloperPortal/configuration.
  2. Add the following entries:
    wrapper.java.additional.310=-Dportal.server.cache.distributed.enabled=true
    wrapper.java.additional.311=-Dportal.server.cache.distributed.cluster.peers.0=<devPortal1_hostname>:47500..47509
    wrapper.java.additional.312=-Dportal.server.cache.distributed.cluster.peers.1=<devPortal2_hostname>:47500..47509
    wrapper.java.additional.313=-Dportal.server.cache.distributed.cluster.peers.2=<devPortal3_hostname>:47500..47509
    Note: The values 311, 312, and 313 mentioned in the code block need not be the same. They are sample values only. You can specify numbers of your choice. As a recommended practice, find the biggest number in the dpo_wrapper.conf file and start incrementing the numbers for the next entries as and when you add them to the file. For example, if you see 2008 as the biggest value, then start assigning 2009, 2010 and so on to further entries.
  3. Click Save.
    The configuration is saved.
  4. Start all Developer Portal nodes one after the other.

Configuring Developer Portal

About this task

You must specify the load balancer URL in the Developer Portal instances of all nodes of the cluster.

Procedure

  1. Sign in to Developer Portal.
  2. Expand the menu options and select Administration.
  3. Click General from the left pane.
  4. In the Load balancer URL field, provide the URL of the external Load balancer.
    For example, http://<ext_lb_hostname>/portal.
  5. Click Save.
    The configuration is saved.

Configuring Load Balancer

About this task

Load balancer distributes the incoming requests to the nodes of a cluster with the aim of making their overall processing more efficient. Load balancer optimizes the response time and avoid unevenly overloading some nodes while other nodes are idle.

You can download and use a load balancer of your choice. You must also configure sticky session in the Load balancer for UI upstreaming.

A sample Nginx load balancer configuration file /etc/nginx/nginx.conf is as follows:
#******* @subdomain@Nginx Config ********
events {
    worker_connections  1024;
}
 
http {
 
     
upstream @subdomain@_ui_upstream {
  ip_hash;
  server @node1@:18101;
  server @node2@:18101;
  server @node3@:18101;
}
  
server {
  listen       80;
  listen       [::]:80;
  server_name  @subdomain@;
  
  access_log  /var/log/nginx/access.log;
  
  # DESIGN time should work only in https.
  location /portal {
    proxy_pass              http://@subdomain@_ui_upstream/portal/;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    real_ip_header          X-Real-IP;
    proxy_http_version      1.1;
    proxy_set_header        Connection "";
  }
  
  location /portal/ {
    proxy_pass              http://@subdomain@_ui_upstream/portal/;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    real_ip_header          X-Real-IP;
    proxy_http_version      1.1;
    proxy_set_header        Connection "";
  }
}
 
}

The nodes in the cluster are synchronized.

Post-configuration

About this task

Perform the following steps after you have completed the high availability configuration:

Procedure

  1. Start the OpenSearch nodes one after the other.
  2. Ensure that the started node is up and running before starting the next node.
  3. Open your browser and enter the following URL in the address bar, and press Enter to verify the cluster configuration
    http://<node_name>:<port_number>/_cluster/health?pretty=true
    For example,
    http://ES-node1:<9393>/_cluster/health?pretty=true
    Sample response
    {
      "cluster_name" : "Dev_portal_cluster",
      "status" : "green",
      "timed_out" : false,
      "number_of_nodes" : 3,
      "number_of_data_nodes" : 3,
      "active_primary_shards" : 18,
      "active_shards" : 36,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 0,
      "delayed_unassigned_shards" : 0,
      "number_of_pending_tasks" : 0,
      "number_of_in_flight_fetch" : 0,
      "task_max_waiting_in_queue_millis" : 0,
      "active_shards_percent_as_number" : 100.0
    }