Interactive installation guide

Requirements for a clustered environment

To set up a clustered environment for your applications, you need the following servers:

• One server to host and ( can be installed on its own dedicated server).
• Two or more nodes to host the application instances.
• Two or more nodes to host the QM application instances.
• One server for the load balancer (HAProxy). You can install a separate load balancer for each application, or use the same load balancer to route traffic to multiple applications.
• One server for the MQTT broker ().
• One server for the database.

Note: The is a Java application that by default, is installed on . To gain performance, consider adding another server in the cluster, dedicated to the .

To set up a system with HAProxy high availability, you need the following servers:

• One server to host .
• Two servers to host (one in standby mode).
• Two or more nodes to host the application instances.
• Two or more nodes to host the QM application instances.
• Two servers for the load balancer (HAProxy).
• Two servers for the MQTT broker ().
• One server for the database.

All servers in the environment must have their fully qualified host names configured. When you update the /etc/hosts file, use the IP fully-qualified-host-name alias [,alias] format. Note that the fully qualified host name must appear before any alias. If it does not, clustering might not be enabled for multiple nodes.

For a list of system requirements, see System requirements.

Set up the HAProxy load balancer

Note: Installation of HAProxy load balancer is not supported on Windows platforms.

You must install and configure HAProxy before the cluster can be activated. For more information about HAProxy, see www.haproxy.org.

Note: You can also configure and use IHS instead of HAProxy, but only one of these two methods must be used in a cluster for load balancing.

I. Installing a non-Dockerized version of HAProxy

HAProxy is part of the official Red Hat Linux repository and can be installed by using the yum command. You must have root user permissions or use the sudo command to do the installation.

1. To install HAProxy, open a command shell and enter the following command:

   yum -y install haproxy

2. By default HAProxy is installed in the /usr/local/haproxy/etc/haproxy directory. Go to the HAProxy installed directory and back up the haproxy.cfg file. For example, create a haproxy.cfg.backup file.
3. Open the haproxy.cfg file for editing and configure it as needed. Use the following example as a reference. For more details about configuration language and available options, see the HAProxy Configuration Manual.
   
   

   Note: If your is fronted by the HAProxy, see this Depolyment wiki document for the haproxy.cfg sample file.

   haproxy.cfg sample file for clustering

   #---------------------------------------------------------------------
   # Example configuration for a possible web application. See the
   # full configuration options online.
   #
   # http://haproxy.1wt.eu/download/1.6/doc/configuration.txt
   #
   #---------------------------------------------------------------------
   
   #---------------------------------------------------------------------
   # Global settings
   #---------------------------------------------------------------------
   global
   # If not using the dockerized version,
   # to have these messages end up in /var/log/haproxy.log you will
   # need to:
   #
   # 1) configure syslog to accept network log events. This is done
   # by adding the '-r' option to the SYSLOGD_OPTIONS in
   # /etc/sysconfig/rsyslog
   #
   # 2) configure local2 events to go to the /var/log/haproxy.log
   # file. A line like the following can be added to
   # /etc/sysconfig/rsyslog
   #
   # local2.* /var/log/haproxy.log
   #
   # 3) uncomment the line below and comment the other log line
   #log 127.0.0.1 local2
   log /dev/log local2
   
   chroot /var/lib/haproxy
   pidfile /var/run/haproxy.pid
   maxconn 6000
   user haproxy
   group haproxy
   daemon
   
   # turn on stats unix socket
   stats socket /var/lib/haproxy/stats
   
   tune.ssl.default-dh-param 2048
   
   #---------------------------------------------------------------------
   # common defaults that all the 'listen' and 'backend' sections will
   # use if not designated in their block
   #---------------------------------------------------------------------
   defaults
   mode http
   option dontlognull
   option http-keep-alive
   option redispatch
   retries 3
   timeout http-request 10s
   timeout queue 1m
   timeout connect 10s
   timeout client 2h
   timeout server 2h
   timeout http-keep-alive 10s
   timeout check 10s
   maxconn 6000
   
   #---------------------------------------------------------------------
   # main frontend which proxys to the backends
   #---------------------------------------------------------------------
   frontend lanneluc-proxy
   bind *:80
   bind *:9443 ssl crt /etc/haproxy/ssl/proxy.pem no-sslv3
   log global
   option httplog
   mode http
   capture cookie SERVERID len 32
   redirect scheme https if !{ ssl_fc }
   maxconn 2000 # The expected number of the users of the system.
   # Limits the sockets to this number of concurrent connections. Extraneous
   # connections will remain in the system's backlog until a connection is
   # released. If unspecified, the limit will be the same as the frontend's
   # maxconn.
   default_backend ccm
   
   
   backend ccm
   
   option forwardfor
   http-request set-header X-Forwarded-Port %[dst_port]
   http-request add-header X-Forwarded-Proto https if { ssl_fc }
   fullconn 1000 # if not specified, HAProxy will set this
   # to 10% of 'maxconn' specified on the frontend
   
   balance leastconn
   cookie SERVERID insert indirect nocache secure attr "SameSite=None"
   
   # minconn = 100, the server will always accept at least 100,
   # but no more than 'maxconn' connections. Should be the same as Liberty
   # settings in server.xml
   # maxconn = 500, the max number of connections that will be sent to a server.
   # The overage will be queued. The total for all nodes should be equal to maxxcon
   # on 'frontend'
   server ccm1 node1.host.com:9443 minconn 100 maxconn 500 ssl check cookie ccm1 verify none
   server ccm2 node2.host.com:9443 minconn 100 maxconn 500 ssl check cookie ccm2 verify none
   server ccm3 node3.host.com:9443 minconn 100 maxconn 500 ssl check cookie ccm3 verify none
   server ccm4 node5.host.com:9443 minconn 100 maxconn 500 ssl check cookie ccm4 verify none
   
   # Uncomment the following section if your MessageBroker is fronted by HAProxy (for example if in High Availability mode)
   #
   # # connect MessageBroker cluster in tcp mode (N servers if HA mode, or a single server if IoT MessageBroker just must be fronted by proxy)
   # listen MessageBroker
   # bind *:1883
   # mode tcp
   # balance leastconn
   # option tcplog
   # server MessageBroker1 [MessageBroker server 1 URI]:1883 check
   # server MessageBroker2 [MessageBroker server 2 URI]:1883 check
   # # connect MessageBroker cluster in HTTP mode to allow gathering of MQTT statistics (6.0.6 and newer only) (N servers if HA mode, or a single server if IoT MessageBroker just must be fronted by proxy)
   # listen MessageBroker_REST
   # bind *:9089 ssl crt /etc/haproxy/ssl/msigh.pem no-sslv3 # Port (9089) must match that configured in MessageBroker, and the same port needs to be specified in MQTTStatsService advanced property
   # mode http
   # option tcp-check # Must tell HAProxy to use TCP port to do health checks
   # balance leastconn
   # # Same number of entries and same servers as in the section above:
   # server MessageBroker_REST1 [MessageBroker server 1 URI]:9089 check port 1883 # Same ports (1883 and 9089) as used elsewhere in this file.
   # server MessageBroker_REST2 [MessageBroker server 2 URI]:9089 check port 1883 # Same ports (1883 and 9089) as used elsewhere in this file.
   
   listen statistics
   bind *:1936
   stats uri /
   stats admin if TRUE
   stats enable
   stats hide-version
   stats auth root:clu8ter8
   stats refresh 5s
   

   haproxy.cfg sample file for QM clustering

   #---------------------------------------------------------------------
   # Example configuration for a possible web application. See the
   # full configuration options online.
   #
   # http://haproxy.1wt.eu/download/1.6/doc/configuration.txt
   #
   #---------------------------------------------------------------------
   
   #---------------------------------------------------------------------
   # Global settings
   #---------------------------------------------------------------------
   global
   # If not using the dockerized version,
   # to have these messages end up in /var/log/haproxy.log you will
   # need to:
   #
   # 1) configure syslog to accept network log events. This is done
   # by adding the '-r' option to the SYSLOGD_OPTIONS in
   # /etc/sysconfig/rsyslog
   #
   # 2) configure local2 events to go to the /var/log/haproxy.log
   # file. A line like the following can be added to
   # /etc/sysconfig/rsyslog
   #
   # local2.* /var/log/haproxy.log
   #
   # 3) uncomment the line below and comment the other log line
   #log 127.0.0.1 local2
   log /dev/log local2
   
   chroot /var/lib/haproxy
   pidfile /var/run/haproxy.pid
   maxconn 6000
   user haproxy
   group haproxy
   daemon
   
   # turn on stats unix socket
   stats socket /var/lib/haproxy/stats
   
   tune.ssl.default-dh-param 2048
   
   #---------------------------------------------------------------------
   # common defaults that all the 'listen' and 'backend' sections will
   # use if not designated in their block
   #---------------------------------------------------------------------
   defaults
   mode http
   option dontlognull
   option http-keep-alive
   option redispatch
   retries 3
   timeout http-request 10s
   timeout queue 1m
   timeout connect 10s
   timeout client 2h
   timeout server 2h
   timeout http-keep-alive 10s
   timeout check 10s
   maxconn 6000
   
   #---------------------------------------------------------------------
   # main frontend which proxys to the backends
   #---------------------------------------------------------------------
   frontend lanneluc-proxy
   bind *:80
   bind *:9443 ssl crt /etc/haproxy/ssl/proxy.pem no-sslv3
   log global
   option httplog
   mode http
   capture cookie SERVERID len 32
   redirect scheme https if !{ ssl_fc }
   maxconn 2000 # The expected number of the users of the system.
   # Limits the sockets to this number of concurrent connections. Extraneous
   # connections will remain in the system's backlog until a connection is
   # released. If unspecified, the limit will be the same as the frontend's
   # maxconn.
   default_backend qm
   
   
   backend qm
   
   option forwardfor
   http-request set-header X-Forwarded-Port %[dst_port]
   http-request add-header X-Forwarded-Proto https if { ssl_fc }
   fullconn 1000 # if not specified, HAProxy will set this
   # to 10% of 'maxconn' specified on the frontend
   
   balance leastconn
   cookie SERVERID insert indirect nocache secure attr "SameSite=None"
   
   # minconn = 100, the server will always accept at least 100,
   # but no more than 'maxconn' connections. Should be the same as QM Liberty
   # settings in server.xml
   # maxconn = 500, the max number of connections that will be sent to a server.
   # The overage will be queued. The total for all nodes should be equal to maxxcon
   # on 'frontend'
   server qm1 node1.host.com:9443 minconn 100 maxconn 500 ssl check cookie ccm1 verify none
   server qm2 node2.host.com:9443 minconn 100 maxconn 500 ssl check cookie qm2 verify none
   server qm3 node3.host.com:9443 minconn 100 maxconn 500 ssl check cookie qm3 verify none
   server qm4 node5.host.com:9443 minconn 100 maxconn 500 ssl check cookie qm4 verify none
   
   # Uncomment the following section if your MessageBroker is fronted by HAProxy (for example if in High Availability mode)
   #
   # # connect MessageBroker cluster in tcp mode (N servers if HA mode, or a single server if IoT MessageBroker just must be fronted by proxy)
   # listen MessageBroker
   # bind *:1883
   # mode tcp
   # balance leastconn
   # option tcplog
   # server MessageBroker1 [MessageBroker server 1 URI]:1883 check
   # server MessageBroker2 [MessageBroker server 2 URI]:1883 check
   # # connect MessageBroker cluster in HTTP mode to allow gathering of MQTT statistics (6.0.6 and newer only) (N servers if HA mode, or a single server if IoT MessageBroker just must be fronted by proxy)
   # listen MessageBroker_REST
   # bind *:9089 ssl crt /etc/haproxy/ssl/msigh.pem no-sslv3 # Port (9089) must match that configured in MessageBroker, and the same port needs to be specified in MQTTStatsService advanced property
   # mode http
   # option tcp-check # Must tell HAProxy to use TCP port to do health checks
   # balance leastconn
   # # Same number of entries and same servers as in the section above:
   # server MessageBroker_REST1 [MessageBroker server 1 URI]:9089 check port 1883 # Same ports (1883 and 9089) as used elsewhere in this file.
   # server MessageBroker_REST2 [MessageBroker server 2 URI]:9089 check port 1883 # Same ports (1883 and 9089) as used elsewhere in this file.
   
   listen statistics
   bind *:1936
   stats uri /
   stats admin if TRUE
   stats enable
   stats hide-version
   stats auth root:clu8ter8
   stats refresh 5s
   

   You must ensure that several important entries in the configuration file are entered correctly:

   • /etc/haproxy/ssl/proxy.pem: This references the certificate that must be available to HAProxy so the proxy server can examine SSL-encrypted traffic. This ability is required for session affinity, which is an important component of clustering in .
   • balance leastconn: This is the balancing strategy. The cookie SERVERID insert indirect nocache secure attr "SameSite=None" setting ensures that the user login information is passed between different sessions. Without this setting, the user will be required to log in again after navigating to any URL that requires user validation.
   • Back-end servers: Replace ccmxqmx with your application names and nodex.host.com:9443 with your server URLs.
   • minconn, maxconn, fullconn: The minimum connection, maximum connection, and full connection parameters.
   • Administration interface:

     listen statistics
     bind *:1936
     stats uri /
     stats admin if TRUE
     stats enable
     stats hide-version
     stats auth root:clu8ter8
     stats refresh 5s
     

     These entries are explained in details in the next sections.

4. Open a command shell and enter the following command to enable the service:

   systemctl enable haproxy

5. Enter the following command to start the service:

   service haproxy start

II. Installing a certificate

HAProxy requires a certificate to parse encrypted traffic. You can use a self-signed certificate. The following commands create a self-signed certificate for your testing environment, but you must obtain a real certificate if required.

OpenSSL prompts you to enter the details for the certificate, such as the host name and so on. Double quotation marks are optional. The host name (CN) is the host name of the server where HAProxy is installed:

cd /usr/local/haproxy/etc
openssl genrsa -des3 -out ssl/proxy.key 2048
openssl req -new -key "ssl/proxy.key" -out "ssl/proxy.csr"
openssl x509 -req -days 365 -in "ssl/proxy.csr" -signkey "ssl/proxy.key" -out "ssl/proxy.crt"
cat ssl/proxy.crt ssl/proxy.csr > ssl/proxy.pem

III. Using a load-balancing rule

HAProxy offers several load-balancing algorithms. In the configuration example, the balance leastconn option is used. The balance leastconn option instructs HAProxy to direct traffic to the back-end server with the least number of connections.

IV. Configuring back-end servers

The back-end servers are the lines that start with "server" in the configuration file. These entries instruct the HAProxy to redirect traffic to the nodes. In the configuration example, there are four nodes. Replace the nodex.host.com parts with the actual host-name of the servers (nodes) on which you install your or servers (replicas).

You can preconfigure HAProxy with as many back-end servers (nodes) as the number of or nodes that you are planning to have. For the initial setup stage, start with one instance of or on one node by commenting out all the other server lines except for the first line:

example:

server ccm1 node1.host.com:9443 minconn 100 maxconn 500 ssl check cookie ccm1 verify none
#server ccm2 node2.host.com:9443 minconn 100 maxconn 500 ssl check cookie ccm2 verify none
#server ccm3 node3.host.com:9443 minconn 100 maxconn 500 ssl check cookie ccm3 verify none
#server ccm4 node5.host.com:9443 minconn 100 maxconn 500 ssl check cookie ccm4 verify none

QM example:

server qm1 node1.host.com:9443 minconn 100 maxconn 500 ssl check cookie qm1 verify none
#server qm2 node2.host.com:9443 minconn 100 maxconn 500 ssl check cookie qm2 verify none
#server qm3 node3.host.com:9443 minconn 100 maxconn 500 ssl check cookie qm3 verify none
#server qm4 node5.host.com:9443 minconn 100 maxconn 500 ssl check cookie qm4 verify none

After HAProxy is configured, if it is running as a service, you can start, stop, and restart it by entering the following command:

sudo service haproxy start|stop|restart|reload

V. Optimizing the number of connections

The minconn, maxconn, and fullconn parameters are used to fine-tune the server and to control the number of simultaneous connections the server can handle. If more connections are established, the overage will be buffered. It can be difficult to determine the optimal numbers for these parameters, because they depend on the expected number of users and the server responsiveness, which can be influenced by available resources, network bandwidth, and so on. The number of connections can be fine-tuned depending on the number of users that are expected to use the clustered applications.

These numbers are also related to how is configured. By default, the instance that runs is configured with coreThreads="100" which is the lowest number of threads that handle incoming requests. The maximum is limited by the available system resources. Therefore, the minconn parameter should be set to the value of the coreThreads setting. The maxconn value in the back-end section (that is, one per back-end server) should be set to the maximum number of users that the server is capable of handling without showing any degradation in performance or response times. In the configuration example provided earlier, the maxconn value is set to 500 on each back end. The total of all back-end maxconn values should equal the maxconn value that is set in the front-end section of the configuration file. For example, if there are four nodes that are each set to maxconn=500, the maxconn value in the front-end should be set to 2000.

VI. Accessing the monitoring console and logs

The last section in the configuration file specifies how to access the HAProxy monitoring console. The console shows the number of nodes and their statuses, as well as the incoming and outgoing traffic. For a well-functioning cluster, assuming all nodes have been brought online at about the same time, the number of sessions (Total and lbTotal) should be roughly the same, and the number of bytes transferred should not vary too much between nodes. There should be no errors or warnings.

HAProxy maintains a log file in this directory: /var/log/haproxy.log. This log file contains information about which requests were redirected to which back-end servers and other HAProxy information such as session-affinity cookies and response codes.

Optional: Set up HAProxy high availability

Optional: Set up a custom loadbalancing

HAProxy provides different parameters for adapting and tuning its behavior, including a property named weight that you can use to identify servers for balancing. This property is only used for the first session request, because all the other requests for this session are routed to the same server with session affinity. The weight must be a value between 1 and 256. When the weight property is not set, the default value of 1 is used.

The weight property can be added to any server declaration in a back end, as shown in the following line:

backend ccm
...
server ccm1 node1.host.com:9443 minconn 100 maxconn 500 ssl check weight 50 cookie ccm1 verify none

backend qm
...
server qm1 node1.host.com:9443 minconn 100 maxconn 500 ssl check weight 50 cookie qm1 verify none

The weight parameter enables you to define a specific weight for your server. However, you must edit the haproxy.cfg file each time that you want to change the weight. To avoid opening the haproxy.cfg file to change the weight, you can use a socket interface to change the weight while the HAProxy server is running. Use the following procedure to install the Socat program, which allows you to send messages to the HAProxy TCP stats socket:

Procedure

1. To install Socat, enter the following command in the shell:

   yum -y install socat

2. Open and edit the haproxy.cfg file to declare the socket and set its permission and level. Add the following lines to the general section of haproxy.cfg. In the following example, the socket in the /var/lib/haproxy/stats.sock directory is used. Only the user that is declared as the HAProxy user in the haproxy.cfg file is permitted to connect to the socket mode 600. The access level is set to administrator to allow updates to the weight parameter by using the socket:

   stats socket /var/lib/haproxy/stats.sock mode 600 level admin
   stats timeout 2m

3. Restart the HAProxy server.
4. Determine the correct weight for the nodes based on the server activities. You can gather system data from all nodes and compute a weight based on the gathered data. After you compute the correct weight based on system data, you can create a script and use the weight for all nodes. In the following example, ccm is the name of the back end, ccm1 is the name of the server node, and the new weight is set to 50: :

   echo "set weight ccm/<serverName> <newWeight>" | socat unix-connect:/var/lib/haproxy/stats.sock stdio
   ex: echo "set weight ccm/ccm1 50" | socat unix- connect:/var/lib/haproxy/stats.sock stdio

5. Determine the correct weight for the nodes based on the server activities. You can gather system data from all nodes and compute a weight based on the gathered data. After you compute the correct weight based on system data, you can create a script and use the weight for all nodes. In the following example, qm is the name of the back end, qm1 is the name of the server node, and the new weight is set to 50:

   echo "set weight qm/<serverName> <newWeight>" | socat unix-connect:/var/lib/haproxy/stats.sock stdio
   ex: echo "set weight qm/qm1 50" | socat unix- connect:/var/lib/haproxy/stats.sock stdio

Set up

Note: Installation of is not supported on Windows platforms.

provides secure, rapid, bidirectional messaging for the Internet of Things (IoT) and mobile environments. is a high performance MQTT broker.

I. Installing the non-Dockerized version of

Procedure

To install a non-Dockerized version of , follow these steps:

1. Download the required packages for the latest 5.0 from ibm.com.

   Note: An IBMid is required to access these files. If you do not have an IBMid, you will be prompted to create one before you can download .

2. Follow the instructions in the documentation to install the non-Dockerized (RPM) version.

II. Logging in to the Web UI

For complete instructions, see the documentation.

III. Configuring

uses organizational configuration objects called hubs to define configuration policies.

Procedure

1. After logging in to the Web UI by following the procedure in the previous task, create a new hub and then create and assign a connection policy, messaging policy and endpoint.
2. Create a new connection policy. Provide a name and a description. Enter a client ID and select MQTT for protocol. Under Specify the resources that an MQTT client is permitted to consume, select Allow Clients with Durable Subscriptions and Allow Persistent Messages check boxes.
3. Go to the Message Policy tab and create a new topic policy. Provide a name and a description for the new topic policy. Enter a client ID and select MQTT for the protocol. In the Topic field enter an asterisk (*) and, under Authority, select both the Publish and Subscribe check boxes. Under Subscriber Settings, ensure that you enter 10000 for the Max Messages value and select Discard Old Messages for the Max Messages Behavior value.
4. Finally, create a new endpoint. Provide a name and a description. In the Port field, specify the port that your clients ( or nodes) use to connect to publish and subscribe to messages. Later, you will add this port to the server.startup file. Select MQTT as the protocol. Set the Max Message Size value to 4096. Under Connection Policies and Messaging Policies, add the connection policy and messaging policy (topic policy) that you created in the previous steps.

IV. Monitoring subscriptions

You can view subscription monitoring statistics by using the Web UI. To access the monitoring subscriptions, go to the Subscriptions page by selecting Monitoring > Subscriptions.

In the Monitoring view, you can see all MQTT clients (nodes), the topics being subscribed to and published, the active consumers, the number of messages, and the buffer sizes. You can use various options to filter the data.

V. Optional: high availability

To implement high availability for (primary and standby nodes), see the following topics in the documentation:

• High Availability overview
• Configuring your system for high availability
• Configuring MessageSight by using the Web UI
• Configuring MessageSight by using REST Administration APIs

() for clustered applications

For () applications that support clustering such as or , the distributed data is managed by a standalone web application called .

The () is a small Java application that consists of a main JAR file, distributedCache.jar, the distributedCache.cfg file for configuration properties, and required dependencies in the jarlib directory. The microservice provides a centralized management and storage location for distributed data for clustered () applications. The is required for cluster operation; if the is not accessible, then all clustered applications will be in a 'wait' state until the is back online. The itself is not an application and may have different configuration requirements from the applications.

Setup considerations

The must be installed and run on a machine that is accessible by all nodes of the clustered application. By default, the microservice is installed on the () machine under the server/clustering/cache directory and is started as part of the startup sequence or on demand when a clustered application asks for it. If the URL advanced property setting has not been set, a clustered application calls to discover the status of the microservice. Then attempts to locate a running process and starts the if it is not running. Often in production environments, is a back end server behind a proxy, such as , or for performance reasons, it is installed on a dedicated server. In these cases, the must be started manually, and the clustered application URL service property must be set.

Note: For optimal performance, the should be moved to a dedicated server independent from .

To move or to reconfigure the URL:

1. Determine the URL to access the . Note the following important factors about the URL:
   • The URL can start with http or https depending on whether SSL is enabled in the distributedCache.cfg file.
   • If the server is fronted by a proxy, use the proxy URL.
   • If the port is masked, then the port should be omitted from the URL. Otherwise, 10001 is the default port which is also configurable in the distributedCache.cfg file.
   • A context path at the end of the URL is not required.

URL example:

https://myproxy.company.com/
https://myserver.company.com:10001/

Note: If you do the following step before the cluster is turned on, the changes will be propagated to other nodes when you replicate the application nodes.

2. Log in to the or application administrative UI and go to Application > Advanced Properties.
3. Search for URL and in the Current Value field, enter the URL of the server.
4. Scroll to the top of the page and click Save.
5. Go to the server where the application is installed and copy the server/clustering/cache directory and all subdirectories to a location on the new host. Note that the host must have JVM 1.8 or higher installed.
6. Open the server/clustering/cache/distributedCache.cfg file for editing.
7. Under the [AuthServer] section, add the URL in as_trusted_url so it can be properly registered with . For example:

as_trusted_url = $S{AuthServer.as_trusted_url, https://myserver.company.com:10001}

Note: Do not use a URL with a trailing slash for the value of the as_trusted_url property in the server/clustering/cache/distributedCache.cfg file.

Configuration properties checklist

Depending on your installation topology, the following configuration properties in the distributedCache.cfg file can be set.

• The microservice should be installed in a location with read and write access. If you install the microservice in a read-only location, you must change the logDir property to point to a writable location.
• Check the [REST] section and update the properties if necessary.
• Ensure to update the [AuthServer] properties if authentication is set to OIDC for . The URL you use in the as_trusted_url, must match the URL you specified in the Advanced Properties in the previous section. Do not append the context root (/dcm) to this property.
• If authentication is done through an LDAP server, ensure to update the [UserRegistry] properties. Note that the LDAP settings in the configuration file are slightly different from the settings you use during the setup, because the is a microservice and not a full application. The format for each setting is provided as comments in the distributedCache.cfg file. For example, the group_roles_mapping property in is set as:

JazzProjectAdmins=LDAPJazzAdmins,JazzAdmins=LDAPJazzAdmins,JazzUsers=LDAPJazzUsers,JazzDWAdmins=LDAPJazzAdmins,JazzGuests=LDAPJazzUsers

However, in it is set as:

{"LDAPJazzAdmins":"JazzProjectAdmins,JazzAdmins,JazzDWAdmins","LDAPJazzUsers":"JazzUsers,JazzDWAdmins"}

In short, in you specify Jazz roles to LDAP group mapping, but in you specify LDAP group to Jazz roles mapping. Note that if these properties are not set correctly, the following error message will be displayed in the log console:

The user registry settings may be set incorrectly. Please refer to the cfg file UserRegistry section.

• If you plan to monitor the performance counters that the microservice publishes, under [Counters], set the Enabled property to true and provide an MQTT broker URL in the broker property. publishes its counters via an MQTT Broker. When this is enabled, counters can be seen in the cluster application's administrative UI internal Counters Page. Alternatively, when counters are enabled in the configuration file, the http://_HOST_NAME:10001/dcm/counters URL displays counter data on demand, (port is applicable only if it is not masked). Publishing frequency is controlled by snapshotFrequency property under the [Logging] section. It is also possible to locally log the usage statistics of . To do this, set logSnapshotStats to true.

Running in secure mode

By default, runs in SSL mode. This is controlled by the useSsl property in the [REST] section. Additional SSL related configurations are specified in the [REST-SslContextFactory] section. Ensure to specify the correct file name and path in keyStorePath.

Configuration samples

For your reference, here are some sample configurations. Note that these samples point out only a few important parameters and not the entire configurations.

Sample 1, is:

• running from its default location on the host,
• not fronted by any proxy,
• using OSLC authentication with an LDAP server.

#===========================================
# REST settings
#===========================================
[REST]

port = $E{CACHE_PORT, 10001}

stylesheet = styles/devops.css

saveConfigAction = RESTART_SERVER

useSsl = true

authType = OIDC

#===========================================
# AuthServer settings
#===========================================
[AuthServer]

auth_url = $S{AuthServer.auth_url, https://JAS_HOST:9643/oidc/endpoint/jazzop}

client_id = dcm

client_name =

as_admin_id = $S{AuthServer.as_admin_id, ADMIN}

as_admin_pass = $S{AuthServer.as_admin_pass, ADMIN}

as_trusted_url = $S{AuthServer.as_trusted_url, https://JTS_HOST:10001}

#============================================
# LDAP UserRegistry settings.
#============================================
[UserRegistry]

base_group_DN = dc=jazz,dc=domain

group_name_property = cn

group_roles_mapping = {"LDAPJazzAdmins":"JazzProjectAdmins,JazzAdmins,JazzDWAdmins","LDAPJazzUsers":"JazzUsers,JazzDWAdmins"}

Sample 2, is:

• configured to run from another machine and NOT from its default location in the host,
• fronted by a proxy server and the port is masked for in the proxy,
• authenticated by a non-LDAP server.

#============================================
# REST settings
#============================================
[REST]

port = $E{CACHE_PORT, 10001}

stylesheet = styles/devops.css

saveConfigAction = RESTART_SERVER

useSsl = true

authType = OIDC

#============================================
# AuthServer settings
#============================================
[AuthServer]

auth_url = $S{AuthServer.auth_url, https://JAS_HOST:9643/oidc/endpoint/jazzop}

client_id = dcm

client_name =

as_admin_id = $S{AuthServer.as_admin_id, ADMIN}

as_admin_pass = $S{AuthServer.as_admin_pass, ADMIN}

as_trusted_url = $S{AuthServer.as_trusted_url, https://IHS_HOST}

IHS environments

If the is fronted by a proxy server, it must be configured to forward requests to the appropriate host and port where the is running. By default, an cluster expects the to be running on the same host as , configured securely on port 10001.

To modify an IHS configuration, make changes to the following files:

a) httpd.conf - Add the following entry to listen on the default port (10001). The default port can be changed in the distributedCache.cfg. Note that the following change is only required if you plan to have the port as part of the URL. If you plan to mask the port from its URL, then no change is required in the httpd.conf file.

Listen 10001
<VirtualHost *:10001>
SSLEnable
SSLServerCert <IHS server cert alias>
</VirtualHost>

b) plugin-cfg.xml - Add the following entry to forward requests on port 10001 to the appropriate server. Note that the only requires /jsa and /dcm context roots to be forwarded.

<ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="microservice_Cluster" PostBufferSize="64" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">
<Server ConnectTimeout="0" ExtendedHandshake="false" MaxConnections="-1" Name="microservice" ServerIOTimeout="9000" WaitForContinue="false">
<Transport Hostname="<JTS_HOSTNAME>" Port="10001" Protocol="https">
<Property Name="keyring" Value="<File Path of keystore file (kdb)>"/>
<Property Name="stashfile" Value="<File Path of stash file file (sth)>"/>
</Transport>
</Server>
</ServerCluster>

<UriGroup Name="microservice_Cluster_URIs">
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/jsa/*"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/dcm/*"/>
</UriGroup>

<Route ServerCluster="microservice_Cluster" UriGroup="microservice_Cluster_URIs" VirtualHostGroup="default_host"/>

c) Update the keystore database files to add an entry for the certificate based on the keystore updates to the plugin-cfg.xml file.

Running the

All the required files and scripts to start and stop the microservice are in the server/clustering/cache directory. To run the microservice, you must have a JVM version 1.8 or later installed. You can start the microservice by using the included script, or you can start it by using the following command:

java –Xms2G –Xmx3G -jar distributedCache.jar

You can change the values of JVM Xms and Xmx arguments in the command as per your requirements.

Note:

• _Install_Dir is the directory where is installed: server/clustering/cache, and JRE_Install_Dir is where JVM 1.8 or higher is installed.
• The startup script will change the current folder to that of the . This allows the to find the SSL keystore file and other required dependencies. If the is started by some other means, not all configurations may work.
• When the starts, it creates a file called service.lck under the home directory (server/clustering/cache). Similarly, the stop script deletes this service.lck file when the is shutdown properly. If you are using other means to shut down the , the service.lck file may still be present in that directory. Ensure to delete the service.lck file before restarting the , otherwise the may not start properly.

Starting the microservice:

To start the microservice, open a command windowshell and enter the following commands:

cd Path_To__Folder
start distributedCache.start.bat JRE_Bin_Path

cd PATH_To__Folder
./distributedCache.start.sh JRE_Bin_Path > /dev/null &

Where:

Path_To__Folder is where the is installed and JRE_Bin_Path is the location of the JRE/Bin folder. The JRE_Bin_Path is optional. If it's not specified, the system considers that is running from its default location and uses the JRE that is provided by the application.

Stopping the microservice:

To stop the microservice, open a command windowshell and enter the following commands:

cd Path_To__Folder
distributedCache.stop.bat JRE_Bin_Path

cd Path_To__Folder
./distributedCache.stop.sh JRE_Bin_Path

Where:

• Path_To__Folder is where the is installed and JRE_Bin_Path is the location of the JRE/Bin folder. The JRE_Bin_Path is optional. If it's not specified, the system considers that is running from its default location and uses the JRE that is provided by the application.

Unregistering from

() is used to authenticate the . To register the microservice, several parameters are configured in the distributedCache.cfg file under the AuthServer and UserRegistry sections as explained in the previous section. The will be registered to the automatically after it is started for the first time. After the is registered with the , the following properties are updated in the distributedCache.cfg file: client_secret and registration_completed = true. By default, will be registered on with client_id as dcm. If you need to update the registration for , you can delete the existing registration and registering again with :

1. Stop the .
2. Edit the distributedCache.cfg file.
1. Reset client_secret to the default value.

   client_secret = $S{AuthServer.client_secret}

2. Remove or comment out the registration_completed = true property.
3. Review the other properties mentioned in the [AuthServer] and [UserRegistry] sections in configuration file and update them accordingly if needed.
3. Unregister from by using the rmclient utility provided by JAS:

cd JAS_Install_Dir\cli
rmclient.bat -a AUTH_SERVER_URL -u ADMIN:ADMIN dcm

cd JAS_Install_Dir/cli
./rmclient -a AUTH_SERVER_URL -u ADMIN:ADMIN dcm

Where AUTH_SERVER_URL is the auth_url specified in the configuration file. The ADMIN:ADMIN parameter is the administrative user credentials, consisting of a user ID and password separated by a colon. And dcm is the Authorization Server client ID for the microservice registartion.

4. When you start the , it will re-register itself with by using the updated properties.

Troubleshooting the cluster

Synchronizing all applications with

After completing the custom setup, you might see the following diagnostic failure message for one or more applications:

Application: the discovery resource at https://JTS_HOST:9443/ccm/rootservices for "/ccm" has different OAuth domains than the entry does.

OAuth domains from the discovery resource: [https://JTS_HOST:9443/ccm] Application OAuth domains: []

This error might be caused by an incomplete synchronization of applications from the repository tools (repotools) commands. To fix the issue, complete the following steps to manually synchronize the applications:

1. First, obtain the application ID that you need to synchronize with . Open a command shellwindow and enter the following commands:

   cd Jazz_Install_Dir/\server
   ./repotools-jtsrepotools-jts -listFriends repositoryURL=https://JTS_HOST:9443/jts adminUserId=adminUserId adminPassword=adminPassword

2. Obtain the application ID from the previous step and enter the following command:

   ./repotools-jtsrepotools-jts -synchFriend repositoryURL=https://JTS_HOST:9443/jts adminUserId=adminUserId adminPassword=adminPassword friendId=FriendID_From_PreviousStep

3. After the repotools commands are successfully completed, return to the diagnostic page and run the diagnostics to ensure no more errors occur.

Accepting all "Content" for requests through proxy

If you installed or another similar proxy server in front of the clustered environment, you must ensure that it is enabled to accept the content (payload) from all HTTP requests such as GET, PUT, POST, and so on. If this capability is not enabled, you might receive HTTP 400 Bad request errors in the proxy server because some of the HTTP GET requests are sent with a payload and rejected by the proxy.

To avoid such errors, in the proxy server's plugin-cfg.xml file, set AcceptAllContent="true".

Discarding HAProxy messages in the log files

The number of requests between or cluster members and can quickly increase and fill the disk space. You can modify the rsyslog config file so that certain HAProxy messages are discarded while logging the rest. To do this, add a tilde (~) to the end of the line that you want to discard. Example:

:msg, regex, "/map/" ~
local2.* /var/log/haproxy.log
& stop