Setting up load balancing for multiple LQEs

1. To locate and start iKeyman, run the following command.

   /opt/IBM/HTTPServer/java/jre/bin/ikeyman

2. To create a new key database file, click Key Database File > New.
   

1. In the New dialog box, from the Key database Type list, select CMS.
   Note: To enable the CMS provider, add the following line to the java.security file.

   security.provider.10=com.ibm.security.cmskeystore.CMSProvider

   If you are using iKeyman in /opt/IBM/HTTPServer/java/jre/bin, then you can find the java.security file here.

   /opt/IBM/HTTPServer/java/jre/lib/security/java.security

   
2. In the File Name field, type the name of your keystore. For instance, IHS_KEY.kdb.
3. In the Location field, click Browse to enter a path for your keystore. For example,

   /opt/IBM/HTTPServer/bin

   Remember: Add this path to the bin directory of the IHS server. Later, you must enter the same path for the httpd.conf file.
4. Click OK.
5. In the Password Prompt dialog box, enter a password and then confirm.
6. Select the Stash password to a file checkbox.
   Remember: Do not specify an expiration date. Otherwise, when this certificate expires you need to create a new stash file.

1. To create a new self-signed certificate, click New Self-Signed.
   
2. In the Create New Self-Signed Certificate dialog box, enter the following information about the server.
   • In the Key Label field, type the name of the certificate. For example, default.
   • In the Common Name field, enter the web server name.
   • In the Validity Period field, type 3650. The value 3650 is entered so that you do not need to create a new certificate in the same year.
   • Click OK.
     
3. Verify that the default certificate was created.
   

mkdir /opt/IBM/WebSphere/Plugins/config/webserver1

1. Start iKeyman on the Liberty server where LQE is deployed.
   Note: If your IHS server is on a different server than your Liberty server, then you can use the iKeyman tool that comes with the ELM installation. For example, in this installation process, the iKeyman tool is located here.

   /opt/IBM/JazzTeamServer/server/java/jre/bin/ikeyman

2. In the iKeyman tool, click the open icon.
   
3. From the Key database type list, select JKS.
4. From the Files of Type list, select All Files.
5. Click Browse and select the location of the Liberty profile keystore. For example,

   /opt/IBM/JazzTeamServer/server/liberty/servers/clm/resources/security/ibm-team-ssl.keystore

   
6. Enter the password.
7. Click Extract Certificate.
   
8. In the New dialog box, in the Certificate file name field, enter the name for the certificate that identifies the LQE server. For example, cert-lqe1.arm.
9. Accept the default location.
   
10. Click OK. Verify that the certificate is created.
11. Copy this certificate into the WebServer1 directory. Now close iKeyman on the Liberty profile server.
12. Repeat steps 2-a to 2-k for the other LQE server.

1. From the Key database type field, select CMS.
2. In the File Name field, enter a name of your keystore file. For example, plugin-key.kdb.
3. Click Browse to select the path of your keystore file. For example, /opt/IBM/WebSphere/Plugins/config/webserver1.
   
4. Click OK. The Password Prompt dialog box opens.
5. Enter the password and confirm.
6. Select the Stash password to a file checkbox.
   Remember: Do not specify an expiration date. Otherwise, when this file expires you must create a new stash file.
   

1. Import the signer certificate from the first Liberty server (cert-lqe1.arm). From the Key database content list, select Signer Certificates.
   
2. Click Add. The Open dialog box opens.
3. Click Browse to go to the /opt/IBM/WebSphere/Plugins/config/WebServer1 directory, and then select the cert-lqe1.arm file.
   
4. Click OK.
5. In the Enter a Label dialog box, type LQE1.
   
6. To import the cert-lqe2.arm file into the keystore, repeat steps 4-a to 4-e. Specify a different label, for example, LQE2.

1. In the Liberty server, update the server.xml configuration file on each clustered node with the unique cloneId.
2. Update the httpSession tag in the server.xml file in all the clustered nodes to add the cloneId field. It is the clone identifier of the cluster member.

   <httpSession invalidateOnUnauthorizedSessionRequestException="true" cookieSecure="true" cloneId="ccm1"/>

   Note: Within a cluster, this identifier (cloneId) must be unique for each node to maintain session affinity. When set, this name overwrites the default name that is generated by the server. The same cloneId is used in the web plug-in configuration to identify Affinity Requests.

1. Copy the plugin-cfg.xml file over to your IHS server and place it in the /opt/IBM/WebSphere/Plugins/config/webserver1 directory.
   Note: In Liberty profile 6.0.4 or later, the plugin-cfg.xml is generated automatically at server startup under the server_install_dir/server/liberty/servers/clm/logs/state directory.
2. Copy the plugin-cfg.xml file for each of the Liberty profile server and rename them to plugin-cfg-lqe1.xml and plugin-cfg-lqe2.xml. Make a copy of one of them and use it to merge the content of the two files and call it plugin-cfg.xml.
3. Open the plugin-cfg.xml configuration file and verify the contents. It contains code snippet that is similar to the following code snippet:

   <?xml version="1.0" encoding="UTF-8"?><!--HTTP server plugin config file for clm generated on 2019.08.26 at 12:05:12 EDT-->
   <Config ASDisableNagle="false" AcceptAllContent="false" AppServerPortPreference="HostHeader" ChunkedResponse="false" FIPSEnable="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="false" TrustedProxyEnable="false" VHostMatchingCompat="false">
      <Log LogLevel="Error" Name="/opt/IBM/WebSphere/Plugins/logs/webserver1/http_plugin.log"/>
      <Property Name="ESIEnable" Value="true"/>
      <Property Name="ESIMaxCacheSize" Value="1024"/>
      <Property Name="ESIInvalidationMonitor" Value="false"/>
      <Property Name="ESIEnableToPassCookies" Value="false"/>
      <Property Name="PluginInstallRoot" Value="/opt/IBM/WebSphere/Plugins"/>
   <!-- Configuration generated using httpEndpointRef=defaultHttpEndpoint-->
   <!-- The default_host contained only aliases for endpoint defaultHttpEndpoint.
            The generated VirtualHostGroup will contain only configured web server ports:
                   webserverPort=80
                   webserverSecurePort=443 -->
      <VirtualHostGroup Name="default_host">
         <VirtualHost Name="*:80"/>
         <VirtualHost Name="*:443"/>
      </VirtualHostGroup>
      <ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="clm_default_node_Cluster" PostBufferSize="0" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60" ServerIOTimeoutRetry="-1">
         <Server CloneID="lqe1" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="default_node_clm" ServerIOTimeout="900" WaitForContinue="false">
            <Transport Hostname="lqe-1.myserver.ibm.com" Port="9080" Protocol="http"/>
            <Transport Hostname="lqe-1.myserver.ibm.com" Port="9443" Protocol="https">
               <Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb"/>
               <Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.sth"/>
            </Transport>
         </Server>
         <PrimaryServers>
            <Server Name="default_node_clm"/>
         </PrimaryServers>
      </ServerCluster>
      <UriGroup Name="default_host_clm_default_node_Cluster_URIs">
         <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/lqe/*"/>
         <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/IBMJMXConnectorREST/*"/>
         <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/ibm/api/*"/>
         <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/ibm/adminCenter/explore-1.0/*"/>
         <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/ibm/adminCenter/serverConfig-1.0/*"/>
         <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/adminCenter/*"/>
      </UriGroup>
      <Route ServerCluster="clm_default_node_Cluster" UriGroup="default_host_clm_default_node_Cluster_URIs" VirtualHostGroup="default_host"/>
   </Config>

1. Examine the contents of the plugin-cfg.xml file and look for the following line.

   <Log LogLevel="Error" Name= ************

2. Verify that the following entry matches the directory that you created for your Liberty plug-in.

   <Log LogLevel="Error" Name="/opt/IBM/WebSphere/Plugins/logs/webserver1/http_plugin.log"/>

3. Update the plugin-cfg.xml file with the values for the keystore and certificate, which were created in the Key database and Importing Jazz signer certificate section sections.

   <Property Name="keyring" Value="keyring.kdb"/>
   <Property Name="stashfile" Value="keyring.sth"/>
   <Property Name="certLabel" Value="LibertyCert"/>
   

4. Modify the plugin-cfg.xml to match the certificate information that you created for the Liberty plug-in.

   <Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/config/WebServer1/plugin-key.kdb"/>
   <Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/config/WebServer1/plugin-key.sth"/>
   <Property Name="certLabel" Value="LQE1"/>
   

5. In the plugin-cfg.xml file, modify the ServerCluster section.

   <ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" 
   IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="lqe_node_Cluster" 
   PostBufferSize="0" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60"
   ServerIOTimeoutRetry="-1">

6. Edit the Server section to have a unique CloneId and Name. The CloneID must match the clone ID that was specified in the server.xml.
7. Duplicate the Server section and edit it to match the LQE2 node.

   <Server CloneID="lqe2" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="lqe2_node" ServerIOTimeout="900" WaitForContinue="false">
            <Transport Hostname="lqe-2.myServer.ibm.com" Port="9080" Protocol="http"/>
            <Transport Hostname="lqe-2.myServer.ibm.com" Port="9443" Protocol="https">
               <Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb"/>
               <Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.sth"/>
               <Property Name="certLabel" Value="LQE2"/>
            </Transport>
         </Server>

8. Edit the PrimaryServers section with the names of the servers specified in the Server section.

   <PrimaryServers>
            <Server Name="lqe1_node"/>
            <Server Name="lqe2_node"/>
         </PrimaryServers>

9. Edit the UriGroup section with the cluster name and leave the entry.

   <UriGroup Name="lqe_node_Cluster_URIs">
         <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/lqe/*"/>
      </UriGroup>

10. Edit the Route section with the names specified in the relevant sections.

    <Route ServerCluster="lqe_node_Cluster" UriGroup="lqe_node_Cluster_URIs" VirtualHostGroup="default_host"/>

The merged plugin-cfg.xml file must contain the following content.

<?xml version="1.0" encoding="UTF-8"?><!--HTTP server plugin config file for clm generated on 2019.08.26 at 08:05:47 EDT-->
<Config ASDisableNagle="false" AcceptAllContent="false" AppServerPortPreference="HostHeader" ChunkedResponse="false" FIPSEnable="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="false" TrustedProxyEnable="false" VHostMatchingCompat="false">
   <Log LogLevel="debug" Name="/opt/IBM/WebSphere/Plugins/logs/webserver1/http_plugin.log"/>
   <Property Name="ESIEnable" Value="true"/>
   <Property Name="ESIMaxCacheSize" Value="1024"/>
   <Property Name="ESIInvalidationMonitor" Value="false"/>
   <Property Name="ESIEnableToPassCookies" Value="false"/>
   <Property Name="PluginInstallRoot" Value="/opt/IBM/WebSphere/Plugins"/>
<!-- Configuration generated using httpEndpointRef=defaultHttpEndpoint-->
<!-- The default_host contained only aliases for endpoint defaultHttpEndpoint.
         The generated VirtualHostGroup will contain only configured web server ports:
                webserverPort=80
                webserverSecurePort=443 -->
   <VirtualHostGroup Name="default_host">
      <VirtualHost Name="*:80"/>
      <VirtualHost Name="*:443"/>
   </VirtualHostGroup>
   <ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="false" LoadBalance="Round Robin" Name="lqe_node_Cluster" PostBufferSize="64" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60" ServerIOTimeoutRetry="-1">
      <Server CloneID="lqe2" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="lqe2_node" ServerIOTimeout="900" WaitForContinue="false">
         <Transport Hostname="lqe-2.myserver.ibm.com" Port="9080" Protocol="http"/>
         <Transport Hostname="lqe-2.myserver.ibm.com" Port="9443" Procol="https">
            <Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb"/>
            <Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.sth"/>
            <Property Name="certLabel" Value="LQE2"/>
         </Transport>
      </Server>
      <Server CloneID="lqe1" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="lqe1_node" ServerIOTimeout="900" WaitForContinue="false">
         <Transport Hostname="lqe-1.myserver.ibm.com" Port="9080" Protocol="http"/>
         <Transport Hostname="lqe-1.myserver.ibm.com" Port="9443" Protocol="https">
            <Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb"/>
            <Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.sth"/>
            <Property Name="certLabel" Value="LQE1"/>
         </Transport>
      </Server>

      <PrimaryServers>
         <Server Name="lqe1_node"/>
         <Server Name="lqe2_node"/>
      </PrimaryServers>
   </ServerCluster>
   <UriGroup Name="lqe_node_Cluster_URIs">
      <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/lqe/*"/>
   </UriGroup>
   <Route ServerCluster="lqe_node_Cluster" UriGroup="lqe_node_Cluster_URIs" VirtualHostGroup="default_host"/>
</Config>

• The port that you are listening on. The port 443 is the default port.
• The virtual hostname (a * means accept all the traffic that is coming on port 443).
• The path to your key file that was created earlier.
• The path to the stash file that was created earlier.

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 0.0.0.0:443
## IPv6 support:

<VirtualHost *:443>
SSLEnable
SSLProtocolDisable SSLv2 SSLv3
 KeyFile /opt/IBM/HTTPServer/bin/IHS_KEY.kdb
 SSLStashFile /opt/IBM/HTTPServer/bin/IHS_KEY.sth
</VirtualHost>

• Windows

  LoadModule was_ap22_module "C:\IBM\WebSphere\Plugins\bin\64bits\mod_was_ap22_http.dll"
  WebSpherePluginConfig "C:\IBM\WebSphere\Plugins\config\WebServer1\plugin-cfg.xml"

• Linux®

  LoadModule was_ap22_module "/opt/IBM/WebSphere/Plugins/bin/64bits/mod_was_ap22_http.so"
  WebSpherePluginConfig "/opt/IBM/WebSphere/Plugins/config/WebServer1/plugin-cfg.xml"

Important parameters that are used in this configuration are described in the following list.

LoadBalance

The following values can be specified for this attribute.

• Round Robin: The round-robin implementation has a random starting point. The first application server is picked randomly. Round Robin is then used to pick application servers from that point forward. This implementation ensures that in multiple process-based web servers, all of the processes do not start by sending the first request to the same application server.
• Random: The random implementation also has a random starting point. However, with this implementation all subsequent servers are also randomly selected. Therefore, the same server might get selected repeatedly while other servers remain idle. The default load-balancing type is Random.

LoadBalanceWeight

LoadBalanceWeight is a starting weight. The value is dynamically changed by the plug-in during run time. The weight of a server (or clone) is reduced each time a request is assigned to that clone. When all weights for all servers drop to 0 or less, the plug-in must readjust all of the weights so that they are greater than 0. Using a starting value of only 2 (default), means that the weights get to 0 quickly and the plug-in is constantly readjusting the weights. Therefore, it is recommended to start with a higher LoadBalanceWeight. The WebSphere Application Server administrative console allows a value up to 20 for the LoadBalanceWeight. However, it is possible to manually edit the plugin-cfg.xml file and specify some other value for LoadBalanceWeight that is higher than 20.

At run time, the LoadBalanceWeight of each application server in a cluster are normalized by their highest common factor. For example, 100, 90, 80 have a common factor of 10. So, these configured weights would be divided by 10 at run time, resulting in actual starting weights of only 10, 9, 8. Setting all clones to the same starting LoadBalanceWeight (for example: 20, 20, 20) results in an actual starting weight of only 1 for each because of normalization. So, it is recommended to set the weight of at least one of the clones to be off by a value of 1. For example, if there are three clones, you might choose the starting value of LoadBalanceWeights to be: 20, 20, 19. After normalization the weights remain unchanged.

Note: Recommended values for LoadBalanceWeight = All clones must have the same value, except one clone must be off by one.

Affinity Requests

Affinity Requests are requests that contain a session cookie (JSESSIONID). Session affinity means that all requests of the same JSESSIONID is sent to the same application server. For example, if the first request is sent to clone5, then the next request from that same client (Affinity Requests) is also sent to clone5, regardless of the value of LoadBalanceWeight. If you use Round Robin for LoadBalance option, then by default, the Affinity Requests do not lower the "weight" (IgnoreAffinityRequests="true"). This choice can cause an uneven distribution across the servers in environments that uses session affinity. But, if IgnoreAffinityRequests="false", then the weight is reduced by each Affinity Request. This leads to a more balanced random environment.

When you use Random, the Affinity Request is still handled correctly and sent to same cloneid, as before. But new requests are routed randomly, and the LoadBalanceWeight is not used.

Note: The IgnoreAffinityRequests option is only available in the web server plug-in version 6.1 or higher.

ConnectTimeout

ConnectTimeout indicates the time for which the plug-in must wait when it is trying to open a socket to the application server. If there are streams already open and available to the application server, the plug-in uses one of those streams. However, sometimes the plug-in must open a new stream to the application server. The value for ConnectTimeout must be small, so that the plug-in opens a new stream to the application server quickly.

A ConnectTimeout value of 0 means never timeout. In that case, the timeout is left up to the operating system's TCP layer, which is not an ideal scenario. It is better to specify a small positive number, such as 5 seconds. The recommended value = 5.

MaxConnections

MaxConnections specifies the maximum number of pending connections to an application server that can be flowing through a web server process at any point in time. Specify one element for each server. It is not used to determine when to fail over (mark the server "down"). When a request is sent from the plug-in to the WebSphere Application Server, it is called a PendingRequest, until the response comes back. If the application running in WebSphere Application Server is handling requests quickly, each request is in PENDING state for a short time. Therefore, under ideal conditions, MaxConnections is not needed and the default value of MaxConnections is -1, which means unlimited.

However, sometimes an application might start to become overwhelmed and the application might not be able to handle the requests as quickly. Therefore, pending requests start to build up. MaxConnections can be used to put a limit on the number of pending requests per server.

When the MaxConnections limit is reached, the plug-in stops sending requests to that application server, but it is not marked down. The optimal value for MaxConnections depends on how quickly the application and the application server respond to each request. If normal responses are returned in less than 1 second, it might be appropriate to set a low value for MaxConnections, like 20. However, if it normally takes several seconds to get a response from the application, you might use a higher value for MaxConnections, such as 100.

Note: If the MaxConnections limit is reached, then the plug-in does not send any more requests to that server until responses come back for the current pending requests. The count of the pendingRequests drops back down below the MaxConnections limit.

Tip:

With MaxConnections="-1", use LogLevel="Stats" to monitor the pendingRequests numbers in the plug-in log, under normal conditions. Then, choose a value for MaxConnections that is significantly higher than the highest number that is displayed in the log. This method enables you to determine a MaxConnections value that is right for your specific environment.

ServerIOTimeout

ServerIOTimeout means how long must the plug-in wait for a response from the application. After the socket is opened, the plug-in sends the request to the application server. The application processes the request and a response is sent back to the client, through the plug-in. How long must that take? What is reasonable, based on the application? There is no single correct answer here. It depends on the application.

If the application is quick to respond, then you can use a lower value for ServerIOTimeout. However, if the application requires more time to process the requests (such as, retrieve data from a database), then you can use a higher number for ServerIOTimeout. When you use the value 0, it means that the plug-in does not timeout the request. This scenario is not the ideal situation.

A positive value means that the plug-in does not mark down the application server after a ServerIOTimeout occurs. So, if you want the plug-in to continue sending requests to the timed-out application server, use a positive value. A negative value means that the plug-in marks that the application server is down after a ServerIOTimeout occurs. So, if you want the plug-in to immediately mark the application as down and fail over to another application server in the same cluster, use a negative value for ServerIOTimeout.