Building a large scale WebSphere Application Server Liberty collective topology

Rapid construction and sample utilization of a large collection of Liberty servers


Liberty collectives are sets of IBM WebSphere Application Server Liberty servers that are configured to be part of the same administrative domain. Configuration and state data about the Liberty collective is stored in an active operational repository. Liberty servers join a collective by registering to the collective controller. These members share information about themselves through the operational repository of the controller. Figure 1 shows an example.

Figure 1. Example collective (1 to N members)
Example collective (1 to n members)
Example collective (1 to n members)

In addition to building the collective, you can define Liberty clusters by a specifying the clusterMember attribute in the server.xml file:

<clusterMember name=”myClusterX” />

There are many benefits to building a Liberty collective for managing your collection of Liberty servers. Utilizing a collective creates an administrative control point for performing server management, cluster management, file transfer, and access to MBean operations. You can perform many operations from a single point of administration, including:

  • Starting and stopping Liberty servers.
  • Starting and stopping Liberty clusters.
  • Distributing files to Liberty servers or Liberty clusters (which is useful for updating server configurations or application installations).
  • Generating the web server plugins.
  • Querying collective member statistics, such as heap and JVM information.

Building and managing a large, highly available collective

Collective controllers can be configured to be highly available replica sets. Collective controllers that belong in the same replica set synchronize their repository information, so that when an outage occurs on any collective controller, the other collective controllers in the replica set can continue to accept and process the workload.

There must be at least three collective controllers in the replica set to establish a quorum and take advantage of the highly available capabilities of the collective.

To complete the configuration of a replica set following the steps below requires:

  • IBM WebSphere Application Server Liberty Network Deployment V8.5.5
  • Jython client

To manually configure a replica set:

  1. Install WebSphere Application Server Liberty Network Deployment.
  2. Create a server using this command:

    server create <Controller Name>

  3. Use this collective command to convert the Liberty server into a collective controller:

    collective create <Controller Name> - -keystorePassword=<Keystore Password>

    This collective controller will be referred to as the initial controller.

  4. A set of output will display. Copy this output into the server.xml file of the newly created controller. The location may vary, but it is typically under [LIBERTY_ROOT]/usr/servers/<Server Name>.
  5. Start the initial controller by issuing the command:

    server start <Controller Name>

  6. On another system, deploy Liberty and create another collective controller by issuing the command:

    server create <Controller Name>

    If you are running this on the same system, you will need to supply a different name than that of the initial controller.

  7. Run the replicate command on the second collective controller, for example:

    collective replicate <Controller Name> -- host=<hostname of initial controller> --port=<SSL port of initial controller> --user=<user name of security defined in the initial controller> --password=<password of user name defined in the initial controller> --keystorePassword=<Keystore password from the initial controller>

  8. Copy and paste the output of the replicate command into the server.xml file of the second collective controller.
  9. Repeat steps 6 through 8 on the third controller.
  10. On a separate machine that has Jython installed, download the sample script available from the WASdev community.
  11. If the Jython client and the initial controller are not on the same system, place the keystore from the initial controller to a location available to the Jython client.
  12. Run this command, using the initial controller as the target:

    ./jython add ––truststore=<path to trust store of initial controller> --truststorePassword=<Truststore Password of Initial Controller> ––host=<hostname of initial controller> --port=<SSL port of initial controller> --user=<user name of security defined in the initial controller> --password=<password of user defined in the initial controller> --endpoint=<hostname and replica port of second controller> --debug

    For example, the actual command might look like this:

    ./jython add –truststore=key.jks --truststorePassword=Management –– --port=9443 --user=adminUser --password=adminPassword --debug

  13. Repeat step 12, using the third controller as the --endpoint hostname.

Creating and joining collective members

When building a large scale collective, you apply a technique that involves “templating” common values in the Liberty server configuration files, such as server.xml,, and jvm.options. You then use the cat and sed commands, which are available on UNIX® systems, to customize the collective members as they are being created. These steps are placed in a shell script that deterministically sets the number of members to create, joins the members to the collective, copies the templates, then replaces the necessary variables. This process is most effective when creating many members on a single system. Once the script is completed, the specified number of collective members are created, each with server name of myMemberXXXXX, where XXXXX is the HTTP port number for easier identification of the collective member.

Listing1 shows a sample server_template.xml file, which is the template for server.xml. Any replaceable or customizable attribute for a Liberty collective member is denoted with rep<Attribute>.

Listing 1. server_template.xml
<server description="myMember_repMemPort">
    <!-- Enable features -->
    <httpEndpoint id="defaultHttpEndpoint"
                  httpsPort="repMemSecPort" >
        <tcpOptions soReuseAddr="true" />
    <variable name="defaultHostName" value="repMemHost" />

    <clusterMember name="repClusterName" />

    <!-- Connection to the collective controller -->
    <collectiveMember controllerHost="repControllerHost"
                      controllerPort="repControllerSecPort" />

In this example, you will replace:

  • server description(repMemPort)
  • collective member hostname(repMemHost)
  • collective member http(repMemPort) and https(repMemSecPort) end points
  • cluster name(repClusterName)
  • collective controller host(repControllerHost)
  • collective controller port number(repControllerSecPort).

You can also take advantage of replacing a variable named defaultHostName(repMemHost) so that variable can be used throughout the server.xml file.

Use this scheme to template your server.xml file, focusing on customizable values that will be unique for each collective member, such as port numbers and server names.

Building up the collective members

The basic algorithm in the shell script runs inside a while loop:

  1. Create the member with the server create command.
  2. Join the member with the collective join command.
  3. Copy the server configuration file template over the existing configuration files.
  4. Replace the variables in the server configuration files using cat and sed commands.
  5. Start the server.

At the top of the shell script, you declare the values that are being substituted in from the sed command and a JVM argument that enables the collective join to proceed without user interaction. By declaring the following JVM argument, SSL keys are exchanged automatically:

export JVM_ARGS

Listing 2 shows some of the values that will be substituted into the collective member's server.xml file. The number of values should correspond directly with the values that need to be substituted in the final server.xml for the collective member.

Listing 2. server.xml substitued values

A simple while loop is used to control the number of collective members that are created on the system. Variables (X and Y) that control the number of iterations the loop executes are also used to calculate the unique port numbers needed for each collective member. A third variable (Z) is used to generate ports that are usable for the non-root users. Listing 3 will create 200 collective members with each server being assigned consecutive ports starting at 11001.

Listing 3. Creating collective members
while [ $X -le $Y ]
Z=`expr $X + 11000`
X=`expr $X + 1`

Once the loop conditions are set, you can turn your attention to populating the server.xml file for each of the created collective members. The basic pseudo code of the substitution is like this:

cat <template file> | sed -e “s/<value to replace in server.xml>/<replacement value from declared variables above>/g” | sed -e “s/<next value to replace>/<replacement value from declared variables above>/g” > <location of the server.xml>

Listing 4 shows an example of a substitution in the script.

Listing 4. Populating server.xml
cat ${ATLAS_TOOL}/server_template.xml | sed -e "s/repMemHost/${MemHost}/g" | sed -e 
"s/repMemPort/$X/g" | sed -e "s/repMemSecPort/$Z/g" | sed -e "s/repControllerHost/$
{ControllerHost}/g" | sed -e "s/repControllerSecPort/ ${ControllerSecPort}/g" | sed -e 
"s/repmyMember/myMember${X}/g" |sed -e "s/repClusterName/${ClusterName}/g" | sed -e 
"s|repWLP_ROOT|${WLP_ROOT}|g" > ${WLP_ROOT}/usr/servers/myMember$X/server.xml

The sed command will replace any instance of repMemHost with the value defined for ${MemHost}, as declared at the top of the shell script, and pipe it into the server.xml of the member being configured (Listing 5).

Listing 5. Value substitution in server.xml
# Modify the following values which are substituted into the server.xml of the Collective Member
export JAVA_HOME
# The following will automatically accept a connection when the certificate connection 
# question is prompted.
export JVM_ARGS

cd ${WLP_ROOT}/bin

while [ $X -le $Y ]
  Z=`expr $X + 11000`
  echo "creating myMember${X} on ${MemHost} ..."
  ./server create myMember${X}
  ./collective join myMember${X} --host=${ControllerHost} --port=${ControllerSecPort} 
--user=${CTL_USER} --password=${CTL_USER_PASSWD} --keystorePassword=${CTL_KEY_PASSWD}

  cat ${ATLAS_TOOL}/server_template.xml | sed -e "s/repMemHost/${MemHost}/g" | sed -e 
"s/repMemPort/$X/g" | sed -e "s/repMemSecPort/$Z/g" | sed -e "s/repControllerHost/$
{ControllerHost}/g" | sed -e "s/repControllerSecPort/${ControllerSecPort}/g" | sed -e 
"s/repmyMember/myMember${X}/g" |sed -e "s/repClusterName/${ClusterName}/g" | sed -e 
"s|repWLP_ROOT|${WLP_ROOT}|g" > ${WLP_ROOT}/usr/servers/myMember
  ./server start myMember${X}
  sleep 3
  echo "=============================="
  X=`expr $X + 1`

Working with the collective

After building the large liberty collective, you can then begin to run system management scripts against some or all of the systems. Be aware that each Jython invocation requires the key.jks file or trust store from the target controller. The controller then dispatches to the appropriate collective member to obtain either the information that's required or the Mbean being invoked. The listings below show a few examples of scripts that were executed against the collective:

  1. Show JVM statistics
  2. Show threadpool information
  3. File transfer
  4. Generate a plugin-cfg.xml file

For reference, here are the variable declarations:

  • CTL_WLP_ROOT=/opt/liberty/wlp
  • ControllerSecPort=9443
  • ControllerName=myController
  • CTL_USER=adminUser
  • CTL_USER_PASSWD=adminPassword
  • CTL_KEY_PASSWD=Management
  • # The member's WLP_ROOT
  • WLP_ROOT=/opt/liberty/wlp
  • JYTHON_KEY_DIR=/opt/jython2.5.3/keys
  • JYTHON_FILEXFER_DIR=/opt/jython2.5.3/Xfer

a. Show JVM statistics

Listing 6 is an example of how a system administrator of a large Liberty Collective can view JVM utilization of each of the registered Liberty servers, such as UsedMemory, FreeMemory and Heap. The script is derived from the Update server config script available from the WASdev community. (See the WebSphere Application Server Information Center for more examples).

Listing 6. Show JVM statistics on collective member (UsedMemory, FreeMemory and Heap)
./jython -J-Xms1024m -J-Xmx2048m show --truststore=${JYTHON_KEY_DIR}/key.jks
--truststorePassword=${CTL_KEY_PASSWD} --host=${ControllerHost} --port=${ControllerSecPort} 
--user=${CTL_USER} --password=${CTL_USER_PASSWD} --serverHost=${MemHost} --serverUserdir=${WLP_ROOT}
/usr --serverName=myMember${X} --queryAtt=UsedMemory --debug
./jython -J-Xms1024m -J-Xmx2048m show --truststore=${JYTHON_KEY_DIR}/key.jks
--truststorePassword=${CTL_KEY_PASSWD} --host=${ControllerHost} --port=${ControllerSecPort} 
--user=${CTL_USER} --password=${CTL_USER_PASSWD} --serverHost=${MemHost} --serverUserdir=${WLP_ROOT}
/usr --serverName=myMember${X} --queryAtt=FreeMemory --debug

./jython -J-Xms1024m -J-Xmx2048m show --truststore=${JYTHON_KEY_DIR}/key.jks 
--truststorePassword=${CTL_KEY_PASSWD} --host=${ControllerHost} --port=${ControllerSecPort} 
--user=${CTL_USER} --password=${CTL_USER_PASSWD} --serverHost=${MemHost} --serverUserdir=${WLP_ROOT}
/usr --serverName=myMember${X} --queryAtt=Heap

b. Show threadpool information

Listing 7 is an example of how a system administrator of a large Liberty collective can view application efficiency or debug threadpool resource contention for any of the registered Liberty servers. The script is derived from the Update server config script available from the WASdev community.

Listing 7. Show threadpool information on collective member
./jython -J-Xms1024m -J-Xmx2048m view --truststore=${JYTHON_KEY_DIR}/
key.jks --truststorePassword=${CTL_KEY_PASSWD} --host=${ControllerHost} --port=${ControllerSecPort}
--user=${CTL_USER} --password=${CTL_USER_PASSWD} --serverHost=${MemHost} --serverUserdir=${WLP_ROOT}
/usr --serverName=myMember${X} --queryAtt=ActiveThreads --debug

./jython -J-Xms1024m -J-Xmx2048m view --truststore=${JYTHON_KEY_DIR}/
key.jks --truststorePassword=${CTL_KEY_PASSWD} --host=${ControllerHost} --port=${ControllerSecPort}
--user=${CTL_USER} --password=${CTL_USER_PASSWD} --serverHost=${MemHost} --serverUserdir=${WLP_ROOT}
/usr --serverName=myMember${X} --queryAtt=PoolSize

c. File transfer

File transfer, both upload and download, is one of the most commonly used operations for managing a large Liberty collective. The ability to copy files enables Liberty administrators and users to update server configurations, apply application updates, download files from the Liberty server, and even deploy Liberty servers to other systems.

Listing 8. File transfer
./jython -J-Xms1024m -J-Xmx2048m put --truststore=${JYTHON_KEY_DIR}/key.jks 
--truststorePassword=${CTL_KEY_PASSWD} --host=${ControllerHost} --port=${ControllerSecPort} 
--user=${CTL_USER} --password=${CTL_USER_PASSWD} --serverHost=${MemHost} --serverUserdir=
${WLP_ROOT}/usr --serverName=myMember${X} --localDir=${JYTHON_FILEXFER_DIR} --debug

d. Generate a plugin-cfg.xml file

Generating a plugin-cfg.xml file for the Liberty collective cluster provides web servers with the necessary information to connect and interact with the back end application servers. The plugin-cfg.xml Jython ebables one script to generate a merged plugin-cfg.xml file that can then be placed on the web server, without having to generate each one individually or manually merge the plugin-cfg.xml from each server.

Listing 9. Generate a plugin-cfg.xml for the cluster
./jython -J-Xms1024m -J-Xmx2048m myCluster22 --truststore={JYTHON_KEY_DIR}/
key.jks --truststorePassword=${CTL_KEY_PASSWD} --host=${ControllerHost} --port=${ControllerSecPort}
--user=${CTL_USER} --password=${CTL_USER_PASSWD}  --debug


The features offered within a Liberty collective provide powerful and extensible capabilities to system administration. Liberty collective controllers can be configured into an HA environment and then leveraged to perform tasks that are distributed to the registered Liberty servers, eliminating the need to write complex scripts that connect to each individual Liberty server. By leveraging shell scripts and readily avaliable UNIX functions like sed, you were able to create a large Liberty collective managed by a replica set of Liberty controllers in a relatively short period of time. You then saw how relatively simple scripts can be used to discover system status, JVM status, and even generate a plugin-cfg.xml file for a cluster that resides in the collective.

Using the techniques described in this article, Liberty administrators can develop and deploy scripts that will increase their efficiency and improve the reliability of a large Liberty collective topology.

Downloadable resources

Related topics


Sign in or register to add and subscribe to comments.

ArticleTitle=Building a large scale WebSphere Application Server Liberty collective topology