Configuring a Liberty collective

You can organize Liberty servers into collectives to support clustering, administration, and other operations that act on multiple Liberty servers. By using collectives, you can efficiently and accurately deliver application services to your application.

Before you begin

Install Liberty with the administrative features. For installation instructions, see the Before you begin section of Setting up the server-management environment for Liberty by using collectives. The Liberty administrative features provide multiple-server management features for collectives, clusters, scaling, dynamic routing, and Admin Center.

For IBM i platformsFor Distributed operating systemsThe collectiveController-1.0 feature and its capabilities are available only in WebSphere® Application Server Network Deployment Liberty and WebSphere Application Server for z/OS® Liberty. The feature is not available in WebSphere Application Server Liberty, or WebSphere Application Server Liberty Core. If you have a WebSphere Application Server Network Deployment Liberty installation, you can use its collectiveController-1.0 feature to work with collective members from WebSphere Application Server Liberty, or WebSphere Application Server Liberty Core installations.

Decide where to get your keys and certificates from for collective communication. You can create certificates and keys with the collective create command. Alternatively, create your own certificates and keys by some other method and use them for the collective.
  • You can store them in a keystore file.
  • For z/OS platformsYou can store them in a SAF key ring.

The Liberty server default keystore type is PKCS12. The JKS keystore type also works. Depending on when the collective keystores were created, they might be JKS keystores that have the .jks file extension or they might be the PKCS12 keystores with the .p12 file extension.

For example, the rootKey.jks keystore file might be the rootKeys.p12 keystore in the configuration and file system.

About this task

A Liberty collective is a set of Liberty servers that are configured as part of the same administrative and operational domain.

Configuration and state data about a Liberty collective is housed in an active operational repository.

Membership in a Liberty collective is optional. Liberty servers join a collective by registering with a collective controller to become members. Members share information about themselves through the operational repository of the controller.

The following rules apply:
  • A Liberty server can be a member of only one collective.
  • Different Liberty servers on the same host can be in different collectives.
  • Liberty servers on the same host that are members of a collective can coexist with Liberty servers that are not members of a collective.
  • Collectives make decisions using a quorum process. This requires that collective controllers can communicate among themselves so they can make decisions about the state of collective members. For communication among collective controllers, you need a reliable and low latency network connection for correct functioning of several network deployment features.

It is a strongly recommended best practice for collectives to be fully contained within a single data center.

Procedure

  1. Create and configure your controller.
    1. Create a server manually to act as the collective controller.

      In this procedure, the collective controller is refereed to as controller1.

      Place the controller in its own Liberty user directory, separate from the member servers for the controller. Specify the user directory on the WLP_USER_DIR environment variable. If necessary, customize the server.env file to contain the correct values for the JAVA_HOME environment variable and other settings.

      For z/OS platformsCreate the following items:
      • A started task cataloged procedure
      • Appropriate security controls
    2. Create the collective controller configuration.
      The collective controller configuration consists primarily of the administrative domain security configuration that is used for secure communication between controllers and members. To create a collective that uses self-signed certificate keystores, run the following command. The --keystorePassword parameter must be specified, but the value for it is ignored on the z/OS operating system.
      wlp/bin/collective create controller1 --keystorePassword=controllerKSPassword
      This command creates XML statements that are added to the server.xml file of the controller. By default, these XML statements are written to the console.
      You can alternatively redirect the controller1 controller into an .xml file by using the --createConfigfile parameter:
      wlp/bin/collective create controller1 --keystorePassword=controllerKSPassword --createConfigFile=$WLP_USER_DIR/servers/controller1/controller.xml
      Put this .xml file in the server directory of the controller.

      By default, this collective command generates public and private SSH keys for collective-wide use. For more information on collective-wide SSH, including how to use existing keys, see Collective-wide SSH key configuration.

      If you set up your own keystore and truststore files, you can add parameters to the collective create command to specify the keystores and truststores to use:
      wlp/bin/collective create controller1 --keystorePassword=controllerKSPassword --createConfigFile=$WLP_USER_DIR/servers/controller1/controller.xml --serverIdentityKeystore="keystore_filespec" --serverIdentityKeystoreAlias="certificate_alias" --collectiveTrustKeystore="truststore_filespec" [–certificateRdn="xxx"]
      For z/OS platformsIf you stored credentials in SAF key rings, these key rings and credentials must be set up in advance. All certificates must be signed and contain the OU=LZCollective value, or the value you specify with the --certificateRDN parameter. Specify the --setupSAF parameter to indicate that the following controller key ring and certificate values are used:
      • SAF key ring for SSH: LZController.SSHRing
      • SAF certificate label for SSH: LZController.SSH
      • SAF key ring name for SSL: LZController.Ring
      • SAF certificate label for SSL: LZController.CERT
      The following example for the collective create command uses these default values. It also includes the --certificateRdn parameter because it assumes that the collective Relative Distinguished Name (RDN) is something other than the OU=LZCollective value.
      wlp/bin/collective create controller1 --keystorePassword=password --createConfigFile=$WLP_USER_DIR/servers/controller1/controller.xml --setupSAF [–certificateRdn="xxx"]
      To specify your own key ring and certificate names on the z/OS operating system, omit the --setupSAF parameter and provide the complete set of key ring and certificate names. For example, run the following command:
      wlp/bin/collective create controller1 --keystorePassword=controllerKSPassword --createConfigFile=$WLP_USER_DIR/servers/controller1/controller.xml --safKeyRing="safkeyring:///LZ1.SSHRing" --safCertificateLabel="LZ1.SSH"     --serverIdentityKeystore="safkeyring:///LZ1.Ring" --serverIdentityKeystoreAlias="LZ1.CERT" --collectiveTrustKeystore="safkeyring:///LZ1.Ring" [–certificateRdn="xxx"] 
      For more information about these required parameters and other optional parameters, run the following command:
      collective help create
    3. Update the server.xml file of the collective controller.
      If the collective create command wrote the configuration output to the console, copy this output and paste it into the controller server.xml file. If you instead used the --createConfigFile parameter to write the configuration to a file, use an include statement to specify this file in the server.xml file. The following code is an example of an include statement:
      <include location="c:\wlp\usr\servers\controller1\controller.xml" />
      To provide an administrative user name and password for the collective, edit the following statement in the server.xml file or the included file:
      <quickStartSecurity userName="" userPassword="" />
      Change this statement by specifying an administrative user name and password like the following example:
      <quickStartSecurity userName="adminUser" userPassword="adminPassword" />
      Alternatively, add registry information to the server.xml file for the controller. Specify your own administrative ID and password through the registry.
    4. Start the collective controller server.
      Run a command like the following one:
      wlp/bin/server start controller1
    5. Verify that the collective controller server started correctly and is ready to receive members.
      1. Open an editor on the collective controller messages log, $WLP_USER_DIR/servers/controller1/logs/messages.log.
      2. Look for the following message:
        CWWKX9003I: CollectiveRegistration MBean is available.
  2. Add the server to the collective as a member.

    The controller and member servers can be on separate hosts but they must be in the same data center. In this example, the controller and member server are on the same host. The member server in the example is server1.

    1. If the server does not exist, create it.
      Run the following command:
      wlp/bin/server create server1
    2. Add the server to the collective as a member.

      The collective join command adds the server to the collective configuration, and also creates server-side XML to define the communication path to the collective controller.

      From the host server, run the collective join command to add the server.

      Before you run the command, ensure that you have these items:
      • A network connection to the collective controller.
      • An administrative user ID and password to perform MBean operations on the controller.
      • Values for the --host, --port, -user, and --password parameters. Obtain these parameters from the server.xml file of the collective controller.
      Run the following command:
      wlp/bin/collective join server1 --host=controllerHostname --port=9443 --user=adminUser --password=adminPassword --keystorePassword=memberKSPassword [--hostname=memberHostname]

      On the --keystorePassword parameter, set a value for the member keystore password, such as memberKSPassword. You can specify different --keystorePassword values for each server that you join to the collective.

      Set the optional --hostname parameter only if the system for the member server has multiple host names or does not have a host name configured. If you set the parameter, the value must match the defaultHostName variable that is defined in the server.xml file.

      To reduce the number of parameters needed, use the --controller parameter instead of the --user, --password, --host, and --port parameters:

      wlp/bin/collective join server1 --controller=user[:password]@host:httpsPort --keystorePassword=memberKSPassword [--hostname=memberHostname]
      By default, self-signed certificates are created. For more information on collective-wide SSH, including how to use existing keys, see Collective-wide SSH key configuration.
      If you already set up your own keystore and truststore files, you can add more parameters to the collective join command to specify the keystore and truststore files to use:
      wlp/bin/collective join server1 --controller=user[:password]@host:httpsPort --keystorePassword=controllerKSPassword --serverIdentityKeystore="keystore_filespec" --serverIdentityKeystoreAlias="certificate_alias" --collectiveTrustKeystore="truststore_filespec" [–certificateRdn="xxx"]
      For z/OS platformsWhen you run the collective join command to add the member server to the collective, you must use SAF key rings and certificates instead of file-based keystore files. If you set up your key ring and certificates in advance, you can specify the --setupSAF parameter to indicate that you are using the following default values:
      • SAF key ring name for SSL: LZMember.Ring
      • SAF certificate label for SSL: LZMember.CERT
      The certificate must be signed and must be the default certificate in the key ring. You must specify the relative distinguished name (RDN) value of OU=LZCollective in the certificate or alternatively specify an RDN value on the --certificateRDN parameter. Use the --certificateRDN parameter when the RDN value is not OU=LZCollective. You can use a SAF key ring and certificate that you already set up for the collective join command instead of keystore files.
      For z/OS platformsThe following collective join command example uses the --setupSAF and --certificateRDN parameters that are described previously:
      wlp/bin/collective join server1 --controller=user[:password]@host:httpsPort --keystorePassword=password --createConfigFile=$WLP_USER_DIR/servers/server1/member.xml --setupSAF [–certificateRdn="xxx"]
      To specify your own key ring and certificate names for the z/OS operating system, omit the --setupSAF parameter. If your collective RDN is something other than a value of OU=LZCollective, provide the key ring and certificate name, along with a value for the --certificateRdn parameter. The following example shows how to specify your own key ring and certificate:
      wlp/bin/collective join server1 --controller=user[:password]@host:httpsPort --keystorePassword=password --createConfigFile=$WLP_USER_DIR/servers/server1/member.xml --serverIdentityKeystore="safkeyring:///LZ2.Ring" --serverIdentityKeystoreAlias="LZ2.CERT" --collectiveTrustKeystore="safkeyring:///LZ2.Ring" [–certificateRdn="xxx"]

      For z/OS platformsIf you need to specify the cryptographic hardware provider (IBMJCECCA) or hybrid provider (IBMJCEHYBRID), include the --safKeystoreType and --safKeystoreProvider parameters for the collective join command.

      For example, the JVM java.security file might specify the following two cryptographic providers.
      • security.provider.1=com.ibm.crypto.hdwrCCA.provider.IBMJCECCA
      • security.provider.2=com.ibm.crypto.provider.IBMJCE
      In this case, include the --safKeystoreType and --safKeystoreProvider parameters to specify the IBMJCECCA cryptographic hardware provider, as shown in the following example.
      wlp/bin/collective join memberName --host=controllerHostname --port=9443 --user=adminUser --password=adminPassword  
      --keystorePassword=memberKSPassword --serverIdentityKeystore="safkeyringhw:///MEMBER.KEY" 
      --serverIdentityKeystoreAlias="MEMBER" --collectiveTrustKeystore="safkeyringhw:///MEMBER.KEY" 
      --safKeystoreType=JCECCARACFKS --safKeystoreProvider=IBMJCECCA
      In a hybrid scenario, the JVM java.security file might specify the following three cryptographic providers.
      • security.provider.1=com.ibm.crypto.ibmjcehybrid.provider.IBMJCEHYBRID
      • security.provider.2=com.ibm.crypto.hdwrCCA.provider.IBMJCECCA
      • security.provider.3=com.ibm.crypto.provider.IBMJCE
      In this case, the --safKeystoreType and --safKeystoreProvider parameters can specify the IBMJCEHYBRID cryptographic hybrid provider, as shown in the following example.
      wlp/bin/collective join memberName --host=controllerHostname  --port=443 --user=adminUser --password=adminPassword
      --keystorePassword=memberKSPassword --serverIdentityKeystore="safkeyringhybrid:///MEMBER.KEY" 
      --serverIdentityKeystoreAlias="MEMBER" --collectiveTrustKeystore="safkeyringhybrid:///MEMBER.KEY"
       --safKeystoreType=JCEHYBRIDRACFKS --safKeystoreProvider=IBMJCEHYBRID

      [22.0.0.10 and later]In version 22.0.0.10 and later, the collective join --help command provides information about the --safKeystoreType and --safKeystoreProvider parameters.

      You can specify deployment variables by using the optional genDeployVariable parameter. For more information, see Generating deployment variables when joining a member server to a collective.

      To write the output of this collective command to a file, instead of to a console screen, specify an optional --createConfigFile parameter. Then, include the outputted file in the collective configuration by adding an include statement to the member server.xml file:
      <include location=outputFilePath />

      By default, the join operation leaves remote procedure call (RPC) credentials undefined. You must specify values for the rpcUser parameter, the rpcUserPassword parameter. You must also specify values for the operating system login user ID and password for the host on which the member server exists. If the member host is registered with the collective controller and the member host is not enabled for SSH, specify an optional --useHostCredentials parameter. When you specify this parameter, the member inherits the RPC credentials from its host registration on the controller. Typically, Linux® hosts are enabled for SSH and Windows hosts are not enabled for SSH; thus, the --useHostCredentials parameter is useful for Windows member hosts. Specifying --useHostCredentials adds <hostAuthInfo useHostCredentials="true" /> to the member server.xml file. You then can run collective member server commands such as start or stop without specifying RPC credentials because the member inherits credentials from its host. See Overriding Liberty server host information for information about the hostAuthInfo element, the --useHostCredentials parameter, and connecting the collective controller to the server. See Setting up RXA for Liberty collective operations for information about enabling SSH on your host computer.

      For information about these required parameters and other optional parameters, run the collective help join command at a command line.

    3. If you are prompted to accept the certificate chain, enter y (yes).
    4. Update the member server.xml file.
      • If the collective join command wrote the configuration output to the console, copy this output and paste it into the server.xml file of the member server.
      • If you specified the --createConfigFile parameter to put the member configuration in a file, add an include statement to the server.xml file of the member server. The following example is of an include statement:
        <include location="c:\wlp\usr\servers\server1\member.xml" />
    5. If you did not specify the --useHostCredentials parameter on the collective join command and the member host is not enabled for SSH, set RPC credentials for the hostAuthInfo element. Set these credentials in the member server.xml file or in the outputted file.
      You can set RPC credentials for the member server in either of two ways:
      • Set the RPC user and password values on the hostAuthInfo element.

        Set the rpcUser option to an operating system login user ID for the host on which the member server exists.

        Set the rpcUserPassword option to the operating system login password for the user ID.

        For example, if you log in to the member computer with the test1 user ID and the test1pwd password, then change the hostAuthInfo element:
        <hostAuthInfo rpcUser="test1" rpcUserPassword="test1pwd" />
      • If the member host is registered with the collective controller, set the useHostCredentials option on the hostAuthInfo element to a value of true for the member server to inherit RPC credentials from its host:
        <hostAuthInfo useHostCredentials="true" />

      See Overriding Liberty server host information for information about the hostAuthInfo element and for an example that shows how to register a member host and run the join command with the --useHostCredentials parameter.

    6. For z/OS platforms For z/OS only, run the updateHost command to specify the member JAVA_HOME directory for the controller:
      wlp/bin/collective updateHost controllerHostname --host=controllerHostname
      --port=9443 --user=adminUser --password=adminPassword
      --hostWritePath=/dir1 --rpcUser=rpcUser
      --rpcUserPassword=rpcUserPassword --hostJavaHome=JAVA_HOME

      To reduce the number of parameters needed, use the --controller parameter instead of the --user, --password, --host, and --port parameters:

      wlp/bin/collective updateHost controllerHostname 
      --controller=adminUser:adminPassword@controllerHostname:9443
      --hostWritePath=/dir1 --rpcUser=rpcUser
      --rpcUserPassword=rpcUserPassword --hostJavaHome=JAVA_HOME

      The --hostWritePath parameter specifies the directories to which the collective controller can write. Paths that are specified by the parameter are also readable.

      The --hostWritePath parameter specifies the directories to which the collective controller can write.

      The --hostJavaHome parameter specifies the absolute path to where the JVM is on the system and where the member server runs, for example /usr/lpp/java/java_1.7_64.

      For more information about specifying the JAVA_HOME environment variable, see Setting the JAVA_HOME variable for Liberty collective members and controllers.

    7. Start the member server.
      wlp/bin/server start member1
    8. Verify that the member server started correctly and is publishing information to the controller.
      1. Open an editor on the member messages log, $WLP_USER_DIR/servers/member1/logs/messages.log.
      2. Look for the following messages in any order:
        CWWKX8112I: The server's host information was successfully published to the collective repository.
        CWWKX8114I: The server's paths were successfully published to the collective repository.
        CWWKX8116I: The server STARTED state was successfully published to the collective repository.
    9. Optional: Use the testConnection command to validate connectivity.
      The command validates RXA connectivity between the controller and the host where the member exists. It also validates JMX secure connectivity between the collective controller and the collective member.
      wlp/bin/collective testConnection hostName,usrDirPath,serverName  --host=controllerHost
      --port=controllerHTTPSPort --user=controllerAdmin 
      --password=controllerAdminPassword--autoAcceptCertificates
      Where the hostName,usrDirPath,serverName tuple describes the member server just joined: The hostName variable is the name of the host where the collective member exists. The userDirPath variable is the user directory of the collective member. serverName is the name of the collective member.
      Alternatively, use the simplified --controller parameter to provide the controller-specific information.
      wlp/bin/collective testConnection hostName,usrDirPath,serverName 
      --controller=user[:password]@host:HttpsPort 
      --autoAcceptCertificates
  3. Add more servers to the collective as members.

    Repeat the previous step to add more servers to the collective.