Installing IBM Spectrum LSF Suite for Workgroups on a large cluster

Install a large IBM Spectrum LSF Suite for Workgroups cluster.

About this task

Install IBM Spectrum LSF Suite for Workgroups with a primary LSF management host and a secondary failover management candidate host (two hosts set in the LSF_Masters option). Three GUI server hosts (GUI_Hosts), LSF Web Services hosts (which starts in Fix Pack 15: LSF_WebService), and database (DB_Host) are installed. The IBM Spectrum LSF Suite for Workgroups installation repository (Deployer) is on another host. A high-availability shared directory (HA_shared_dir and SHARE_CONFIGURATION_DIR) is configured on a separate host. Several compute hosts (LSF_Servers) are also installed.

This configuration separates some of the host roles to provide failover of the LSF management host. Jobs continue to run as long as one LSF management host is available. This cluster also deploys GUI services and LSF Web Services) on three GUI servers. The database is hosted on one of the GUI hosts. This task shows you how to verify that the high-availability shared directory (HA_shared_dir and SHARE_CONFIGURATION_DIR) is available, how to set up host roles on each server, and how to verify your installation.

1. Log in as root to the host you want to use as the deployment host, and change to the /opt/ibm/lsf_installer/playbook directory to customize the installation.
   1. Edit the lsf-inventory file to set up host roles for your cluster.

      LSF_Masters, GUI_Hosts, LSF_WebService, and DB_Host

      Specify the names of primary and secondary LSF management hosts in the LSF_Masters option. Specify the name of the secondary LSF management candidate host in the GUI_Hosts, LSF_WebService, and DB_Host options. For example,

      [LSF_Masters]
      hosta
      hostb
      ...
      [GUI_Hosts]
      hostc
      hostd
      hoste
      ...
      [LSF_WebService]
      hostc
      hostd
      hoste
      ...
      [DB_Host]
      hostc

      LSF_Servers and LSF_Clients

      List your LSF server hosts in the LSF_Servers option, one host or host name expression per line. The expression in the following example configures 48 server hosts named host86b1 to host88e4:

      [LSF_Servers]
      host8[6:8][b:e][1-4]

      LSF_Clients

      List some LSF client hosts for users to submit jobs from in the LSF_Clients option, one host or host name expression per line. Remember that these hosts do not run work. The expression in the following example configures a three client hosts (hostb1, hostb2, hostb3):

      [LSF_Clients]
      hostb[1:3]

   2. Edit the lsf-config.yml file to set the cluster name and other properties.
      For example,

      LSF
        # Set my_cluster_name to the name of the cluster.
        my_cluster_name: cluster1
      ...

      Remember: Make sure to maintain the indentation in the lsf-config.yml file.

      Note: In version 10.2.0.9 and later, to install Kibana for use with Elasticsearch, set the parameter Install_Kibana: True. Kibana will be installed on the first GUI_Host defined in the lsf-inventory file.
      Because the LSF GUI host runs on a separate host from the primary LSF management host, you need to specify a shared directory (HA_shared_dir). The directory must be mounted on both management hosts. For example,

      LSF
      ...
      #  HA_shared_dir: none
        HA_shared_dir: /shared/lsfsuite
      ...

      Leave system monitoring enabled (Enable_Monitoring: True). One of the GUI servers is also configured with the database host role, so do not set the JDBC_string option.

      Because LSF Web Services runs in high availability mode, you must specify a directory (SHARED_CONFIGURATION_DIR) and a cluster name LWS_CLUSTER_NAME. For example:

      LWS
      ...
        LWS_CLUSTER_NAME: "myLwsCluster"
        SHARED_CONFIGURATION_DIR: "/shared/lsfsuite"
      ...

2. After you set up the lsf-inventory and lsf-config.yml files, run two pre-installation tests.
   1. Check the configuration file, and correct any errors.

      ansible-playbook -i lsf-inventory lsf-config-test.yml

   2. Run the pre-deployment test.

      ansible-playbook -i lsf-inventory lsf-predeploy-test.yml

      This test runs on each host to check network connectivity and host name resolution, minimum disk space, and available memory. The test takes a few minutes to run.

   Correct any errors from the tests and run them again before you run the installation.
3. Run the installation with the lsf-deploy.yml playbook to deploy your cluster.

   ansible-playbook -i lsf-inventory lsf-deploy.yml

   Note: By using the --limit option, new LSF_Servers can be installed and existing LSF_Servers can be reinstalled.

   ansible-playbook -i lsf-inventory --limit {Some Host or hosts} lsf-deploy.yml

   When the installation is finished, it gives you a URL for the IBM Spectrum LSF Suite for Workgroups portal and a URL for the LSF Web Services API endpoint.

   For example,

   http://hostc.company.com:8080
   https://hostc.company.com:8448

   Where hostc is the name of one of the GUI hosts that you configured in the lsf-inventory file. You can use any of the configured GUI servers in this URL.
4. Run some commands to verify the installation.
   1. Log out of the deployment host, and log in to a host in the cluster.
   2. Run the lsid to see your cluster name and management host name.

      lsid
      IBM Spectrum LSF 10.1.0.3, Sep 13 2017
      Suite Edition:  IBM Spectrum LSF Suite for
      Workgroups 10.2.0
      Copyright International Business Machines Corp, 1992-2017.
      US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
      
      My cluster name is cluster1
      My master name is hosta

   3. Run the lshosts command to see both LSF management hosts (they are members of the management group indicated by the mg resource). The 48 LSF server hosts and the three client hosts are also listed.

      lshosts
      HOST_NAME      type    model    cpuf ncpus   maxmem  maxswp  server RESOURCES 
      hosta        X86_64  Intel_EM   60.0    20    63.9G    31.9G    Yes  (mg)
      hostb        X86_64  Intel_EM   60.0    20    63.9G    31.9G    Yes  (mg)
      host86b1     X86_64  Opteron8   86.0     1     7.9G       1G    Yes  ()
      host86b1     X86_64  Opteron8   86.0     1     7.9G       1G    Yes  ()
      ...
      host88e3     X86_64  Opteron8   86.0     1     7.9G       1G    Yes  ()
      host88e4     X86_64  Opteron8   86.0     1     7.9G       1G    Yes  ()
      hostc        X86_64  Intel_EM   60.0    16   255.8G     3.9G    Yes  (mg)
      hostd        X86_64  Intel_EM   60.0    16   255.8G     3.9G    Yes  (mg)
      hoste        X86_64  Intel_EM   60.0    16   255.8G     3.9G    Yes  (mg)
      hostb1       X86_64    PC6000  116.1     1    31.9G     3.9G     No  ()
      hostb2       X86_64    PC6000  116.1     1    31.9G     3.9G     No  ()
      hostb3       X86_64  Opteron8   60.0     1     7.9G       1G     No  ()

   4. Run the bhosts command to check that the status of each host is ok, and the cluster is ready to accept work.

      bhosts
      HOST_NAME          STATUS       JL/U    MAX  NJOBS    RUN  SSUSP  USUSP    RSV
      host86b1           ok              -     1      0       0      0      0      0
      host86b1           ok              -     1      0       0      0      0      0
      ...
      host88e3           ok              -     1      0       0      0      0      0
      host88e4           ok              -     1      0       0      0      0      0
      hosta              ok              -     20     0       0      0      0      0
      hostb              ok              -     20     0       0      0      0      0
      hostc              ok              -     16     0       0      0      0      0
      hostd              ok              -     16     0       0      0      0      0
      hoste              ok              -     16     0       0      0      0      0

5. Log in to the GUI as the lsfadmin user.

   If the lsfadmin user was created by the installation and did not exist in your system, you might need to create a password for lsfadmin with the passwd command.

   1. Open your browser and enter the GUI portal URL from the installation.
   2. Log in with the lsfadmin user or any other user that you know exists.
   On the Resources > Dashboard page, you can see the 53 hosts that you deployed to.