Installing IBM Spectrum LSF Suite for Enterprise on a shared file system

Use the shared-directory installation to install components of IBM Spectrum LSF Suite for Enterprise on a shared file system. The shared directory installation enables you to update the LSF GUI hosts independently of the LSF management, server, and client hosts. You can also patch the contents of the shared directory from the deployment host.

1. With the NFS or IBM Spectrum Scale file system mounted on all of the hosts you want to install, log in as root to the host you want to use as the deployment host and test access from the deployment host as root.
   For example, the following commands make sure that root can write to the /gpfs/lsf_suite/ directory.

   cd /gpfs/lsf_suite
   touch testfile
   ls -l testfile
   -rw-r--r--. 1 root root 0 Nov 1 2017 testfile

   The testfile should be owned by root.

2. Change to the /opt/ibm/lsf_installer/playbook directory to customize the installation.

3. Edit the lsf-inventory file to set up host roles for your cluster.

   LSF_Masters, GUI_Hosts, LSF_WebService, and DB_Host

   Specify the names of primary and secondary LSF management hosts in the LSF_Masters option. Specify the name of the secondary LSF management candidate host in the GUI_Hosts, LSF_WebService, and DB_Host options.

   For example,

   [LSF_Masters]
   hosta
   hostb
   ...
   [GUI_Hosts]
   hostb
   ...
   [LSF_WebService]
   hostb
   ...
   [DB_Host]
   hostb

   LSF_Servers and LSF_Clients

   List your LSF server hosts in the LSF_Servers option, one host or host name expression per line. The expression in the following example configures four server hosts (hosta1, hosta2, hosta3, and hosta4):

   [LSF_Servers]
   hosta[1:4]

   LSF_Clients

   List some LSF client hosts in the LSF_Clients option for users to submit jobs from, one host or host name expression per line. These hosts do not run workloads. The expression in the following example configures three hosts (hostb1, hostb2, hostb3):

   [LSF_Clients]
   hostb[1:3]

4. Test that the shared file system is mounted on all the hosts.
   For example, to test that the file /gpfs/lsf_suite/testfile is accessible from all the hosts in the cluster, run the following command from the /opt/ibm/lsf_installer/playbook directory on the deployment host:

   ansible all -i lsf-inventory -m command -a "ls /gpfs/lsf_suite/testfile"

   This command runs on all the hosts listed in the lsf-inventory file, and uses the command module to run the ls command as an argument.
5. Edit the lsf-config.yml file to set the cluster name and other properties.
   Remember: Make sure you maintain indentation in the lsf-config.yml file.
   1. Set the cluster name.
      For example,

      LSF
        # Set my_cluster_name to the name of the cluster.
        my_cluster_name: cluster1
      ...

   2. Use the NFS_install_dir option to specify the shared directory that is mounted on all the hosts. The directory for the HA_shared_dir option is implied when the NFS_install_dir is set.

      Use the SHARED_CONFIGURATION_DIR and LWS_CLUSTER_NAME options to specify the shared directory that is mounted on LSF Web Services hosts.

      For example:

      LSF
      ...
        HA_shared_dir: none
      ...
        NFS_install_dir: /gpfs/lsf_suite
      ...
      LWS 
      ... 
        LWS_CLUSTER_NAME: "myLwsCluster" 
        SHARED_CONFIGURATION_DIR: "/gpfs/lsf_suite" 
      ...

   3. In version 10.2.0.9 and later, to install Kibana for use with Elasticsearch, set the parameter Install_Kibana: True. Kibana will be installed on the first GUI_Host defined in the lsf-inventory file.

6. Run some pre-installation tests and correct any errors.
   1. Check the configuration file.

      ansible-playbook -i lsf-inventory lsf-config-test.yml

   2. Run the pre-deployment test.

      ansible-playbook -i lsf-inventory lsf-predeploy-test.yml

      This test runs on each host to check network connectivity and host name resolution, minimum disk space, and available memory. The test takes a few minutes to run.

   Correct any errors from the tests and run them again before you run the installation.

7. If you want LSF_Masters to install LSF RPMs locally, move into group_vars/all and change the variable LSF_MASTERS_ON_LOCAL from the default value of false to true.
   Otherwise, the shared directory will be used by default.

8. Perform the shared directory installation. Navigate to the /opt/ibm/lsf_installer/playbook directory on the deployment host.
   1. Build the shared directory contents.
      Run:

      ansible-playbook -i lsf-inventory lsf-nfs-setup.yml

   2. Deploy the cluster.
      Run:

      ansible-playbook -i lsf-inventory lsf-deploy.yml

      Note: Using the --limit option allows new LSF_Servers to be installed and existing LSF_Servers can be reinstalled.

      ansible-playbook -i lsf-inventory --limit {Some Host or hosts} lsf-deploy.yml

      This playbook deploys the software on the LSF management and GUI hosts, sets up the necessary symbolic links to the shared directory, and installs the local component LSF server hosts that use the shared directory for the LSF binary files.

      When the installation is finished, it gives you a URL for the IBM Spectrum LSF Suite for Enterprise portal and a URL for the LSF Web Services API endpoint.

      For example,

      http://hostb.company.com:8080
      https://hostb.company.com:8448

      Where hostb is the name of the GUI host that you configured in the lsf-inventory file.

9. Run some commands to verify the installation.
   1. Log out of the deployment host, and log in to a host in the cluster.
   2. Run the lsid command to see your cluster name and management host name.
      For example:

      lsid
      IBM Spectrum LSF 10.1.0.3, Sep 13 2017
      Suite Edition:  IBM Spectrum LSF Suite for
      Enterprise 10.2.0
      Copyright International Business Machines Corp, 1992-2017.
      US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
      
      My cluster name is cluster1
      My master name is hosta

   3. Run the lshosts command to see both LSF management hosts (they are members of the management group indicated by the mg resource) The four LSF server hosts and the three client hosts are also listed in this example.

      lshosts
      HOST_NAME      type    model    cpuf ncpus   maxmem  maxswp  server RESOURCES 
      hosta        X86_64  Intel_EM   60.0    16   255.8G    3.9G    Yes  (mg)
      hostb        X86_64  Intel_EM   60.0    16   255.8G    3.9G    Yes  (mg)
      hosta1       X86_64  Opteron8   60.0     1     7.9G      1G    Yes  ()
      hosta2       X86_64  Opteron8   60.0     1     7.9G      1G    Yes  ()
      hosta3       X86_64  Opteron8   60.0     1     7.9G      1G    Yes  ()
      hosta4       X86_64  Opteron8   60.0     1     7.9G      1G    Yes  ()
      hostb1       X86_64    PC6000  116.1     1    31.9G    3.9G     No  ()
      hostb2       X86_64    PC6000  116.1     1    31.9G    3.9G     No  ()
      hostb3       X86_64  Opteron8   60.0     1     7.9G      1G     No  ()

   4. Run the bhosts command to check that the status of each host is ok, and the cluster is ready to accept work.

      bhosts
      HOST_NAME          STATUS       JL/U    MAX  NJOBS    RUN  SSUSP  USUSP    RSV
      hosta1             ok              -      1     0       0      0      0      0
      hosta2             ok              -      1     0       0      0      0      0
      hosta3             ok              -      1     0       0      0      0      0
      hosta4             ok              -      1     0       0      0      0      0
      hosta              ok              -     16     0       0      0      0      0
      hostb              ok              -     16     0       0      0      0      0

   5. Log in to one of the server hosts to check that it's using the shared directory.
      For example,

      # ssh hosta1
      # cd /opt/ibm/lsf_suite
      # ls
      ext  lsf
      # ls -al 
      total 0
      drwxr-xr-x. 3 lsfadmin root 28 Nov   2 13:26 .
      drwxr-xr-x. 6 root     root 92 Nov   2 13:26 ..
      drwxr-xr-x. 2 lsfadmin root  6 Nov   2 13:26 ext
      lrwxrwxrwx. 1 root     root 39 Nov   2 13:26 lsf -> /gpfs/lsf_suite/lsf

      See that the lsf directory actually comes from the shared directory /gpfs/lsf_suite/lsf.

10. Log in to the GUI as the lsfadmin user.
    Note: If the lsfadmin user was created by the installation and did not exist in your system, you might need to create a password for lsfadmin with the passwd command.
    1. Open your browser and enter the GUI portal URL from the installation.
    2. Log in with the lsfadmin user or any other user that you know exists.
    On the Resources > Dashboard page, you can see the six hosts that you deployed to.