Installing the fabric management server

The installation of the fabric management server is performed by the customer.

1. F1 (M1): Physically place the fabric management server on the data center floor.
2. F2 (M2): Install and configure the operating system on the fabric management server.
3. F3 (M2): If you are connecting the fabric management server or servers to a public Ethernet network (not the service, nor the cluster VLAN), do so at this time.
4. F4 (M2): Install and cable the host channel adapters (HCAs) in the fabric management servers. The HCAs must be installed before proceeding to the next step. Cabling of the HCAs to the fabric can wait, but do not start the Fabric Manager software until the fabric management server HCAs have been cabled to the fabric.
5. F5 (M2): To install the QLogic InfiniServ host stack and Fast Fabric Toolset, use the InfiniServ Fabric Access Software Users Guide. The following items are the key steps to the installation:
   1. Untar the InfiniServ tar file.
   2. Run the INSTALL script by using the appropriate flags as described in the QLogic documentation.
      Note: Do not enable IPoIB on the fabric management server, or do not install the IPoIB capability. Otherwise, the multicast groups might be negatively affected by IPoIB on the fabric management server by setting up groups that are not valid for the compute servers and I/O servers on the fabric.
   3. Restart the InfiniServ stack.
6. F5 (M2): Set up the Fast Fabric Toolset by completing the following tasks:
   1. Configure the Fast Fabric Toolset according to the instructions in the Fast Fabric Toolset Users Guide. When configuring the Fast Fabric Toolset, consider the following application of the Fast Fabric Toolset within high-performance computing (HPC) clusters:
      • The master node, referred to in the Fast Fabric Toolset Users Guide, is considered to be the Fast Fabric Toolset host in IBM® HPC clusters.
      • You do not have to set up rsh and ssh access to the servers from the Fast Fabric Toolset host.
      • You do not use the message-passing-interface (MPI) performance tests, because they are not compiled for the IBM host stack.
      • HPL is not applicable.
      • You generally only use parameters that list the switch chassis.
      • You never issue commands to hosts.
   2. Update the following Fast Fabric configuration files. These files list the switch and Fabric Manager servers that make up the fabric. This function provides the ability to report and process commands across the fabric concurrently.
      • The /etc/sysconfig/iba/chassis file must have the list of all the switch chassis in the fabric. Each chassis is listed on a separate line of the file. You can use either the IP address or the resolvable host name for the chassis address.
      • If you planned for groups of switches, create a file for each group.
      • The /etc/sysconfig/iba/hosts file contains a list of all the fabric management servers.
      • If you planned for groups of fabric management servers, create a file for each group.
      • Set up the /etc/sysconfig/fastfabric.conf file with the appropriate FF_ALL_ANALYSIS and FF_FABRIC_HEALTH environmental variable values. These values include the fabric, chassis, and subnet manager analysis. The subnet manager analysis depends on the type of subnet manager you are using. There is a commented entry for FF_ALL_ANALYSIS that includes all possible analysis tools. You only need the hostsm or esm (embedded subnet manager) entry.
        • If you have a host-based subnet manager, edit the entry to look similar to the following example:

          export FF_ALL_ANALYSIS="${FF_ALL_ANALYSIS:-fabric chassis hostsm}"

        • If you have an embedded subnet manager, edit the entry to look similar to the following example:

          export FF_ALL_ANALYSIS="${FF_ALL_ANALYSIS:-fabric chassis esm}"

        • Using a pattern that matches the names of your switches, set up the FF_FABRIC_HEALTH variable. The following example shows that the default names were left in place. The default names begin with SilverStorm. It also removes the errors that exceed threshold:export:

          FF_FABRIC_HEALTH="${FF_FABRIC_HEALTH:- -s -o errors -o slowlinks –F
           nodepat:SilverStorm*}""

      • Also, if applicable, ensure that the /etc/sysconfig/iba/esm_chassis file has the list of switch IP addresses for switches that are running the embedded-subnet manager.
   3. Ensure that the /etc/sysconfig/iba/ports file has a list of ports on the fabric management server. The format is a single line that lists the HCA ports on the fabric management server that are attached to the subnets. There should be one port for each subnet. The format for identifying a port is [hca]:[port]. If four ports are connected, the ports file should have a single line: 1:1 1:2 2:1 2:2
   4. Ensure that the tcl and Expect code is installed on the fabric management server. The code should be at least at the following levels. You can check them by using the rpm -qa | grep expect and rpm -qa | grep tcl commands.
      • expect-5.43.0-16.2
      • tcl-8.4.12-16.2
   5. If this is the primary data collection point for fabric diagnosis, ensure that this is noted. One method would be to add this information to the /etc/motd file.
7. F6 (M2): If you are using a host-based Fabric Manager, install it by using the Fabric Manager Users Guide. The following utilities are helpful rpms to install:
   1. iview_agent-4_2_0_0_xx.rpm (4_2_0_0_xx refers to the level of the agent code)
   2. iview_fm-4_2_0_0_xx.rpm (4_2_0_0_xx refers to the level of the Fabric Manager code)
   3. sm_query (sm_query is a utility to obtain information from the subnet manager)
      Note: Do not start the Fabric Managers until the switch fabric is installed and cabled completely. Otherwise, you cause unnecessary log activity from the Fabric Manager, which could cause confusion when you try to verify fabric operation.
   4. Run the iview_fm stop command to ensure that the subnet manager is stopped until it is required.

      Verify that the subnet manager is stopped by running the ps –ef|grep iview command.

8. F6 (M2): Configure the host-based Fabric Manager by updating the iview_fm.config file by using the Fabric Manager Users Guide.

   There is a separate instance of the various fabric management components running to manage each subnet. In the iview_fm.config file, configure each instance of each component.

   1. At the beginning of the parameter settings in the iview_fm.config file, you must configure each component of each instance of the Fabric Manager to start when you start the Fabric Manager. Below, each attribute begins with SM_X_<attribute>, where X is the subnet manager instance on the fabric management server. To see an example of how these parameters would look for the iview_fm.config file that is used for managing four subnets, see Example: Setting up of host-based fabric manager.

      BM_X_start=yes
      FE_X_start=yes
      PM_X_start=yes
      SM_X_start=yes

      Note: Any instances that are not in use should be set to start=no, such as, SM_2_start=no.
   2. Point to the correct HCA for each Fabric Manager instance:

      SM_X_device=<hca>
      PM_X_device=<hca>
      BM_X_device=<hca>
      FE_X_device=<hca>

   3. Point to the correct port on the HCA for each Fabric Manager instance:

      SM_X_port=<hca port>
      PM_X_port=<hca port>
      BM_X_port=<hca port>
      FE_X_port=<hca port>

   4. Set the priority for each Fabric Manager instance, such as SM_X_priority=<priority>:

      SM_X_priority=<priority>
      PM_X_priority=< priority>
      BM_X_priority=< priority>
      FE_X_priority=<priority>

   5. For LMC=2, use SM_X_lmc   2.
   6. For the maximum transfer unit (MTU), use the value that you calculated in Planning for maximum transfer units (MTUs). SM_X_def_mc_mtu=0x5 #0x4=2 KB; 0x5=4 KB
   7. For the MTU rate, use the value that you calculated in Planning for maximum transfer units (MTUs). SM_X_def_mc_rate=0x6 # 0x3 for SDR; 0x6 for DDR.
   8. For the global identifier (GID) prefix, use SM_X_gidprefix=<GID prefix value>
   9. For node appearance or disappearance threshold of 10, use SM_x_node_appearance_msg_thresh=10
9. Cable the fabric management server to the InfiniBand fabric.
   Note: The switches must have been installed as instructed in Installing and configuring vendor InfiniBand switches.
10. F7 (M2): Use a static IP address for the cluster VLAN for the fabric management servers. Assign and configure this address. This is required for remote logging to and remote command processing from the Cluster Systems Management/Management Server (CSM/MS).
11. F8 (M3): Cable the fabric management server to the cluster VLAN. It must be on the same VLAN as the switches.
12. Before proceeding, ensure that the fabric management server is cabled to the InfiniBand fabric and that the switches are powered on.
13. F9 (M4): Perform final fabric management server configuration and verification:
    1. If you are using a host-based subnet manager, make sure that the embedded subnet managers are not running (unless you plan to use both):
       • i. Run the cmdall -C 'smControl status' command.
       • ii. If one or more embedded subnet managers are running, stop them by using the cmdall -C 'smControl stop' command.
       • iii. Ensure that the embedded subnet manager does not start on reboot by using the cmdall –C ‘smConfig startAtBoot no' command.
    2. If you are using a host-based subnet manager, enable and start the Fabric Manager by using instructions from the Fabric Manager Users Guide. The key commands are as follows:
       • i. /etc/init.d/iview_fm enable
       • ii. /etc/init.d/iview_fm start
    3. Verify correct security configuration for switches by ensuring that each switch has the required user and password enabled.
       • i. Run the cmdall -C 'loginMode' command.
       • ii. If the return value is not set to 0, enable it.
       • iii. Run the cmdall -C 'loginMode 0' command.
14. Set up passwordless SSH communication between the fabric management server and the switches and other fabric management servers. If this is not wanted, you need to set up password information for the Fast Fabric Toolset; skip to step 15.
    1. Generate the key on the fabric management server. Depending on local security requirements, you typically do this for the root on the fabric management server. Typically, you use the /usr/bin/ssh-keygen -t rsa command.
    2. Set up the secure fabric management server to switch communication by using the following instructions:
       1. Exchange the key by using the cmdall –C ‘sshKey add “Fabric/MS key”' command, where Fabric/MS key is the key.
          Note: The key is in the ~/.ssh/id_rsa.pub file. Use the entire contents of the file as the Fabric/MS key. Remember to type quotation marks around the key, and single quotation marks around the entire sshKey add command.
       2. Ensure that the following information is in the /etc/fastfabric.conf export file: FF_LOGIN_METHOD="${FF_LOGIN_METHOD:-ssh}"
    3. Set up secure communication between the fabric management servers by using one of the following methods:
       • Use the setup_ssh command in the Fast Fabric Toolset.
       • Use the Fast Fabric Toolset iba_config menu. Select Fast Fabric > Host setup > Setup Password-less ssh/scp.
       • Use typical key exchange methods between Linux servers.
15. If you chose not to set up passwordless SSH from the fabric management server to switches and to other fabric management servers, you must update the /etc/sysconfig/fastfabric.conf file with the correct password for admin. The following procedure uses the password xyz. For detailed instructions, see the Fast Fabric Users Guide.
    1. Edit the /etc/sysconfig/fastfabric.conf file and ensure that the following lines are included in the file and are not commented out. FF_LOGIN_METHOD and FF_PASSWORD are used for fabric management server access. FF_CHASSIS_LOGIN_METHOD and FF_CHASSIS_ADMIN_PASSWORD are used for switch chassis access.

       export FF_LOGIN_METHOD="${FF_LOGIN_METHOD:-telnet}"
       
       export FF_PASSWORD="${FF_PASSWORD:-}"
       
       export FF_CHASSIS_LOGIN_METHOD="${FF_CHASSIS_LOGIN_METHOD:-telnet}"
       
       export FF_CHASSIS_ADMIN_PASSWORD="${FF_CHASSIS_ADMIN_PASSWORD:- xyz}

    2. Run the chmod 600 /etc/sysconfig/fastfabric.conf command. This command ensures that only the root can use the Fast Fabric tools and recognize the updated password.
16. It is a good practice to enter the configuration information for the server in its /etc/motd file. Use the information from the QLogic Fabric Management work sheets.
17. If you want to monitor the fabric by running the health check on a regular basis, review Setting up periodic fabric health checking. Do not set up regular health checks until the fabric has been installed and verified.