Installing IBM Spectrum Fusion HCI

IBM Spectrum Fusion HCI comes with the bootstrapping software that is installed from the factory. The IBM support representative completes the initial verification and physically connects the system to network and power. Then, they conduct the network setup of the appliance, which connects the appliance to the data center network. This procedure configures all the default nodes (Three controllers and three compute nodes). If you ordered more nodes, then they get installed as well. Use this information to install IBM Spectrum Fusion HCI. Finally, Storage and Backup software get installed on these nodes.

Before you begin

  • A connection to the internet is needed to install the software that operates the IBM Spectrum Fusion HCI appliance. If you plan to do an offline installation of IBM Spectrum Fusion, mirror external-operators and OpenShift® Container Platform image repositories to your registry. For the actual steps, see Enterprise registry for IBM Spectrum Fusion HCI installation.
  • Important: IBM Spectrum Fusion HCI requires Red Hat OpenShift Container Platform. If you purchased OpenShift subscriptions from IBM, see Activating Red Hat OpenShift Container Platform subscriptions purchased from IBM. For the supported version of Red Hat OpenShift Container Platform, see https://www.ibm.com/support/pages/node/6829067.
  • Important: Before the installation of IBM Spectrum Fusion HCI, enable IBM Spectrum Fusion HCI Software to be downloaded. For steps to enable, see Activating IBM Spectrum Fusion HCI Software to be downloaded.
  • After the rack arrives at your destination, coordinate with your IBM Technical Sales team to remove the pallet and follow the printed manual to unbox the rack and locate it in your data center.
  • Ensure that IBM Spectrum Fusion HCI is installed in a restricted access location, such that the area is accessible only to skilled and instructed persons with proper authorization.
  • Follow the OpenShift Container Platform guidelines to create a certificate. Consider the following points for the chain of certificates:
    • Wildcard certificate must be the first certificate in the file.
    • Any intermediate certificates must follow the wildcard certificate.
    • End of the file must have root CA certificate.
  • Go through the Site-readiness topics and confirm whether your premise satisfies the requirements.
  • The following are the high-level steps to set up and configure the hardware appliance:
    1. You must unpack the appliance and locate it in your data center. For planning and prerequisites, see Planning and prerequisites.
    2. The Service Support Representative (SSR) assists you in network cabling and power connections by using the details that are provided by your network team.
    3. The SSR assists you with the Network setup stage of the installation. The network setup includes the configuration of your network, such as high-speed switch and IP configuration. For more information, see Network planning.
  • If you plan to use a proxy server for internet access, then do the following steps:
  • The Container Network Interface (CNI) network (daemon network) is created for Scale core pods. By default, IP addresses are assigned for scale daemon network. You can override the default IP address before you begin the Final installation. For the procedure, see Configuring Scale daemon network IP parameters.
    Note: You can override the default IP addresses only before you start the Final Installation wizard.
  • If you plan to install site 2 in a Metro sync DR setup, ensure that the following prerequisites are met:
    • IBM Spectrum Scale on the site 1 is healthy and all IBM Spectrum Scale core pods are up and running.
    • Ensure that the disk count is same on site 1 and site 2.
  • If you installed IBM Spectrum Fusion HCI version 2.3 by using offline or online installation mode, then ensure that you do not change the mode during the upgrade to 2.4 version. To change the installation mode, reinstall IBM Spectrum Fusion HCI 2.4.

Step 1 - start the installer

Your IBM Systems Support Representative (IBM SSR) shares an IP address or URL after they connect the appliance to your network. Open the following URL:
http://<host IP address>:3000/isfsetup

Replace the URL that you received in <host IP address>

The URL takes you to the IBM Spectrum Fusion HCI installer welcome page.

Step 2 - accept license agreement

On the License Agreement page, review the terms and conditions of the license. If you want to keep a copy, download the license. After you go through and accept the license agreements, click I accept the license agreement, and click Continue to proceed with the installation wizard.

Step 3 - go through installation summary

The Getting started page explains the installation steps and the estimated time. The estimated time for the Spectrum Fusion software installation to complete is 120 minutes. The Cluster expansion takes another 60 minutes to complete.

Step 4 - precheck the network

The Network precheck page runs an automatic network check against all of the nodes in the appliance. It checks whether each node has an assigned IP address and hostname. If all nodes are marked with a green Connected status, you can proceed to the next step of the installer.

Any node that does not pass the network check is marked with a red Disconnected status. It means that either DHCP or DNS configuration for the node is not available. For more information about the prerequisite, see Setting up the DNS and DHCP for IBM Spectrum Fusion appliance.

Work with your network team to ensure that DNS and DHCP are configured for all nodes in the appliance. After the DHCP or DNS changes, click Restart precheck to initiate a new network check. If you want changes to your node settings, contact your network team.

Click Next to go to the Image registry settings.

Step 5 - set up image registry

IBM Spectrum Fusion installs Red Hat OpenShift Container Platform and IBM Spectrum Fusion software by using the images that are hosted in the Red Hat and IBM entitled registries. If you want to use your private image registry, you can install both Red Hat OpenShift and IBM Spectrum Fusion HCI software from images, which are maintained in a container registry that you manage. For steps to plan and use your own enterprise registry, see Enterprise registry for IBM Spectrum Fusion HCI installation.

As part of this step, you can configure a proxy to connect to the repository. Using a proxy is most common for connecting to the public image registries as it requires connecting from your private network to public websites.

Choose whether to use the Public image registry or Private image registry option.

Public image registry
To use the public image registry, you need a pull secret and an entitlement key.
  • Enter the Pull secret. It is an authorization token that stores Docker credentials that you can use to access a registry. Your cluster needs this secret to access and pull OpenShift images from the quay.io container registry. If you do not have a pull secret, click Get Pull secret. It takes you to https://cloud.redhat.com/openshift/install/pull-secret.
  • Enter the Entitlement key. It is a product code that is used to pull images from IBM Entitlement Registry. Your cluster needs this key to gain access to IBM Spectrum Fusion images in the IBM Entitlement Registry. If you do not have a key, click Get Entitlement key. It takes you to IBM Container Library. For steps to obtain the key, see the Activating IBM Spectrum Fusion HCI Software to be downloaded.
Private image registry

If you select the Private image registry, you must first mirror the Red Hat and IBM Spectrum Fusion images to your private registry. For more information about mirroring, see Mirroring your images to the enterprise registry.

You can choose to host the Red Hat and IBM Spectrum Fusion images in separate repositories, or use the same repository.
  • Single repository
    Enter the following details for the enterprise registry.
    • Enter the URL of the private registry in the Repository path.
      For example, 
      https://<enterprise registry>:<custom port>/<mirrorpath>
      . If you use anything other than default port (443), then provide the custom port.
    • Enter the Username for the private registry.
    • Enter the API key/ Password for the private registry.
  • Multiple repositories

    Enter the following details for both OpenShift images repository and IBM Spectrum Fusion images repository:

    • Enter the URL of the respective private image registry OpenShift images repository path or IBM Spectrum Fusion images repository path in the Repository path field.
      For example, URLs for OpenShift and IBM Spectrum Fusion images repository paths:
      https://<enterprise registry for IBM Spectrum Fusion>:<custom port>/<mirrorpath>
or 
https://<enterprise registry for Red Hat OpenShift>:<custom port>/<mirrorpath>
      See the following sample values:
      https://registryhost.com:443/fusion-mirror
or
https://registryhost.com:443/mirror-ocp

      If you use anything other than default port (443), then provide the custom port.

    • Enter the Username for the private registry. Make sure that this user has access to the private registry.
    • Enter the API key/ Password for the private registry.

If you need to use a proxy to connect to the external network, select the Connect through a proxy option. To connect through a proxy, enter the URL for the proxy server in the Host address field. If your proxy requires authentication, then enter a Username and Password.

Click Next to go to the Disaster recovery page.

Step 6 - select cluster in Disaster recovery page

  • Select stand-alone cluster in Disaster recovery page

    • If you plan for a single rack installation without disaster recovery, then in the Disaster recovery page, select stand-alone cluster and click Next to go to Global data platform page.

      Note: Even if you choose a stand-alone cluster during installation now, you can make it as the site 1 in a Metro sync DR pair at a later point in time. For the procedure to convert, see Convert stand-alone rack to site 1. To connect this converted site 1 to site 2, set up a second site by using the kubeapi credentials of this stand-alone cluster, which you can retrieve directly from OpenShift Container Platform.
  • Select Metro sync DR pair in Disaster recovery page

    • IBM Spectrum Fusion HCI can optionally be deployed in a Metro sync DR configuration in which two separate IBM Spectrum Fusion HCI clusters are hosted in separate data centers. Data is synchronized between the two clusters, allowing data recovery during loss of data center. For more information about the prerequisites, see Metro sync DR (Disaster Recovery), General Metro sync DR prerequisites, and Setting up the tiebreaker.

      If you install the site 1 in Metro sync DR, then ensure that you install the site 2 as well. As a prerequisite to select second site in Metro sync DR, you must have already installed the first site.

      If you select the site 2 in a disaster recovery pair, then enter Kubernetes API URL and Storage service account credentials of site 1 to configure disaster recovery.

      You can get the URL and credentials of site 1 from the Disaster recovery user interface page of the site 1. For more information about how to retrieve the credentials, see Metro sync DR for IBM Spectrum Fusion HCI.

      Click Next to go to Global data platform page.

Step 7 - configure Global data platform

The Global data platform provides storage capabilities of IBM Spectrum Fusion HCI with three storage configurations, Strong data resiliency, Stronger data resiliency, and Better storage efficiency.
  • If there are less than 11 storage nodes in the appliance, the Strong data resiliency building block configuration is only available.
  • If there are 11 storage nodes in the appliance, you can choose between Strong data resiliency and Better storage efficiency building block configurations.
  • If there are more than 11 storage nodes in the appliance, you can choose between Stronger data resiliency and Better storage efficiency building block configuration.

A Stronger data resiliency option structured into the optimal set of (4+2P) building blocks based on the number of nodes in the rack. If there are enough nodes for two building blocks, IBM Spectrum Fusion creates a two building block configuration that increases the number of simultaneous node failures that the appliance can withstand. For example, if you set up a rack and choose a (4+2P) building block configuration then for every six nodes, you get two nodes or disk failure tolerance. If there are less number of nodes for two building block configuration, a Strong data resiliency option appears with one (4+2P) building block configuration. For example, if you set up a rack and choose a (4+2P) building block configuration then for every six nodes, you get one node or disk failure tolerance.

The Better storage efficiency option uses a single building block configuration, consisting of a minimum of eleven storage nodes. The building block resiliency is achieved by using (8+3p) erasure coding. This means that the cluster can withstand a maximum of three storage node failures, with the data recovered from other nodes in the building block.

For example, if you set up a rack and choose a (8+3P) building block configuration, then for every eleven nodes, you get three node/disk failure tolerance. If you have more than eleven storage nodes, the default choice is Stronger data resiliency because it can withstand more node or disk failures, which is typically preferred in a production environment. The Better storage efficiency option must be used when you want to maximize the amount of usable capacity that you get out of the rack.

Note: The building block configuration that you choose applies to all storage nodes that you add in the future when you expand the rack. For example, if you start with a single (4+2p) Stronger data resiliency building block and later add six additional storage nodes, then those new nodes are used to create a second (4+2p) Stronger data resiliency building block.
Important: You cannot change the building block configuration postinstallation.
Note: You do not have to configure the Global data platform for site 2 in a Metro sync DR setup because IBM Spectrum Fusion HCI automatically applies the configuration of the primary rack or site 1 to site 2.

Step 8 - customize storage block size

An advanced setting is available that allows you to customize the Block size that is set for IBM Spectrum Fusion HCI’s Global data platform. Because OpenShift clusters run a mix of workloads with different I/O characteristics, it is recommended that you use the default 4 MiB block size, which is optimized for mixed workloads.

If the applications that get deployed to the OpenShift cluster are specifically workloads that drive large or small IOs, you can customize the block size for better performance.

The available block sizes for Stronger data resiliency or (4+2p) erasure code are 1 MiB and 4 MiB.

The available block sizes for Better storage efficiency or (8+3p) erasure code are 1 MiB, 4 MiB, and 16 MiB.

Note: Considerations before you choose a block size:
  1. For general workloads, it is recommended to choose 4 MiB block size.
  2. If the cluster primarily deals with many small files, it is recommended to choose a small block size such as 1 MiB.
  3. For large files or streaming workloads, it is recommended to choose 16 MiB block size.

For more information about block size, see Block size.

Step 9 - customize network

The Network customization page shows the network configuration to setup OpenShift, as well as the network configuration for the internal storage network of IBM Spectrum Fusion. You can use the default values in this page and customize values only whenever you have specific network requirements. The OpenShift network section allows you to override the default network CIDRs, which might collide with the data center networks. Also, it provides the flexibility to choose your own range during network planning. In the OpenShift network section, enter the following network configuration for Red Hat OpenShift:
  • Enter Pod network CIDR. It is the IP address pools from which pod IP addresses are allocated. The default value is 10.128.0.0/14.
    Note: The Pod network CIDR must be different between site 1 and site 2 of Metro sync DR.
  • Enter Service network CIDR. It is the IP address pool for the services. The default value is 172.30.0.0/16.
    Note: The Service network CIDR must be different between site 1 and site 2 of Metro sync DR.
  • Enter Pod network Host Prefix. It is the subnet prefix length to assign to each individual node. For example, if hostPrefix is set to 23, then each node is assigned a /23 subnet out of the given CIDR. A hostPrefix value of 23 provides 510 (2^(32 - 23) - 2) pod IP addresses. The default value is 23.
The Storage network section displays the details of the internal storage network of IBM Spectrum Fusion HCI. Because this network is completely internal to the rack, you might not need to make modifications to any of the values. If you plan for Metro sync DR configuration, you might have to modify these values. In that scenario, the storage network is stretched between two data centers. In the Storage network section, enter the following network configuration details for the internal storage network.
  • Enter CIDR address. It is the network subnet that is used for the multus additional storage network of the scale core pods. The default value is 192.168.128.0/18.
  • Enter Gateway address. It is the gateway address for the scale core Pods on the multus additional storage network that connects to the other site in a Metro sync DR.
    Note: In case of Metro sync DR, the gateway must be pre-configured in the customer data center.
  • Enter IP address range. It is the IP address pools from which scale core pod IP addresses are allocated for the multus additional storage network. The default range is 192.168.128.11 - 192.168.191.254.
  • Enable or disable jumbo frames.

    Normal Ethernet packets have an MTU size of 1500 bytes, while jumbo packets are larger packets with an MTU size of up to 9000 bytes. Jumbo packets improve network performance but require switch support. When storage traffic is internal, it can be set to jumbo as IBM Spectrum Fusion switches support it. But in the case of Metro sync DR, the storage traffic is routed through your data center switches to the other site.

    Note: The site 1 or stand-alone rack is enabled by default for jumbo frames. Based on the selection made during installation of site 2 in the Metro sync DR setup, the site 1 gets updated.
Note: The CIDR address, Gateway address, and IP address range must be unique across site 1 and site 2.

Click Next to go to the Custom certificate wizard page.

Step 10 - configure custom certificate

The Custom certificate page allows you to optionally configure a custom certificate for OpenShift. By default, the OpenShift gets configured with a self-signed certificate. However, it is recommended that you upload a certificate that is provided by Certificate Authority (CA). Applying a custom certificate during the installation ensures that the certificate is used immediately by OpenShift. If you do not apply custom certificate during installation, then you can do it later from OpenShift. For more information about how to apply custom certificate from OpenShift, see Ingress Operator in OpenShift Container Platform.

In the Custom certificate wizard page, drag and drop to upload a .crt file of a size that does not exceed 1 MB or enter the details as text input.

Enter the Private key and click Next. The OpenShift initialization page gets displayed.

Initializing OpenShift cluster

The final step of this phase of installation is to create a three node Red Hat OpenShift cluster. This minimal cluster is used in the next phase of the installation to orchestrate building out the cluster and configuring the Global Data Platform for IBM Spectrum Fusion HCI.

As the OpenShift cluster is created, you can monitor the progress. In case of failures, collect logs to analyze the errors or if the initialization failed, view logs to troubleshoot the issue and click Retry. If you need to change any information entered in previous install steps, click Change install settings.

Step 11 - copy credentials

After the OpenShift cluster gets successfully created, you can view the credentials for the OpenShift cluster. It is important that you save these credentials before installation proceeds with the next phase of the installation because you cannot access the OpenShift cluster without this credential.

In the OpenShift credentials section, click Password and CoreOS Key link to download your Red Hat OpenShift and select I have downloaded the OpenShift Password and CoreOS Key. After you select I have downloaded the OpenShift Password and CoreOS Key, the step 2 Launch section gets enabled. To obtain the password from the downloaded file, run the following commands:
  1. Go to Downloads folder: 
    cd ~/Downloads
  2. List the files in the folder: 
    ls -ltr
  3. Extract the contents of ocpkeys compressed file.
  4. Go to the auth folder: 
    cd clusterconfigs/auth
  5. Open kubeadmin-password in edit mode and copy the password: 
    vi kubeadmin-password
  6. Go to the extracted folder /install, save the CoreOS Key:
    id_rsa
    In the installation folder, id_rsa is a CoreOS key that can be used to connect to CoreOS nodes.

Copy the Username and Password from the IBM Spectrum Fusion HCI user interface and secure the username and password for future reference. You must note down and secure the password as it cannot be recovered after the installation proceeds to the next steps. These credentials are configured as single sign-on between Red Hat OpenShift and IBM Spectrum Fusion.

Step 12- install Fusion software

Click IBM Spectrum Fusion. The login page of OpenShift Container Platform is displayed in a new browser tab. Enter the credentials that you noted down and click Log in to resume with the installation.

Cluster expansion

The Fusion software installation stage completes and Cluster expansion stage begins.

View the progress of OpenShift configuration on your nodes. After the statuses of all nodes change to Configured, the Global data platform installation section is enabled to configure storage on all your nodes. If your installation includes disaster recovery, then connections get established between your disaster recovery sites. For site 2 in a Metro sync DR setup, the configuration for Global Data Platform is same as that of site 1.

Note: For any Metro sync DR site 2 or tiebreaker installation issues, contact IBM support.

Step 13- complete the installation

After all configurations are successful, click Launch Fusion to go to the IBM Spectrum Fusion HCI user interface. Click Collect logs to download the logs of individual configurations.

In the welcome message box of IBM Spectrum Fusion HCI user interface, click Install to install the IBM Spectrum Fusion HCI services like Global data platform and Backup & restore service. You can also click Skip for now to skip this installation and do it at a later point in time.

For the procedure to install and work with services, see Managing IBM Spectrum Fusion services.

If you encounter errors in the OpenShift installation wizard, see Installation and upgrade issues. If you encounter errors in the Provisioning and software installation wizard, check the logs. For more information about accessing these logs, see Collecting log files of final installation.

What to do next

  • After successful installation, follow the instructions:
    • If you are using enterprise registry for installation, delete the catalog source named ibm-operator-catalog and opencloud-operator from the openshift-marketplace namespace.
    • If you are using the online installation mode, edit catalog source named ibm-operator-catalog and add the following lines to the spec section:
      updateStrategy:
    registryPoll:
      interval: 45m
      Example catlogsource yaml: 
      apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-operator-catalog
  namespace: openshift-marketplace
spec:
  displayName: IBM Operator Catalog
  publisher: IBM
  sourceType: grpc
  image: icr.io/cpopen/ibm-operator-catalog:latest
  updateStrategy:
    registryPoll:
      interval: 45m
  • To verify the installation, see Validating IBM Spectrum Fusion HCI installation.
  • Optionally, after the storage is available, configure the OpenShift Container Platform image registry. For the procedure to configure, see Changing the image registry’s management state section and Configuring registry storage for bare metal and other manual installations section of OpenShift documentation.
    Run the following command to make this registry accessible outside the cluster. 
    oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge