Installing IBM Storage Fusion HCI System

IBM Storage Fusion HCI System comes with the bootstrapping software that is installed from the factory. The IBM support representative completes the initial verification and physically connects the system to network and power. Then, they conduct the network setup of the appliance, which connects the appliance to the data center network. This procedure configures all the default nodes (Three controllers and three compute nodes). If you ordered more nodes, then they get installed as well. Use this information to install IBM Storage Fusion HCI System. Finally, Storage and Backup software get installed on these nodes.

Before you begin

A connection to the internet is needed to install the software that operates the IBM Storage Fusion HCI System appliance. If you plan to do an offline installation of IBM Storage Fusion, mirror external-operators and OpenShift® Container Platform image repositories to your registry. For the actual steps, see Enterprise registry for IBM Storage Fusion HCI System installation.
Important: IBM Storage Fusion HCI System requires Red Hat OpenShift Container Platform. If you purchased OpenShift subscriptions from IBM, see Activating Red Hat OpenShift Container Platform subscriptions purchased from IBM. For the supported version of Red Hat OpenShift Container Platform, see https://www.ibm.com/support/pages/node/6829067.
Important: Before the installation of IBM Storage Fusion HCI System, enable IBM Storage Fusion HCI System Software to be downloaded. For steps to enable, see Activating IBM Storage Fusion HCI System Software to be downloaded.
IBM’s delivery service removes the rack from the pallet, unpacks, and moves the rack to its final location on the data center floor. If you instruct to leave the pallet or rack packed in the staging area, then it is your responsibility to remove, unpack, and move the rack to its final location. Follow the printed manual to unbox the rack and locate it in your data center.
If client-provided rack is used, then the following system components are delivered in two boxes that are shipped on one or more pallets:
- First box has four switches (two High-speed and two management) and all nodes
- Second box have rails, cables, and manuals
Ensure that you move the components to the data centre and all hardware is in a clear space near to your frame.
Ensure that IBM Storage Fusion HCI System is installed in a restricted access location, such that the area is accessible only to skilled and instructed persons with proper authorization.
If you plan to use a custom certificate, follow the OpenShift Container Platform guidelines to create a certificate. Consider the following points for the chain of certificates:
- Wildcard certificate must be the first certificate in the file.
- Any intermediate certificates must follow the wildcard certificate.
- End of the file must have root CA certificate.
Note: Ensure that the certificate is well-known and not a self-signed certificate.
Go through the Site-readiness topics and confirm whether your premise satisfies the requirements.
The following are the high-level steps to set up and configure the hardware appliance:
1. You must unpack the appliance and locate it in your data center. For planning and prerequisites, see Planning and prerequisites.
  If you purchased the system without an IBM rack, the IBM SSR unpacks the components from their boxes and installs them into the client-provided rack.
2. The Service Support Representative (SSR) assists you in network cabling and power connections by using the details that are provided by your network team.
3. The SSR assists you with the Network setup stage of the installation. The network setup includes the configuration of your network, such as high-speed switch and IP configuration. For more information, see Network planning.
If you plan to use a proxy server for internet access, then do the following steps:
- See Proxy configuration.
- Contact IBM Support team.
The Container Network Interface (CNI) network (daemon network) is created for IBM Spectrum® Scale Erasure Code Edition (ECE) core pods. By default, IP addresses are assigned for scale daemon network. You can override the default IP address before you begin the Final installation. For the procedure, see Configuring Scale daemon network IP parameters.
Note: You can override the default IP addresses only before you start the Final Installation wizard.
If you plan to install site 2 in a Metro-DR setup, ensure that the following prerequisites are met:
- IBM Spectrum Scale on the site 1 is healthy and all IBM Spectrum Scale core pods are up and running.
- Ensure that the disk count is same on site 1 and site 2.
If you installed IBM Storage Fusion HCI System version 2.4 by using offline or online installation mode, then ensure that you do not change the mode during the upgrade to 2.5 version. To change the installation mode, reinstall IBM Storage Fusion HCI 2.5.
Run the command to pull the image and verify that you have access to the IBM Storage Fusion HCI System images.
Note: You can use either Podman or Docker to verify the access.
```
podman login cp.icr.io -u cp -p "CLIENT ENTITLEMENT KEY"
podman pull cp.icr.io/cp/isf/isf-validate-entitlement@sha256:1a0dbf7c537f02dc0091e3abebae0ccac83da6aa147529f5de49af0f23cd9e8e
```
If the pull is successful, then you have a valid entitlement for IBM Storage Fusion HCI System images.
For high availability cluster, you must have received three racks from IBM with a minimum of six nodes per rack. Also, you must buy a pair of spine switches from IBM. Cables of required length to attach the spine switches can be IBM provided or client provided.

Step 1- start the installer

Your IBM Systems Support Representative (IBM SSR) shares an IP address or hostname after they connect the appliance to your network. Open the following URL:

http://<host IP address>:3000/isfsetup

Replace <host IP address> with the hostname or IP you received from SSR.

The URL takes you to the IBM Storage Fusion HCI System installer welcome page.

Step 2- accept license agreement

On the License Agreement page, review the terms and conditions of the license. If you want to keep a copy, download the license. After you go through and accept the license agreements, click I accept the license agreement, and click Continue to proceed with the installation wizard.

Step 3- go through installation summary

The Getting started page explains the installation steps and the estimated time. The estimated time for the Storage Fusion software installation to complete is 120 minutes. The Cluster expansion takes another 60 minutes to complete.

Step 4- precheck the network

The Network precheck page runs an automatic network check against all of the nodes in the appliance. It checks whether each node has an assigned IP address and hostname. If all nodes are marked with a green Connected status, you can proceed to the next step of the installer.

Note: In a high-availability cluster, it displays all the nodes present in all three racks. In a single rack, check whether you can see all your nodes.

Any node that does not pass the network check is marked with a red Disconnected status. It means that either DHCP or DNS configuration for the node is not available. For more information about the prerequisite, see Setting up the DNS and DHCP for IBM Storage Fusion appliance.

Work with your network team to ensure that DNS and DHCP are configured for all nodes in the appliance. After the DHCP or DNS changes, click Restart precheck to initiate a new network check. If you want changes to your node settings, contact your network team.

Click Next to go to the Image registry settings.

Step 5- set up image registry

IBM Storage Fusion installs Red Hat OpenShift Container Platform and IBM Storage Fusion software by using the images that are hosted in the Red Hat and IBM entitled registries. If you want to use your private image registry, you can install both Red Hat OpenShift and IBM Storage Fusion HCI System software from images, which are maintained in a container registry that you manage. For steps to plan and use your own enterprise registry, see Enterprise registry for IBM Storage Fusion HCI System installation.

As part of this step, you can configure a proxy to connect to the repository. Using a proxy is most common for connecting to the public image registries as it requires connecting from your private network to public websites.

Choose whether to use the Public image registry or Private image registry option.

Public image registry

To use the public image registry, you need a pull secret and an entitlement key.

Enter the Pull secret. It is an authorization token that stores Docker credentials that you can use to access a registry. Your cluster needs this secret to access and pull OpenShift images from the quay.io container registry. If you do not have a pull secret, click Get Pull secret. It takes you to https://cloud.redhat.com/openshift/install/pull-secret.
Enter the Entitlement key. It is a product code that is used to pull images from IBM Entitlement Registry. Your cluster needs this key to gain access to IBM Storage Fusion images in the IBM Entitlement Registry. If you do not have a key, click Get Entitlement key. It takes you to IBM Container Library. For steps to obtain the key, see the Activating IBM Storage Fusion HCI System Software to be downloaded.

Private image registry

If you select the Private image registry, you must first mirror the Red Hat and IBM Storage Fusion images to your private registry. For more information about mirroring, see Mirroring your images to the enterprise registry.

You can choose to host the Red Hat and IBM Storage Fusion images in separate repositories, or use the same repository.

Single repository
Enter the following details for the enterprise registry.
- Enter the URL of the private registry in the Repository path.
  For example,
```
https://<enterprise registry>:<custom port>/<mirrorpath>
```
  If you want to use custom port, then provide the custom port details.
- Enter the Username for the private registry.
- Enter the API key/ Password for the private registry.
Multiple repositories
Enter the following details for both OpenShift images repository and IBM Storage Fusion images repository:
- Enter the URL of the respective private image registry OpenShift images repository path or IBM Storage Fusion images repository path in the Repository path field.
  For example, URLs for OpenShift and IBM Storage Fusion images repository paths:
```
https://<enterprise registry for IBM Storage Fusion>:<custom port>/<mirrorpath>
or 
https://<enterprise registry for Red Hat OpenShift>:<custom port>/<mirrorpath>
```
  See the following sample values:
```
https://registryhost.com:443/fusion-mirror
or
https://registryhost.com:443/mirror-ocp
```
  If you use anything other than default port (443), then provide the custom port.
- Enter the Username for the private registry. Make sure that this user has access to the private registry.
- Enter the API key/ Password for the private registry.

If you need to use a proxy to connect to the external network, select the Connect through a proxy option. To connect through a proxy, enter the URL for the proxy server in the Host address field. If your proxy requires authentication, then enter a Username and Password.

Click Next to go to the Disaster recovery page.

Step 6- select cluster in Disaster recovery page

Select stand-alone cluster in Disaster recovery page
- If you plan for a single rack installation without disaster recovery, then in the Disaster recovery page, select stand-alone cluster and click Next to go to Global data platform page.
  
  Note: Even if you choose a stand-alone cluster during installation now, you can make it as the site 1 in a Metro-DR pair at a later point in time. For the procedure to convert, see Convert stand-alone rack to site 1. To connect this converted site 1 to site 2, set up a second site by using the kubeapi credentials of this stand-alone cluster, which you can retrieve directly from OpenShift Container Platform.
Select Metro-DR pair in Disaster recovery page
- IBM Storage Fusion HCI System can optionally be deployed in a Metro-DR configuration in which two separate IBM Storage Fusion HCI System clusters are hosted in separate data centers. Data is synchronized between the two clusters, allowing data recovery during loss of data center. For more information about the prerequisites, see Metro-DR (Disaster Recovery), General Metro-DR prerequisites, and Setting up the tiebreaker.
  
  If you install the site 1 in Metro-DR, then ensure that you install the site 2 as well. As a prerequisite to select second site in Metro-DR, you must have already installed the first site.
  
  If you select the site 2 in a disaster recovery pair, then enter Kubernetes API URL and Storage service account credentials of site 1 to configure disaster recovery.
  
  You can get the URL and credentials of site 1 from the Disaster recovery user interface page of the site 1. For more information about how to retrieve the credentials, see Metro-DR for IBM Storage Fusion HCI System.
  
  Click Next to go to Global data platform page.

Step 7- configure Global Data Platform

Note: The Global data platform wizard page is not available for high availability cluster. For high availability three rack cluster, you do not have to choose a building block configuration.

The Global data platform wizard page provides storage capabilities of IBM Storage Fusion HCI System with three storage configurations, Strong data resiliency, Stronger data resiliency, and Better storage efficiency.

If there are less than 11 storage nodes in the appliance, the Strong data resiliency building block configuration is only available.
If there are 11 storage nodes in the appliance, you can choose between Strong data resiliency and Better storage efficiency building block configurations.
If there are more than 11 storage nodes in the appliance, you can choose between Stronger data resiliency and Better storage efficiency building block configuration.

A Stronger data resiliency option structured into the optimal set of (4+2P) building blocks based on the number of nodes in the rack. If there are enough nodes for two building blocks, IBM Storage Fusion creates a two building block configuration that increases the number of simultaneous node failures that the appliance can withstand. For example, if you set up a rack and choose a (4+2P) building block configuration then for every six nodes, you get two nodes or disk failure tolerance. If there are less number of nodes for two building block configuration, a Strong data resiliency option appears with one (4+2P) building block configuration. For example, if you set up a rack and choose a (4+2P) building block configuration then for every six nodes, you get one node or disk failure tolerance.

The Better storage efficiency option uses a single building block configuration, consisting of a minimum of eleven storage nodes. The building block resiliency is achieved by using (8+3p) erasure coding. This means that the cluster can withstand a maximum of three storage node failures, with the data recovered from other nodes in the building block.

For example, if you set up a rack and choose a (8+3P) building block configuration, then for every eleven nodes, you get three node/disk failure tolerance. If you have more than eleven storage nodes, the default choice is Stronger data resiliency because it can withstand more node or disk failures, which is typically preferred in a production environment. The Better storage efficiency option must be used when you want to maximize the amount of usable capacity that you get out of the rack.

Note: The building block configuration that you choose applies to all storage nodes that you add in the future when you expand the rack. For example, if you start with a single (4+2p) Stronger data resiliency building block and later add six additional storage nodes, then those new nodes are used to create a second (4+2p) Stronger data resiliency building block.

Important: You cannot change the building block configuration postinstallation.

Note: You do not have to configure the Global data platform for site 2 in a Metro-DR setup because IBM Storage Fusion HCI System automatically applies the configuration of the primary rack or site 1 to site 2.

Step 8- customize storage block size

Note: For high availability rack cluster, you do not have to choose a storage block size. The (4+3p) erasure code is the only option available.

An advanced setting is available that allows you to customize the Block size that is set for IBM Storage Fusion HCI System’s Global data platform. Because OpenShift clusters run a mix of workloads with different I/O characteristics, it is recommended that you use the default 4 MiB block size, which is optimized for mixed workloads.

If the applications that get deployed to the OpenShift cluster are specifically workloads that drive large or small IOs, you can customize the block size for better performance.

For single rack, the available block sizes for Stronger data resiliency or (4+2p) erasure code are 1 MiB and 4 MiB.

For high-availability cluster, the available block sizes for Stronger data resiliency or (4+3p) erasure code are 1 MiB and 4 MiB.

The available block sizes for Better storage efficiency or (8+3p) erasure code are 1 MiB, 4 MiB, and 16 MiB.

Note: Considerations before you choose a block size:

For general workloads, it is recommended to choose 4 MiB block size.
If the cluster primarily deals with many small files, it is recommended to choose a small block size such as 1 MiB.
For large files or streaming workloads, it is recommended to choose 16 MiB block size.

For more information about block size, see Block size.

Step 9- customize network

The Network customization page shows the network configuration to setup OpenShift, as well as the network configuration for the internal storage network of IBM Storage Fusion. You can use the default values in this page and customize values only whenever you have specific network requirements. The OpenShift network section allows you to override the default network CIDRs, which might collide with the data center networks. Also, it provides the flexibility to choose your own range during network planning. In the OpenShift network section, enter the following network configuration for Red Hat OpenShift:

Enter Pod network CIDR. It is the IP address pools from which pod IP addresses are allocated. The default value is 10.128.0.0/14.
Note: The Pod network CIDR must be different between site 1 and site 2 of Metro-DR.
Enter Service network CIDR. It is the IP address pool for the services. The default value is 172.30.0.0/16.
Note: The Service network CIDR must be different between site 1 and site 2 of Metro-DR.
Enter Pod network Host Prefix. It is the subnet prefix length to assign to each individual node. For example, if hostPrefix is set to 23, then each node is assigned a /23 subnet out of the given CIDR. A hostPrefix value of 23 provides 510 (2^(32 - 23) - 2) pod IP addresses. The default value is 23.

The Storage network section displays the details of the internal storage network of IBM Storage Fusion HCI System. Because this network is completely internal to the rack, you might not need to make modifications to any of the values. If you plan for Metro-DR configuration, you might have to modify these values. In that scenario, the storage network is stretched between two data centers. In the Storage network section, enter the following network configuration details for the internal storage network.

Enter CIDR address. It is the network subnet that is used for the multus additional storage network of the scale core pods. The default value is 192.168.128.0/18.
Enter Gateway address. It is the gateway address for the scale core Pods on the multus additional storage network that connects to the other site in a Metro-DR.
Note: In case of Metro-DR, the gateway must be pre-configured in the customer data center.
Enter IP address range. It is the IP address pools from which scale core pod IP addresses are allocated for the multus additional storage network. The default range is 192.168.128.11 - 192.168.191.254.
Enable or disable jumbo frames.
Normal Ethernet packets have an MTU size of 1500 bytes, while jumbo packets are larger packets with an MTU size of up to 9000 bytes. Jumbo packets improve network performance but require switch support. When storage traffic is internal, it can be set to jumbo as IBM Storage Fusion switches support it. But in the case of Metro-DR, the storage traffic is routed through your data center switches to the other site.

Note: The site 1 or stand-alone rack is enabled by default for jumbo frames. Based on the selection made during installation of site 2 in the Metro-DR setup, the site 1 gets updated.

Note: The CIDR address, Gateway address, and IP address range must be unique across site 1 and site 2.

Click Next to go to the Custom certificate wizard page.

Step 10 - configure custom certificate

The Custom certificate page allows you to optionally configure a custom certificate for OpenShift. By default, the OpenShift gets configured with a self-signed certificate. However, it is recommended that you upload a certificate that is provided by Certificate Authority (CA). Applying a custom certificate during the installation ensures that the certificate is used immediately by OpenShift. If you do not apply custom certificate during installation, then you can do it later from OpenShift. For more information about how to apply custom certificate from OpenShift, see Ingress Operator in OpenShift Container Platform.

In the Custom certificate wizard page, drag and drop to upload a .crt file of a size that does not exceed 1 MB or enter the details as text input.

Enter the Private key and click Next. The OpenShift initialization page gets displayed.

Initializing OpenShift cluster

The final step of this phase of installation is to create a three node Red Hat OpenShift cluster. This minimal cluster is used in the next phase of the installation to orchestrate building out the cluster and configuring the Global Data Platform for IBM Storage Fusion HCI System.

As the OpenShift cluster is created, you can monitor the progress. In case of failures, collect logs to analyze the errors or if the initialization failed, view logs to troubleshoot the issue and click Retry. If you need to change any information entered in previous install steps, click Change install settings.

Step 11- copy credentials

After the OpenShift cluster gets successfully created, you can view the credentials for the OpenShift cluster. Save these credentials before installation proceeds with the next phase of the installation because you cannot access the OpenShift cluster without this credential.

Note: After you save the password and download the ocpkey.zip file, the IBM Storage Fusion HCI System or Red Hat OpenShift URL launches the IBM Storage Fusion HCI System console or OpenShift Console.

In the OpenShift credentials section, click Password and CoreOS Key link to download your Red Hat OpenShift and select I have downloaded the OpenShift Password and CoreOS Key. After you select I have downloaded the OpenShift Password and CoreOS Key, the step 2 Launch section gets enabled.

To obtain the password from the downloaded file, run the following commands:

Go to Downloads folder:
```
cd ~/Downloads
```
List the files in the folder:
```
ls -ltr
```
Extract the contents of ocpkeys compressed file.
Go to the auth folder:
```
cd clusterconfigs/auth
```
Open kubeadmin-password in edit mode and copy the password:
```
vi kubeadmin-password
```
Go to the extracted folder /install, save the CoreOS Key:
```
id_rsa
```
In the installation folder, id_rsa is a CoreOS key that can be used to connect to CoreOS nodes.

Copy the Username and Password from the IBM Storage Fusion HCI System user interface and secure the username and password for future reference. You must note down and secure the password as it cannot be recovered after the installation proceeds to the next steps. These credentials are configured as single sign-on between Red Hat OpenShift and IBM Storage Fusion.

Note: After you save the password and download the ocpkey.zip file, the URL points you to the OCP address. If your URL does not automatically point to the OCP address, then check the Network Preparation tab in your TDA installation worksheet to ensure that the DNS wildcard domain name is added to the DNS server. Test your connectivity with https://DNSentryingressendpointipaddress.

Step 12-install Fusion software

Click IBM Storage Fusion. The login page of the IBM Storage Fusion HCI System console displays in a new browser tab. Enter the credentials that you noted down and click Log in to resume with the installation.

Note: If there are any issues to access the console, then verify that your DNS server has a wildcard DNS A/AAAA or CNAME record that refers to OpenShift ingress. Test your connectivity with https://console-ibm-storage-fusion-ns.apps.<cluster_name>.<base_domain> .

Cluster expansion

The Fusion software installation stage completes and Cluster expansion stage begins.

View the progress of OpenShift configuration on your nodes. After the statuses of all nodes change to Configured, the Global data platform installation section is enabled to configure storage on all your nodes. If your installation includes disaster recovery, then connections get established between your disaster recovery sites. For site 2 in a Metro-DR setup, the configuration for Global Data Platform is same as that of site 1.

Note: For any Metro-DR site 2 or tiebreaker installation issues, contact IBM support.

Step 13-complete the installation

After all configurations are successful, do the following steps to Collect logs for each configuration:

Click the drop-down of the respective configuration.
Click Collect logs.
It downloads all the necessary logs.

Once you collect logs for all the available configurations, click Launch Fusion to go to the IBM Storage Fusion HCI System user interface.

For the procedure to install and work with services, see Managing IBM Storage Fusion services.

If you encounter errors in the OpenShift installation wizard, see Installation and upgrade issues. If you encounter errors in the Provisioning and software installation wizard, check the logs. For more information about accessing these logs, see Collecting log files of final installation.

What to do next

To verify the installation, see Validating IBM Storage Fusion HCI System installation.
For high-availability cluster, go to IBM Storage Fusion HCI System user interface and view rack details. For the procedure to view rack details, see Adding racks.
Optionally, after the storage is available, configure the OpenShift Container Platform image registry. For the procedure to configure, see Changing the image registry’s management state section and Configuring registry storage for bare metal and other manual installations section of OpenShift documentation.
Run the following command to make this registry accessible outside the cluster.
```
oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
```
If you want to install Noobaa service, then install Red Hat OpenShift Data Foundation and deploy it as Multicloud Object Gateway (MCG) only mode to provide object service. For the procedure, see Deploying stand-alone MCG on the IBM Storage Fusion HCI System for Quay.