Installing IBM Spectrum Fusion HCI

Use this information to install IBM Spectrum Fusion HCI.

Before you begin

A connection to the internet is required to install the software that operates the IBM Spectrum Fusion HCI appliance. If you plan to do an offline installation of IBM Spectrum Fusion, mirror external-operators and OpenShift® Container Platform image repositories to your registry. For the actual steps, see Enterprise registry for IBM Spectrum Fusion HCI installation.
Important: You need IBM Spectrum Fusion software entitlement to access IBM Spectrum Fusion images. For steps to register your customer number in Entitlement System Support, see Registering customer number in Entitled System Support (ESS).
Important: IBM Spectrum Fusion HCI requires Red Hat OpenShift Container Platform, which you must purchase. OpenShift is not pre-entitled with the IBM Spectrum Fusion HCI appliance. For more information, see Step 5 of Registering customer number in Entitled System Support (ESS) topic.
Ensure that IBM Spectrum Fusion HCI is installed in a restricted access location, such that the area is accessible only to skilled and instructed persons with proper authorization.
Follow the OpenShift Container Platform guidelines to create a certificate. Consider the following points for the chain of certificates:
- Wildcard certificate must be the first certificate in the file
- Any intermediate certificates must follow the wildcard certificate
- End of the file must have root CA certificate
Go through the Site-readiness topics and confirm whether your premise satisfies the requirements.
The following are the high-level steps to set up and configure the hardware appliance:
1. You must unpack the appliance and locate it in your data center. For planning and prerequisites, see Planning and prerequisites.
2. The Service Support Representative (SSR) assists you in network cabling and power connections by using the details that are provided by your network team.
3. The SSR assists you with the Network setup stage of the installation. The network setup includes the configuration of your network, such as high-speed switch and IP configuration. For more information, see Network planning.
If you plan to use a proxy server for internet access, then do the following steps:
- See Firewall requirements for IBM Spectrum Fusion HCI.
- Contact IBM Support team.
The Container Network Interface (CNI) network (daemon network) is created for Scale core pods. By default, IP addresses are assigned for scale daemon network. You can override the default IP address before you begin the Final installation. For the procedure, see Configuring Scale daemon network IP parameters.
Note: You can override the default IP addresses only if you have not started the Final Installation Wizard.
If you plan to install site 2 in a Metro sync DR setup, ensure that the following prerequisites are met:
- IBM Spectrum Scale on the site 1 is healthy and all IBM Spectrum Scale core pods are up and running.
- Ensure that the disk count is same on site 1 and site 2.

About this task

IBM Spectrum Fusion HCI comes with the required bootstrapping software that is installed from the factory.

The IBM support representative completes the initial verification and physically connects the system to network and power. Then, they conduct the network setup of the appliance, which connects the appliance to the data center network.

This procedure configures all the default nodes (Three controllers and three compute nodes). If you ordered additional nodes, then they get installed as well. Finally, Storage and Backup software get installed on these nodes. For upsize procedure, see Upsizing nodes.

Procedure

Start the OpenShift installation wizard by using the http://<host IP address>:3000/isfsetup URL.

Note: This host IP address is what you got at the end of the Network setup stage of the installation.

The URL navigates you to the IBM Spectrum Fusion HCI installer License agreement page. This step is the beginning of OpenShift initialization stage of the installation.
In the License agreement page, go through the license agreement links.
If you agree to the terms and conditions of the license, click I accept the license agreement.
Click Continue to proceed with the installation wizard after you have read and accepted all the license agreements.
The Getting started page gets displayed.
In the Getting started page, click Next.
Getting started page explains the installation steps and the estimated time. The estimated time for the Spectrum Fusion software installation to complete is 120 minutes. The Cluster expansion takes another 60 minutes to complete.
The Network precheck wizard page gets displayed.
In the Network precheck wizard page, observe the progress of nodes.
The Network precheck wizard page runs an automatic network check against all of the nodes in the appliance. This checks whether each node has an assigned IP address and hostname. If all nodes are marked with a green Connected status, then you can proceed to the next step of the installer. Any nodes that do not pass the network check are marked with a red Failed status. This means that either DHCP or DNS is not configured for the node. For more information about the prerequisite, see Setting up the DNS and DHCP for IBM Spectrum Fusion appliance.
Work with your network team to ensure that DNS and DHCP are configured for all nodes in the appliance. After you make all the DHCP or DNS changes, click Restart precheck to initiate a new network check.
If you want changes to your node settings, contact your network team.
Click Next.
The Image registry settings page gets displayed.
IBM Spectrum Fusion HCI installs Red Hat OpenShift Container Platform and IBM Spectrum Fusion software by using images that are hosted in the Red Hat and IBM entitled registries. If you want to use your own private image registry, you can install both OpenShift and IBM Spectrum Fusion HCI software from the images that are maintained in a container registry that you manage. If you plan to use your own enterprise registry, see Enterprise registry for IBM Spectrum Fusion HCI installation.
In the Image registry wizard page, enter the following details to configure the image registry:

Optionally, you can configure a proxy to connect to the repository. Using a proxy is most common for connecting to public image registries as it requires connecting from your private network to public websites.

Choose whether to use the Public image registry or Private image registry option.
- If you select the Public image registry, then enter the following details:
  - Enter the Pull secret. It is an authorization token that stores Docker credentials that you can use to access a registry. Your cluster needs this secret to access and pull OpenShift images from the quay.io container registry. If you do not have a pull secret, click Get Pull secret. It takes you to https://cloud.redhat.com/openshift/install/pull-secret. For steps to obtain the pull secret, see Before you begin.
  - Enter the Entitlement key. It is a product code that is used to pull images from IBM Entitlement Registry. Your cluster needs this key to gain access to IBM Spectrum Fusion images in the IBM Entitlement Registry. If you do not have a key, click Get Entitlement key. It takes you to IBM Container Library. For steps to obtain the key, see the Before you begin.
- If you select the Private image registry, you must first mirror the Red Hat and IBM Spectrum Fusion images to your private registry. For more details about mirroring, see Mirroring your images to the enterprise registry.
  - Single repository
    
    Enter the following details for the enterprise registry.
    Enter the URL of the private registry in the Repository path.
    For example,
    https://<enterprise registry>:<custom port>/<mirrorpath>
    If you use anything other than default port (443), then provide the custom port.
    
    Enter the Username for the private registry.
    
    Enter the API key/ Password for the private registry.
  - Multiple repositories
    Enter the following details for both OpenShift images repository and IBM Spectrum Fusion images repository.
    
    Enter the URL of the respective private image registry OpenShift images repository path or IBM Spectrum Fusion images repository path in the Repository path field.
    For example, see the URL for OpenShift and IBM Spectrum Fusion images repository paths.
    https://<enterprise registry for IBM Spectrum Fusion>:<custom port>/<mirrorpath> or https://<enterprise registry for Red Hat OpenShift>:<custom port>/<mirrorpath>
    
    See the following sample values:
    https://registryhost.com:443/fusion-mirror or https://registryhost.com:443/mirror-ocp
    
    If you use anything other than default port (443), then provide the custom port.
    
    Enter the Username for the private registry. Make sure that the user should have access to private registry.
    
    Enter the API key/ Password for the private registry.
Enter the following details in the Proxy settings section:
Select Connect directly or Connect through a proxy option. If you select the Connect through a proxy option, you must enter the Host address, Port, and optional authentication (Username and Password) of your private enterprise registry.
Click Next to go to the Disaster recovery page.

IBM Spectrum Fusion HCI can optionally be deployed in a Metro Disaster Recovery (Metro sync DR) configuration in which two separate HCI clusters are hosted in separate data centers. Data is synchronized between the two clusters, allowing data to be recovered in the event of the loss of a data center. For more information about Metro sync DR and its prerequisites, see Metro sync DR (Disaster Recovery), General Metro sync DR prerequisites, and Setting up the tiebreaker.
In the Disaster recovery page, select whether the installation is for a standalone cluster or a disaster recover pair (first or second).
- The standalone cluster is selected by default. It is a single rack installation.
  Note: Even if you choose a standalone cluster during installation, you can make it as the first site in a Metro sync DR pair at a later point in time. To connect site 2 and convert this to site 1 in the future, set up a second site by using the kubeapi credentials of this standalone cluster, which you can retrieve directly from OpenShift Container Platform.
- If you select the first site in Metro sync DR, then you need to also install the second site for Metro sync DR.
- If you select the second site in Metro sync DR, then as a prerequisite you must have the first site already available. If you are planning for a Metro sync DR setup, then ensure you meet all the prerequisites. For more information about the prerequisites, see Metro sync DR (Disaster Recovery), General Metro sync DR prerequisites, and Setting up the tiebreaker.
If you select the Second site in a disaster recovery pair, then enter Kubernetes API URL and Fusion service API token of site 1 to begin the disaster recovery configuration.
You can get the URL and credentials of site 1 from the Disaster recovery user interface page of the first site. For more information about how to retrieve the credentials, see Metro sync DR for IBM Spectrum Fusion HCI.
Click Next to go to the Global data platform page.
In the Global data platform page, select Strong data resiliency, Stronger data resiliency or Better storage efficiency.

Note: The Global data platform option is not available for the second site in the Disaster recovery page.

The Global data platform provides storage capabilities of IBM Spectrum Fusion HCI with three storage configurations, Strong data resiliency, Stronger data resiliency and Better storage efficiency.
- If there are less than 11 storage nodes in the appliance, the Strong data resiliency building block configuration is only available.
- If there are 11 storage nodes in the appliance, you can choose between Strong data resiliency and Better storage efficiency building block configurations.
- If there are more than 11 storage nodes in the appliance, you can choose between Stronger data resiliency and Better storage efficiency building block configuration.
The Strong data resiliency option uses one building block configuration, which consists of a minimum of 6 storage nodes. The building block resiliency is achieved by using (4+2p) erasure coding. It means that a building block can withstand a maximum of 2 storage node failures, with data being recovered from other storage nodes in the building block. For example, if you set up a rack and choose a (4+2P) building block configuration then for every 6 nodes, you get 2 node or disk failure tolerance.
The Stronger data resiliency option uses two building blocks configuration, each of which consists of a minimum of 6 storage nodes. The building block resiliency is achieved by using (4+2p) erasure coding. It means that a building block can withstand a maximum of 2 storage node failures, with data being recovered from other storage nodes in the building block. For example, if you set up a rack and choose a (4+2P) building block configuration then for every 6 nodes, you get 2 node or disk failure tolerance.
The Better storage efficiency option uses a single building block configuration, consisting of a minimum of 11 storage nodes. The building block resiliency is achieved by using (8+3p) erasure coding. This means that the cluster can withstand a maximum of 3 storage node failures, with the data being recovered from other nodes in the building block. For example, if you set up a rack and choose a (8+3P) building block configuration, then for every 11 nodes, you get 3 node/disk failure tolerance.
If you have more than 11 storage nodes, the default choice is Stronger data resiliency because it can withstand more node or disk failures, which is typically preferred in a production environment. The Better storage efficiency option must be used when you want to maximize the amount of usable capacity that you get out of the appliance.

Note: The building block configuration that you choose applies to all storage nodes that you add in the future when expanding the appliance. For example, if you start with a single (4+2p) Stronger data resiliency building block and later add six additional storage nodes, then those new nodes are used to create a second (4+2p) Stronger data resiliency building block.

Important: You cannot change the building block configuration post installation.
Click Show advanced menu.
Choose the Block size that best suits for your workloads.
An advanced setting is available that allows you to customize the Block size that is set for IBM Spectrum Fusion HCI’s Global data platform. Because OpenShift clusters run a mix of workloads with different I/O characteristics, it is recommended that you use the default 4 MiB block size, which is optimized for mixed workloads.
If the applications that get deployed to the OpenShift cluster are specifically workloads that drive large or small IOs, you can customize the block size for better performance.

Note: The available block sizes for Stronger data resiliency or (4+2p) erasure code are 1 MiB and 4 MiB.
The available block sizes for Better storage efficiency or (8+3p) erasure code are 1 MiB, 4 MiB, and 16 MiB.
Note: Considerations before you choose a block size:
1. For general workloads, it is recommended to choose 4 MiB block size.
2. If the cluster is primarily deals with many small files, it is recommended to choose a small block size such as 1 MiB.
3. For large files or streaming workloads, it is recommended to choose 16 MiB block size.
For more information see, https://www.ibm.com/docs/en/spectrum-scale/5.1.5?topic=considerations-block-size.
Click Next to go to the Network customization page.
The Network customization page shows the network configuration that gets set up for OpenShift, as well as the network configuration that is used for the internal storage network of IBM Spectrum Fusion HCI. You can use the default values in this page, and customization is needed only if you have specific network requirements.
In the Network customization wizard page, enter the following details:
The OpenShift network section allows you to override the default network CIDRs, which might collide with the data center networks. Also, it provides the flexibility to choose your own range while network planning. In the OpenShift network section, enter the following network configuration for Red Hat OpenShift:
- Enter Pod network CIDR. It is the IP address pools from which pod IP addresses are allocated. The default value is 10.128.0.0/14.
  Note: The Pod network CIDR must be different between site 1 and site 2 of Metro sync DR.
- Enter Service network CIDR. It is the IP address pool for the services. The default value is 172.30.0.0/16.
  Note: The Service network CIDR must be different between site 1 and site 2 of Metro sync DR.
- Enter Pod network Host Prefix. It is the subnet prefix length to assign to each individual node. For example, if hostPrefix is set to 23, then each node is assigned a /23 subnet out of the given CIDR. A hostPrefix value of 23 provides 510 (2^(32 - 23) - 2) pod IP addresses. The default value is 23.
The Storage network section displays the details of the internal storage network of IBM Spectrum Fusion HCI. Because this network is completely internal to the appliance, you might not need to make modifications to any of the values. If you plan for Metro sync DR configuration, you might have to modify these values. In that scenario, the storage network is stretched between two data centers. In the Storage network section, enter the following network configuration details for the internal storage network.
- Enter CIDR address. It is the network subnet that is used for the multus additional storage network of the scale core pods. The default value is 192.168.128.0/18.
- Enter Gateway address. It is the gateway address for the scale core Pods on the multus additional storage network that connects to the other site in a Metro sync DR.
  Note: In case of Metro sync DR the gateway must be pre-configured in the customer data center.
- Enter IP address range. It is the IP address pools from which scale core pod IP addresses are allocated for the multus additional storage network. The default range is 192.168.128.11 - 192.168.191.254.
- Enable or disable jumbo frames.
  Normal Ethernet packets have an MTU size of 1500 bytes, while jumbo packets are larger packets with an MTU size of up to 9000 bytes. Jumbo packets improve network performance but require switch support. When storage traffic is internal, it can be set to jumbo as IBM Spectrum Fusion switches support it. But in the case of Metro sync DR, the storage traffic is routed through your data center switches to the other site.
  
  Note: The Site 1 or standalone rack is enabled by default for jumbo frames. Based on the selection made during installation of Site 2 in the Metro sync DR setup, the Site 1 gets updated accordingly.
Note: The CIDR address, Gateway address, and IP address range must be unique across site A and site B.
Click Next to go to the Custom certificate wizard page.
The Custom certificate page allows you to optionally configure a custom certificate for OpenShift. By default, the OpenShift gets configured with a self-signed certificate. However, it is recommended that you upload a certificate that is provided by Certificate Authority (CA). Applying a custom certificate during the install ensures that the certificate is used immediately by OpenShift. If you do not apply custom certificate during installation, then you can do it later as a day 2 operation from OpenShift. For more information about how to apply custom certificate from OpenShift, see Ingress Operator in OpenShift Container Platform.
In the Custom certificate wizard page, either drag and drop to upload .crt file of a size that does not exceed 1 MB or enter the details as text input.
Enter the Private key and click Next.
The OpenShift initialization page gets displayed.
Monitor the progress of the OpenShift initialization.
It creates a three node OpenShift cluster. This minimal cluster is used to orchestrate the cluster and to configure the Global data platform for IBM Spectrum Fusion HCI. In case of failures, collect logs to analyze the errors, fix the issue, and click Change install settings.
After the OpenShift cluster gets successfully created, you can view the credentials for the OpenShift cluster. It is important that you save these credentials before proceeding with the next phase of the install because you cannot access the OpenShift cluster without this credentials.
In the OpenShift credentials section, click Password and CoreOS Key link to download your Red Hat OpenShift and select I have downloaded the OpenShift Password and CoreOS Key.
After you select I have downloaded the OpenShift Password and CoreOS Key, the step 2 Launch section gets enabled. To obtain the password from the downloaded file, run the following commands:
1. Go to Downloads folder:
```
cd ~/Downloads
```
2. List the files in the folder:
```
ls -ltr
```
3. Extract the contents of ocpkeys compressed file.
4. Go to the auth folder:
```
cd clusterconfigs/auth
```
5. Open kubeadmin-password in edit mode and copy the password:
```
vi kubeadmin-password
```
6. Go to the extracted folder /install, save the CoreOS Key:
```
id_rsa
```
  In the install folder, id_rsa is a CoreOS key that can be used to connect to CoreOS nodes.
Copy the Username and Password from the IBM Spectrum Fusion user interface and secure the username and password for future reference.

Important: You must note down and secure the password, as it cannot be recovered after the installation proceeds to the next steps.

These credentials are configured as single sign-on between Red Hat OpenShift and IBM Spectrum Fusion.
Click IBM Spectrum Fusion.
The login page of OpenShift Container Platform is displayed in a new browser tab.
Enter the credentials that you noted down and click Log in to resume with the installation.
The Fusion software installation stage completes, and Cluster expansion stage begins.
View the progress of OpenShift configuration on your nodes. After the status of all nodes changes to Configured, the Global data platform installation section is enabled to configure storage on all your nodes. If your installation includes disaster recovery, then connections get established between your disaster recovery sites. For site 2 in a Metro sync DR setup the configuration for Global Data Platform is same as that of site 1.

Note: For any Metro sync DR site 2 or tiebreaker installation issues, contact IBM support.
After all configurations are successful, click Launch Fusion to go to the IBM Spectrum Fusion user interface.
Click Collect logs to download the logs of individual configurations.
In the welcome message box of IBM Spectrum Fusion HCI user interface, click Install to install the IBM Spectrum Fusion HCI data protection service. You can also click Skip for now to skip this installation and do it at a later point in time.
For the procedure to install and work with services, see Managing services.

If you encounter errors in the OpenShift installation wizard, see Installation and upgrade issues. If you encounter errors in the Provisioning and software installation wizard, check the logs. For more information about accessing these logs, see Collecting log files of final installation.

What to do next

After successful installation, follow the instructions:
- If you are using the online installation mode,
  Edit catalog source named ibm-operator-catalog and, if ibm-operator-catalog source exists, add the following lines to the spec section:
```
updateStrategy:
    registryPoll:
      interval: 45m
```
  Example catlogsource yaml:
```
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-operator-catalog
  namespace: openshift-marketplace
spec:
  displayName: IBM Operator Catalog
  publisher: IBM
  sourceType: grpc
  image: icr.io/cpopen/ibm-operator-catalog:latest
  updateStrategy:
    registryPoll:
      interval: 45m
```
- If you are using enterprise registry for installation,
  Delete the catalog source named ibm-operator-catalog from the openshift-marketplace namespace if the ibm-operator-catalog source exists.
To verify the installation, see Validating IBM Spectrum Fusion HCI installation.

Optionally, after the storage is available, configure the OpenShift Container Platform image registry:

Run the following command to change Removed to Managed for managementState value:

oc edit configs.imageregistry.operator.openshift.io cluster

For example:

apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
  creationTimestamp: "2022-05-12T09:11:56Z"
  finalizers:
  - imageregistry.operator.openshift.io/finalizer
  generation: 1
  name: cluster
  resourceVersion: "3508213"
  uid: fc6eee2a-5dab-44ca-9b45-b7870dfa046d
spec:
  logLevel: Normal
  managementState: Removed
  observedConfig: null
  operatorLogLevel: Normal
  proxy: {}
  replicas: 1
  requests:
    read:
      maxWaitInQueue: 0s
    write:
      maxWaitInQueue: 0s
  rolloutStrategy: RollingUpdate
  storage: {}
  unsupportedConfigOverrides: null

Run the following command to ensure that you have one of storage class set as default.
```
oc patch storageclass ibm-spectrum-fusion-mgmt-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
```
Here, replace ibm-spectrum-fusion-mgmt-sc with the storage class of your choice.

Do the following steps to add storage to Spec section for OpenShift Container Platform image registry:

Run the following command to open configs.imageregistry.operator.openshift.io in edit mode:
```
oc edit configs.imageregistry.operator.openshift.io
```
Add the following to Spec section:
```
storage:
  pvc:
    claim:
```

For example:

apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
  creationTimestamp: "2022-05-12T09:11:56Z"
  finalizers:
  - imageregistry.operator.openshift.io/finalizer
  generation: 1
  name: cluster
  resourceVersion: "3518342"
  uid: fc6eee2a-5dab-44ca-9b45-b7870dfa046d
spec:
  logLevel: Normal
  managementState: Managed
  storage:
  pvc:
    claim:
  observedConfig: null
  operatorLogLevel: Normal
  proxy: {}
  replicas: 1
  requests:
    read:
      maxWaitInQueue: 0s
    write:
      maxWaitInQueue: 0s
  rolloutStrategy: RollingUpdate
  storage: {}
  unsupportedConfigOverrides: null

Run the following command to expose the registry outside the cluster:
```
oc patch configs.imageregistry.operator.openshift.io/cluster --patch ‘{“spec”:{“defaultRoute”:true}}’ --type=merge
```
Note: For more information about image registry operator configuration parameters, see https://docs.openshift.com/container-platform/4.10/registry/configuring-registry-operator.html.