Configuring GlusterFS during IBM Cloud Private installation

Configure GlusterFS when you install your IBM® Cloud Private cluster.

Complete these steps to configure GlusterFS:

Configure a host group with nodes that you are using to configure GlusterFS. You can either use dedicated nodes for GlusterFS, or you can configure GlusterFS on nodes that are also used as worker nodes.
- If you are using dedicated nodes to configure GlusterFS, complete these steps. These nodes are not used as worker nodes.
  - Configure a custom host group with the dedicated GlusterFS storage nodes. For more information about how to add a host group, see Defining custom host groups.
    
    Following is an example configuration of a host group with dedicated GlusterFS storage nodes. You add this configuration in the /<installation_directory>/cluster/hosts file.
    
    Note: The worker nodes and the GlusterFS nodes are not the same.
```
[worker]
2.2.2.2
...
2.2.2.9
.
.
[hostgroup-glusterfs]
6.6.6.6
...
6.6.6.9
```
- If you are configuring GlusterFS on nodes that are also used as worker nodes, complete these steps.
  - In the /<installation_directory>/cluster/hosts file, add the same nodes as worker nodes and as a custom host group.
  - In the /<installation_directory>/cluster/config.yaml file, configure no_taint_group parameter with the custom host group name.
    Following is an example configuration of shared storage nodes. You add this configuration in the /<installation_directory>/cluster/hosts file.
```
[worker]
2.2.2.2
...
2.2.2.9
.
.
[hostgroup-glusterfs]
2.2.2.2
...
2.2.2.9
```
    Following is an example configuration of adding the no_taint_group parameter. You add this configuration in the /<installation_directory>/cluster/config.yaml file.
```
no_taint_group: ["hostgroup-glusterfs"]
```
If firewall is enabled, add the list of required ports to the /<installation_directory>/cluster/config.yaml file. Locate the section firewall_enabled: true. Add the following ports for the custom host group that you created with dedicated GlusterFS storage nodes.
Following is an example configuration of the custom host group hostgroup-glusterfs:
```
firewall_open_ports:
  hostgroup-glusterfs:
    - 24007/tcp
    - 24008/tcp
    - 2222/tcp
    - 49152-49251/tcp
```
Note: If you use existing worker nodes to deploy GlusterFS, you must manually open all the ports on all the nodes.
Enable GlusterFS Storage. Set storage-glusterfs: enabled under the management services list in the /<installation_directory>/cluster/config.yaml file.
```
management_services:
  istio: disabled
  vulnerability-advisor: disabled
  storage-glusterfs: enabled
  storage-minio: disabled
```
Add the following piece of code to the config.yaml file:

## GlusterFS Storage Settings
storage-glusterfs:
  nodes:
    - ip: <worker_node_m_IP_address>
      devices:
        - <link path>/<symlink of device aaa>
        - <link path>/<symlink of device bbb>
    - ip: <worker_node_n_IP_address>
      devices:
        - <link path>/<symlink of device ccc>
    - ip: <worker_node_o_IP_address>
      devices:
        - <link path>/<symlink of device ddd>
  storageClass:
    create: true
    name: glusterfs
    isDefault: false
    volumeType: replicate:3
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    volumeNamePrefix: icp
    additionalProvisionerParams: {}
    allowVolumeExpansion: true
  gluster:
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 1Gi
  heketi:
    backupDbSecret: heketi-db-backup
    authSecret: "heketi-secret"
    maxInFlightOperations: "20"
    dbSyncupDelay: "10"
    tls:
      generate: true
      issuer: "icp-ca-issuer"
      issuerKind: "ClusterIssuer"
      secretName: ""
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 1Gi
  nodeSelector:
    key: hostgroup
    value: glusterfs    
  prometheus:
    enabled: true
    path: "/metrics"
    port: 8080
  tolerations: []  
  podPriorityClass: "system-cluster-critical"

Following are the parameter descriptions. For a list of available parameters, see Configuration Opens in a new tab .
Note: If you are configuring the default storage class and volume type parameters, ensure that you use the correct syntax. Use isDefault: <true or false> and volumeType: <volume type>.

ip is the IP address of the node on which you are configuring GlusterFS.
devices is the full path to the symlink of the storage device.

Note: Do not add the device name. Add the Symlinks.
storageClass creates a storage class for GlusterFS.
- create is the option to configure a storage class. The default value is true. If you do not want to create a storage class, specify false.
- name is the storage class name. The default value is glusterfs. The storage class name must comply with the Kubernetes naming convention: up to 253 characters and only lowercase alphanumeric characters, -, and ..
- isDefault is the option to make the GlusterFS storage class as the default storage class. The default value is false. If you want this storage class to be the default, specify true.
- volumeType is a configuration parameter for the volume. You can set this value to replicate:3, which creates a replica volume of count 3. The valid parameter values are: none; replicate:<replicate_count>; disperse:<data>:<redundancy_count>. The default value is replicate:3.
  Note: If you are configuring GlusterFS on a single node, set volumeType: none.
- reclaimPolicy is the GlusterFS storage class reclaim policy. Default value is Delete.
- volumeBindingMode is the GlusterFS storage class volume binding mode. Default value is Immediate.
- volumeNamePrefix is the GlusterFS storage class volume name prefix. Default value is icp.
- additionalProvisionerParams are extra storage class provisioner parameters.
- allowVolumeExpansion is to define whether expansion of volume is allowed. Default value is true.
  
  Note: At any point in time later, you can create more storage classes to use GlusterFS storage. See Creating a storage class for GlusterFS.
gluster sets the parameters for GlusterFS configuration.
- resources defines the maximum and minimum CPU and memory that is required.
  - requests
    - cpu is the minimum amount of CPU required. Default value is 500 millicpu (m).
    - memory is the minimum amount of memory required. Default value is 512 Mi.
  - limits
    - cpu is the maximum amount of CPU allowed. Default value is 1000m.
    - memory is the maximum amount of memory allowed. Default value is 1Gi.
heketi sets the parameters for Heketi configuration.
- backupDbSecret is the Heketi database to be backed up to a kubernetes secret. Default value is heketi-db-backup.
- authSecret is the secret that has the encrypted password of the Heketi user 'admin'. Note: The secret is automatically created by using the <default_admin_password> that is added in the /<installation_directory>/cluster/config.yaml file.
- maxInFlightOperations is the maximum number of simultaneous requests for persistent volume. The default value is 20.
- dbSyncupDelay is the delay in seconds to sync up data in the Heketi database with backup secret. The default value is 10.
- tls sets transport layer security (TLS) configuration parameters.
  - generate is a Boolean flag. When set to true, the flag creates a certificate by using the IBM Cloud Private certificate authority (CA). If you do not set the flag, you must create a secret that contains a private key, TLS certificate, and a CA certificate. You must provide the secret name in the heketi.tls.secretName parameter.
  - issuer is the name of the IBM Cloud Private certificate issuer.
  - issuerKind is the kind of CA from which the signed x509 certificates is obtained. Valid values are ClusterIssuer and Issuer.
  - secretName is the certificate that you provide as secret object. The name of key must be tls.key and the name of the certificate must be tls.crt. You must set this parameter when you set the heketi.tls.generate flag to false.
- resources is the maximum and minimum CPU and memory that is required.
  - requests
    - cpu is the minimum amount of CPU required. Default value is 500m.
    - memory is the minimum amount of memory required. Default value is 512Mi.
  - limits
    - cpu is the maximum amount of CPU allowed. Default value is 1000m.
    - memory is the maximum amount of memory allowed. Default value is 1Gi.
prometheus sets the option to enable or disable prometheus monitoring.
Note: Prometheus provides a dashboard of statistics about health and usage of GlusterFS. If you want to use Prometheus for GlusterFS, you must enable it now. You cannot enable it after your IBM Cloud Private cluster is installed.
- enabled is the setting to enable or disable prometheus monitoring. Default value is true.
- path is the Heketi path to pull the metrics. Default value is /metrics.
- port is the port on which the Heketi service is exposed. Default value is 8080.
nodeSelector sets the node selector label for the storage resources. Use the name of the custom host group that you created for GlusterFS configuration. For example, the custom host group hostgroup-glusterfs creates nodes with the label hostgroup=glusterfs.
- key is the label key. Default value is hostgroup.
- value is the label value. Default value is glusterfs.
podPriorityClass is the priority class preference for the GlusterFS and Heketi pods.
tolerations is the list of tolerations that you want to specify whether the storage nodes have additional taints.
Configuration of GlusterFS is complete. Proceed with the IBM Cloud Private installation.