Scaling Watson Discovery

After you install the service, you can change settings such as the number of replicas of the microservices to use, or the replica size to increase or decrease the capacity of the application.

In a multitenant environment, you install the Watson Discovery service one time, and then you can deploy up to 10 separate instances of the service. The computing resources, such as CPU and memory, that are provisioned for the installation are shared by all of the deployed instances. For planning purposes, consider the total size of artifacts, such as collections and enrichments, from across all of the service instances, not the size per instance.

Unless you specify otherwise, when you install Watson Discovery, a production-ready deployment is created. It has at least two replicas of each pod to support production-scale workloads. However, if you have complicated projects or high volumes of use, you might want to increase the capacity of your deployment. Alternatively, if you have low usage, you might want to decrease the capacity of your deployment to reduce costs.

To scale a deployment, you must have administrative privileges for the project where the deployment is hosted.

You can change the configuration of a deployment by changing the YAML file that is associated with the deployment. To change the file, you use one of the following commands:

Edit command

Opens an editor where you can make changes to the YAML definition. The command has the following syntax:

oc edit WatsonDiscovery wd

Patch command

Patches an existing resource by merging the configuration details from a local YAML file into the existing configuration. The command has the following syntax:

oc patch WatsonDiscovery wd --patch "$(cat $path-to-file)" --type='merge'

Scaling the number of replicas

To change the number of replicas that are used by a microservice, edit the custom resource. The operator applies the change to the current deployment for you.

Create a YAML file that specifies the number of replicas for the target pods.
For example, to increase the number of API Gateway pods to 3 , create a file that is named patch.yaml with the following content:
```
spec:
  api:
    replicas: 3
```
Apply the patch to the custom resource by using the patch command. For example:
```
oc patch WatsonDiscovery wd --type=merge --patch "$(cat patch.yaml)"
```

Sample YAML file

The following YAML file sample shows values that you can change.

spec:
  tooling:
    minerapp: # Content mining application
      replicas: 2
    tooling: # Discovery product user interface
      replicas: 2
  api:  # API gateway pod
    replicas: 2
  hdp:  # Hadoop worker
    worker:
      replicas: 3
  ingestion:  # Crawler pod
    crawler:
      replicas: 2
  statelessapi:  # Analyze API
    runtime:
      replicas: 2
  wksml:  # Machine learning model enrichment
    replicas: 2

Scaling an existing persistent volume claim size

A persistent volume claim (PVC) is a request for a piece of storage (a persistent volume) in a cluster. To change the resources that are used by a Discovery microservice, you must edit the claim. After you change the details of your request, the cluster manages how to make the increased or reduced amount of resources available to the service.

To scale your deployment of Watson Discovery, complete the following steps:

From the system where the service is running, edit the YAML file for the resource that you want to change.

Use the following command to list the persistent volume claims that are in use by Watson Discovery:
```
oc get pvc --selector 'app.kubernetes.io/name in (wd, discovery)'
```
Look for the PVC name for the resource that you want to change, and then use the following command to open its associated YAML file for editing:
```
oc edit pvc {PVC NAME FROM PREVIOUS STEP}
```
To edit the configuration of the Elastic search resource, the command might look like this, for example:
```
oc edit pvc data-wd-ibm-elasticsearch-es-server-client-0
```

Edit the YAML file for the PVC pod.

The following settings only can be changed for a persistent volume claim (PVC) after the PVC is created.

  elasticsearch:
    clientNode:
      persistence:
        size: 1Gi     # PVC size for the Elasticsearch client node. Cannot be changed in custom resource.
    dataNode:
      persistence:
        size: 40Gi     # PVC size for the Elasticsearch data node. Cannot be changed in custom resource.
    masterNode:
      persistence:
        size: 2Gi     # PVC size for the Elasticsearch coordinator node. Cannot be changed in custom resource.
  etcd:
    storageSize: 10Gi     # PVC size for etcd data store. Cannot be changed in custom resource.
  minio:
    persistence:
      size: 100Gi     # PVC size for the MinIO data store. Cannot be changed in custom resource.
  postgres:
    database:
      storageRequest: 30Gi     # PVC size for the PostgreSQL database. Cannot be changed in custom resource.
  rabbitmq:
    persistentVolume:
      size: 5Gi     # PVC size for the RabbitMQ data store. Cannot be changed in custom resource.

In the following example, the storage size for the Elastic search client node is changed from 1 Gi to 2 Gi:

spec:
  ...
  elasticsearch:
    clientNode:
      persistence:
        size: 2Gi

After you edit the YAML file for the PVC, save and close it.

Give the cluster a few minutes to pick up and apply the changes.

To verify that the changes were applied, run the following command:
```
oc get pvc {PVC NAME}
```

Important: You cannot make this change persist. For example, you cannot update the custom resource for the service with these changes to apply them automatically. Instead, you must repeat the previous steps each time a persistent volume claim is added as a result of increasing the number of replicas, for example.

Preparing for system maintenance

If you want to scale down the service entirely so that you can perform system maintenance, for example, you can do so.

Back up the system data so that you can restore the service if you run into any problems. For more information, see Backing up and restoring data.

Quiesce the service components.

oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"shared":{"quiesce":{"enabled": true}}}}'

Wait for service components to be fully quiesced, and then quiesce the data stores components in the next step.

Quiesce the data stores.

oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"postgres":{"quiesce":{"enabled": true}}}}'
oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"minio":{"quiesce":{"enabled": true}}}}'
oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"elasticsearch":{"quiesce":{"enabled": true}}}}'
oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"etcd":{"quiesce":{"enabled": true}}}}'
oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"rabbitmq":{"quiesce":{"enabled": true}}}}'

Complete your maintenance tasks.

After system maintenance is finished, disable quiesce for the data stores.

oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"postgres":{"quiesce":{"enabled": false}}}}'
oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"minio":{"quiesce":{"enabled": false}}}}'
oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"elasticsearch":{"quiesce":{"enabled": false}}}}'
oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"etcd":{"quiesce":{"enabled": false}}}}'
oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"rabbitmq":{"quiesce":{"enabled": false}}}}'

Disable quiesce for the service components.

oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"shared":{"quiesce":{"enabled": false}}}}'