Scaling Watson Discovery
After you install the service, you can change settings such as the number of replicas of the microservices to use, or the replica size to increase or decrease the capacity of the application.
In a multitenant environment, you install the Watson Discovery service one time, and then you can deploy up to 10 separate instances of the service. The computing resources, such as CPU and memory, that are provisioned for the installation are shared by all of the deployed instances. For planning purposes, consider the total size of artifacts, such as collections and enrichments, from across all of the service instances, not the size per instance.
Unless you specify otherwise, when you install Watson Discovery, a production-ready deployment is created. It has at least two replicas of each pod to support production-scale workloads. However, if you have complicated projects or high volumes of use, you might want to increase the capacity of your deployment. Alternatively, if you have low usage, you might want to decrease the capacity of your deployment to reduce costs.
You can use the scaleConfig
setting in the service custom resource (CR) to set
the scaling configuration for a service. For more information, see Manually scaling resources for
services
Alternatively, you can change the configuration of a deployment by changing the YAML file that is associated with the deployment.
- Edit command
- Opens an editor where you can make changes to the YAML definition. The command has the following
syntax:
oc edit WatsonDiscovery wd
- Patch command
- Patches an existing resource by merging the configuration details from a local YAML file into
the existing configuration. The command has the following
syntax:
oc patch WatsonDiscovery wd --patch "$(cat $path-to-file)" --type='merge'
Scaling the number of replicas
- Create a YAML file that specifies the number of replicas for the target pods.For example, to increase the number of API Gateway pods to
3
, create a file that is namedpatch.yaml
with the following content:spec: api: replicas: 3
- Apply the patch to the custom resource by using the patch command. For
example:
oc patch WatsonDiscovery wd --type=merge --patch "$(cat patch.yaml)"
Sample YAML file
spec:
tooling:
minerapp: # Content mining application
replicas: 2
tooling: # Discovery product user interface
replicas: 2
api: # API gateway pod
replicas: 2
hdp: # Hadoop worker
worker:
replicas: 3
ingestion: # Crawler pod
crawler:
replicas: 2
statelessapi: # Analyze API
runtime:
replicas: 2
wksml: # Machine learning model enrichment
replicas: 2
Scaling an existing persistent volume claim size
A persistent volume claim (PVC) is a request for a piece of storage (a persistent volume) in a cluster. To change the resources that are used by a Discovery microservice, you must edit the claim. After you change the details of your request, the cluster manages how to make the increased or reduced amount of resources available to the service.
- From the system where the service is running, edit the YAML file for the resource that you want
to change.
- Use the following command to list the persistent volume claims that are in use by Watson
Discovery:
oc get pvc --selector 'app.kubernetes.io/name in (wd, discovery)'
- Look for the PVC name for the resource that you want to change, and then use the following
command to open its associated YAML file for editing:
To edit the configuration of the Elastic search resource, the command might look like this, for example:oc edit pvc {PVC NAME FROM PREVIOUS STEP}
oc edit pvc data-wd-ibm-elasticsearch-es-server-client-0
- Edit the YAML file for the PVC pod. The following settings only can be changed for a persistent volume claim (PVC) after the PVC is created.
elasticsearch: clientNode: persistence: size: 1Gi # PVC size for Elasticsearch client node. Cannot be changed in custom resource. dataNode: persistence: size: 40Gi # PVC size for Elasticsearch data node. Cannot be changed in custom resource. masterNode: persistence: size: 2Gi # PVC size for Elasticsearch coordinator node. Cannot be changed in custom resource. etcd: storageSize: 10Gi # PVC size for etcd data store. Cannot be changed in custom resource. minio: persistence: size: 100Gi # PVC size for MinIO. Cannot be changed in custom resource. postgres: database: storageRequest: 30Gi # PVC size for the PostgreSQL. Cannot be changed in custom resource. rabbitmq: persistentVolume: size: 5Gi # PVC size for the RabbitMQ data store. Cannot be changed in custom resource.
In the following example, the storage size for the Elastic search client node is changed from 1 Gi to 2 Gi:spec: ... elasticsearch: clientNode: persistence: size: 2Gi
- After you edit the YAML file for the PVC, save and close it.
Give the cluster a few minutes to pick up and apply the changes.
- Use the following command to list the persistent volume claims that are in use by Watson
Discovery:
- To verify that the changes were applied, run the following command:
oc get pvc {PVC NAME}
Preparing for system maintenance
If you want to scale down the service entirely so that you can perform system maintenance, for example, you can do so.
- Back up the system data so that you can restore the service if you run into any problems. For more information, see Backing up and restoring data.
- Stop the
service.
All of the pods that are associated with the service are quiesced, except for the PostgreSQL pods. The PostgreSQL service does not support a quiesce operation. Instead, these pods are set into a maintenance mode called fencing where data is protected while the system is maintained. In fencing mode, the pods continue to run in a non-ready state.oc patch WatsonDiscovery wd --type=merge \ --patch='{"spec":{"shutdown": "true"}}'
- To enable Red Hat® OpenShift® Container Platform to drain the
worker nodes fully, the PostgreSQL pods must
be stopped. Use the following command to stop and drain the
nodes:
oc adm drain
- Complete your maintenance tasks.
- After system maintenance is finished, restart the
service.
oc patch WatsonDiscovery wd --type=merge \ --patch='{"spec":{"shutdown": "false"}}'