Scaling Watson Discovery

After you install the service, you can change settings such as the number of replicas of the microservices to use, or the replica size to increase or decrease the capacity of the application.

In a multitenant environment, you install the Watson Discovery service one time, and then you can deploy up to 10 separate instances of the service. The computing resources, such as CPU and memory, that are provisioned for the installation are shared by all of the deployed instances. For planning purposes, consider the total size of artifacts, such as collections and enrichments, from across all of the service instances, not the size per instance.

Unless you specify otherwise, when you install Watson Discovery, a production-ready deployment is created. It has at least two replicas of each pod to support production-scale workloads. However, if you have complicated projects or high volumes of use, you might want to increase the capacity of your deployment. Alternatively, if you have low usage, you might want to decrease the capacity of your deployment to reduce costs.

You can use the scaleConfig setting in the service custom resource (CR) to set the scaling configuration for a service. For more information, see Manually scaling resources for services.

Alternatively, you can change the configuration of a deployment by changing the YAML file that is associated with the deployment.

To edit the YAML file, you must have administrative privileges for the project where the deployment is hosted. To change the file, you use one of the following commands:
Edit command
Opens an editor where you can make changes to the YAML definition. The command has the following syntax:
oc edit WatsonDiscovery wd
Patch command
Patches an existing resource by merging the configuration details from a local YAML file into the existing configuration. The command has the following syntax:
oc patch WatsonDiscovery wd --patch "$(cat $path-to-file)" --type='merge'

Scaling the Elasticsearch cluster

To scale Elasticsearch, complete the following steps:
  1. Verify the status of the Elasticsearch cluster.
  2. Update the index configuration settings to scale up the Elasticsearch cluster.
  3. Restore the index replica counts to their original values after scaling up the Elasticsearch cluster.

Verify the status of the Elasticsearch cluster

Before you scale up the Elasticsearch cluster, make sure that the cluster status is green.

To verify the status of the Elasticsearch cluster, complete the following steps:
  1. Run the following command to view the cluster status.
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl --fail -s -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} "${ELASTIC_ENDPOINT}/_cluster/health" | jq ".status" || true'
"green"

    If the cluster health status is green, continue with updating the index configuration settings to scale up the Elasticsearch cluster.

  2. If the cluster health status is yellow, run the following command to determine the indices that have yellow status and note their rep (replica) count.
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl --fail -s -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} "${ELASTIC_ENDPOINT}/_cat/indices?v=true&s=rep" || true'
health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   example-index-3 CHRLwzfXSV6zNLadcw8IPw   2   0          0            0       416b           416b
yellow open   example-index-1 yzgSyHSgS5G3zLzf-Z30-w   2   1          0            0       416b           416b
yellow open   example-index-2 _f0PAHYfTfuy2K_OLYv-dg   2   1          0            0       416b           416b

    In this example, the indices example-index-1 and example-index-2 have yellow health status, and both are configured to have one replica.

  3. Determine the number of Elasticsearch data nodes in the cluster. The replica count (rep + 1) must not exceed the number of data nodes in the cluster. Run the following command to check the data node count.
    > oc get pods -l run=elastic,role=data                           

    NAME                                    READY   STATUS    RESTARTS   AGE
wd-ibm-elasticsearch-es-server-data-0   2/2     Running   0          19h

    In this example, the replica counts for indices example-index-1 and example-index-2 exceed the number of data nodes minus one, so they have no available data nodes to route.

  4. Set the replica count on the over-replicated indices to the number of data nodes minus one.
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl -XPUT -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} -H "Content-Type: application/json" "${ELASTIC_ENDPOINT}/example-index-1,example-index-2/_settings" -d "{\"index.number_of_replicas\":0}" || true'
{"acknowledged":true}

    The replica count is reduced to match the available data nodes.

  5. Verify whether the cluster status is now green.
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl --fail -s -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} "${ELASTIC_ENDPOINT}/_cluster/health" | jq ".status" || true'
"green"

    If the cluster status is green, Update the index configuration settings to scale up the Elasticsearch cluster.

    If the cluster status is not green, verify that no errors are returned when you run the following command.

    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl -XPUT -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} -H "Content-Type: application/json" "${ELASTIC_ENDPOINT}/example-index-1,example-index-2/_settings" -d "{\"index.number_of_replicas\":0}" || true'
{"acknowledged":true}

Update the index configuration settings to scale up the Elasticsearch cluster

To update the index configuration, complete the following steps:
  1. Verify whether the .ltrstore index exists.
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl --fail -s -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} "${ELASTIC_ENDPOINT}/_cat/indices/.*" | grep ".ltrstore" || true'

green open .ltrstore Lbf1tZ9wSoyhdHMzEqtuHg 1 1 6 0 60.4kb 30.2kb

    A response such as the following is returned if the .ltrstore index exists.

    oc rsh -n zen -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} "${ELASTIC_ENDPOINT}/.ltrstore/_settings"' | jq 
{
  ".ltrstore": {
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "hidden": "true",
        "number_of_shards": "1",
        "auto_expand_replicas": "0-2",
 ...
    If this command does not return the index information, you do not need to update the index settings. If this command returns the index information, run the following command to update the index settings.
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl -XPUT -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} -H "Content-Type: application/json" "${ELASTIC_ENDPOINT}/.ltrstore/_settings" -d"{\"index\": {\"auto_expand_replicas\": \"false\"}}"'

{"acknowledged":true}
  2. Verify whether the index is configured correctly. 
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl --fail -s -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} "${ELASTIC_ENDPOINT}/.ltrstore/_settings" | jq ".[\".ltrstore\"].settings.index.auto_expand_replicas"'

"false"
    This command should return false. If this command does not return false, run the command to update the index settings again.
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl -XPUT -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} -H "Content-Type: application/json" "${ELASTIC_ENDPOINT}/.ltrstore/_settings" -d"{\"index\": {\"auto_expand_replicas\": \"false\"}}"'

{"acknowledged":true}
  3. Follow the Manually scaling resources for services procedure for Watson Discovery. Alternatively, you can scale Watson Discovery by editing the custom resource. For more information, see Scaling the number of replicas.
  4. Ensure that all Elasticsearch pods are in the 2/2 Running state.
    > oc get pod | grep wd-ibm-elasticsearch

wd-ibm-elasticsearch-es-server-client-0                    2/2     Running     0               7m10s
wd-ibm-elasticsearch-es-server-client-1                    2/2     Running     0               8m13s
wd-ibm-elasticsearch-es-server-data-0                      2/2     Running     0               7m17s
wd-ibm-elasticsearch-es-server-data-1                      2/2     Running     0               8m15s
wd-ibm-elasticsearch-es-server-master-0                    2/2     Running     0               7m43s
  5. Restore the configuration of .ltrstore.
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl -XPUT -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} -H "Content-Type: application/json" "${ELASTIC_ENDPOINT}/.ltrstore/_settings" -d"{\"index\": {\"auto_expand_replicas\": \"0-2\"}}"'

{"acknowledged":true}

    If this command returns {"acknowledged":true}, your Elasticsearch cluster is now configured for horizontal scaling.

  6. Verify whether the index is correctly configured. This command should return 0-2.
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl --fail -s -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} "${ELASTIC_ENDPOINT}/.ltrstore/_settings" | jq ".[\".ltrstore\"].settings.index.auto_expand_replicas"'

"0-2"
    If this command does not return 0-2, then try to restore the configuration by running the previous command again.
    > oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl -XPUT -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} -H "Content-Type: application/json" "${ELASTIC_ENDPOINT}/.ltrstore/_settings" -d"{\"index\": {\"auto_expand_replicas\": \"0-2\"}}"'

{"acknowledged":true}

Restoring the index replica counts to their original values

Restore the index replica counts to their original values after successfully updating the index configuration settings to scale up the Elasticsearch cluster..

If any replica counts were changed, run the following command to restore them to their original values.
> oc rsh -c elasticsearch wd-ibm-elasticsearch-es-server-master-0 bash -c 'curl -XPUT -k -u ${ELASTIC_USER}:${ELASTIC_PASSWORD} -H "Content-Type: application/json" "${ELASTIC_ENDPOINT}/example-index-1,example-index-2/_settings" -d "{\"index.number_of_replicas\":1}" || true'
{"acknowledged":true}

Scaling the number of replicas

To change the number of replicas that are used by a microservice, edit the custom resource. The operator applies the change to the current deployment for you.

  1. Create a YAML file that specifies the number of replicas for the target pods.
    For example, to increase the number of API Gateway pods to 3 , create a file that is named patch.yaml with the following content:
    spec:
  api:
    replicas: 3
  2. Apply the patch to the custom resource by using the patch command. For example:
    oc patch WatsonDiscovery wd --type=merge --patch "$(cat patch.yaml)"

Sample YAML file

The following YAML file sample shows values that you can change.
spec:
  tooling:
    minerapp: # Content mining application
      replicas: 2
    tooling: # Discovery product user interface
      replicas: 2
  api:  # API gateway pod
    replicas: 2
  hdp:  # Hadoop worker
    worker:
      replicas: 3
  ingestion:  # Crawler pod
    crawler:
      replicas: 2
  enrichment:  # Enrichment service pod
    service:
      replicas: 4
  statelessapi:  # Analyze API
    proxy:
      replicas: 2

Scaling out the number of enrichment-service pods

Note: Before you scale out, make sure that the cluster has enough CPU and memory resources so that the pods can start without any issues.
Criteria to scale-out
Following are the criteria to consider for scaling out:
  • You want to increase the number of enrichments that can be run simultaneously.
  • You want to increase the number of documents that one enrichment can process in parallel.
How many pods to increase

In the default configuration, one enrichment-service pod can serve 2 to 4 models. It is an approximate estimation and the actual resource consumption depends on the enrichment model size and type. You can increase the number of replicas based on the models that are used to process the enrichment concurrently.

How to scale the number of replicas
Add the following patch to the YAML file.
spec:
  enrichment:  # Enrichment Service pod
    service:
      replicas: 4

Scaling an existing persistent volume claim size

A persistent volume claim (PVC) is a request for a piece of storage (a persistent volume) in a cluster. To change the resources that are used by a Discovery microservice, you must edit the claim. After you change the details of your request, the cluster manages how to make the increased or reduced amount of resources available to the service.

To scale your deployment of Watson Discovery, complete the following steps:
  1. From the system where the service is running, edit the YAML file for the resource that you want to change.
    1. Use the following command to list the persistent volume claims that are in use by Watson Discovery:
      oc get pvc --selector 'app.kubernetes.io/name in (wd, discovery)'
    2. Look for the PVC name for the resource that you want to change, and then use the following command to open its associated YAML file for editing: 
      oc edit pvc {PVC NAME FROM PREVIOUS STEP}
      To edit the configuration of the Elastic search resource, the command might look like this, for example:
      oc edit pvc data-wd-ibm-elasticsearch-es-server-client-0
    3. Edit the YAML file for the PVC pod.
      The following settings only can be changed for a persistent volume claim (PVC) after the PVC is created. 
        elasticsearch:
    clientNode:
      persistence:
        size: 1Gi     # PVC size for Elasticsearch client node. Cannot be changed in custom resource.
    dataNode:
      persistence:
        size: 40Gi     # PVC size for Elasticsearch data node. Cannot be changed in custom resource.
    masterNode:
      persistence:
        size: 2Gi     # PVC size for Elasticsearch coordinator node. Cannot be changed in custom resource.
  etcd:
    storageSize: 10Gi     # PVC size for etcd data store. Cannot be changed in custom resource.
  minio:
    persistence:
      size: 100Gi     # PVC size for MinIO. Cannot be changed in custom resource.
  postgres:
    database:
      storageRequest: 30Gi     # PVC size for the PostgreSQL. Cannot be changed in custom resource.
  rabbitmq:
    persistentVolume:
      size: 5Gi     # PVC size for the RabbitMQ data store. Cannot be changed in custom resource.
      In the following example, the storage size for the Elastic search client node is changed from 1 Gi to 2 Gi:
      spec:
  ...
  elasticsearch:
    clientNode:
      persistence:
        size: 2Gi
    4. After you edit the YAML file for the PVC, save and close it.

    Give the cluster a few minutes to pick up and apply the changes.

  2. To verify that the changes were applied, run the following command: 
    oc get pvc {PVC NAME}
  3. If the wd-ibm-elasticsearch-es-server-snap PVC is expanded, the following patch must be run. The size parameter must match the current PVC size.
    oc patch wd wd --type=merge --patch '{"spec": {"elasticsearch": {"snapshot": {"size": "200Gi"}}}}'
Important: You cannot make this change persist. For example, you cannot update the custom resource for the service with these changes to apply them automatically. Instead, you must repeat the previous steps each time a persistent volume claim is added as a result of increasing the number of replicas, for example.

Preparing for system maintenance

If you want to scale down the service entirely so that you can perform system maintenance, for example, you can do so.

  1. Back up the system data so that you can restore the service if you run into any problems. For more information, see Backing up and restoring data.
  2. Stop the service.
    oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"shutdown": "true"}}'
    All of the pods that are associated with the service are quiesced, except for the PostgreSQL pods. The PostgreSQL service does not support a quiesce operation. Instead, these pods are set into a maintenance mode called fencing where data is protected while the system is maintained. In fencing mode, the pods continue to run in a non-ready state.
  3. To enable Red Hat® OpenShift® Container Platform to drain the worker nodes fully, the PostgreSQL pods must be stopped. Use the following command to stop and drain the nodes:
    oc adm drain
  4. Complete your maintenance tasks.
  5. After system maintenance is finished, restart the service.
    oc patch WatsonDiscovery wd --type=merge \
--patch='{"spec":{"shutdown": "false"}}'