Scaling IBM MDM Publisher

IBM® MDM Publisher supports vertical or horizontal scaling to increase performance and allow it to load larger data sets.

Horizontal scaling

MDM Publisher supports horizontal scaling by using a scalable Spark pod with a single mdm-publisher container.

To scale the MDM Publisher bulk load operation up or down, depending on the dataset size you are trying to process, modify the number of executors that MDM Publisher will spawn, or modify their corresponding CPU and memory parameters. For more information about changing the number of executors, see the publish-config.xml property descriptions in Configuring IBM MDM Publisher.

Vertical scaling

MDM Publisher also supports vertical scaling. To increase MDM Publisher performance and enable loading of larger data sets, you can add more resources to the Kubernetes nodes and increase the resource allotment for your MDM Publisher deployment.

The resource map YAML files include properties that control many different aspects of MDM Publisher behavior. There is one main resource map YAML file for each deployment type:

For Red Hat OpenShift deployment configuration: ${INSTALL_LOC}/mdm-publisher/ibm-publisher-services-prod/values-openshift.yaml
For Kubernetes deployment configuration: ${INSTALL_LOC}/mdm-publisher/ibm-publisher-services-prod/values-k8s.yaml
For Minikube deployment configuration: ${INSTALL_LOC}/mdm-publisher/ibm-publisher-services-prod/values-minikube.yaml

To increase the resource allotment:

Edit the YAML file that corresponds to your deployment type.

Modify the following configuration section:

publisher:
  image: mdm-publisher
  imageTag: 0.1.623
  productTag: 0.1.623
  imagePullPolicy: IfNotPresent
  deployment:
    resources:
      requests:
        cpu: 4
        memory: 8Gi
    limits:
        cpu: 4
        memory: 8Gi

After completing your changes and saving the file, run the following command to apply the changes to your MDM Publisher instance:
```
${INSTALL_LOC}/mdm-publisher/bin/update_configuration.sh
```
When prompted with the message As part of this MDM Publisher configuration update, are you also updating endpoints? (Yy/Nn), reply N. This ensures that only the properties files are updated.
Wait for the mdm-publisher-0 pod to re-enter a READY state before trying to access it again.