Backup & restore hub performance and scaling

Use this topic to understand the number of spokes you can scale up to and the number of spoke clusters that can be connected to a single hub. It outlines the capacities and the scalability considerations to ensure efficient backup and restore operations.

The Backup & restore hub service is efficiently designed to handle large-scale backup and restore concurrent jobs across clusters. With proven scalability and performance. The system is tested successfully to handle up to 1000 concurrent jobs. It serves as a reference point and not a limitation; the hub can scale further with appropriate resource allocation.

Sizing blueprints for varying workloads

The following table provides the recommended configurations for each required service when handling different numbers of concurrent jobs. As the number of concurrent jobs increases, you can scale out (increase the replicas) or scale up (increase the resource limits) for the required services as outlined:

Note:
  • Services not listed in the blueprint table do not require scaling. These services can handle the load without modification because they do not have scaling requirements for up to 1000 concurrent jobs.
  • Some services, such as mongodb-v2 or operator-control-manager pods in the ibm-backup-restore namespace, cannot be scaled out by increasing replicas. For these services, increase the CPU and memory resources to ensure optimal performance as the number of concurrent jobs increases.
  • The guardian-bridge service automatically scales to handle the additional load and provides backward compatibility when connected spokes are at a version earlier than IBM Fusion 2.13.
Table 1. Sizing blueprints
Clusters Concurrent Jobs Mongodb-v2 Kafka broker pool Backup service Backup policy service Application service Job manager service Window end time (hr) CancelJobAfter (ms) Max concurrent reconciles
Replicas CPU Memory Ephemeral Replicas Replicas Replicas Replicas Replicas
10 100 3 1 1 GiB 0.5 GiB 3 1 1 1 1 4 3600000 10
20 200 3 2 1 GiB 0.5 GiB 3 1 1 1 1 4 3600000 10
30 300 3 3 1.5 GiB 1 GiB 6 1 1 1 1 5 3600000 10
40 400 3 3.5 1.5 GiB 1 GiB 6 1 1 1 1 5 3600000 15
50 500 3 4 2 GiB 1.5 GiB 9 1 1 1 1 6 3600000 15
60 600 3 4.5 2 GiB 1.5 GiB 9 1 1 1 1 6 7200000 15
70 700 3 5 2 GiB 1.5 GiB 12 1 1 1 1 7 7200000 15
80 800 3 6 2.5 GiB 1.75 GiB 12 2 1 1 2 8 10800000 20
90 900 3 7 2.5 GiB 1.75 GiB 15 2 2 2 2 9 14400000 20
100 1000 3 8 3 GiB 2 GiB 15 3 2 2 2 10 18000000 20

Scaling Services

Use the following commands to scale out the desired replicas or scale up resource limits for the required services:
mongodb-v2 - scale up resource limits
The mongodb-v2 StatefulSet cannot be scaled out. Increase CPU, memory, and ephemeral storage limits as the workload grows:
oc set resources sts mongodb-v2 --limits=cpu=<desired-cpu>,memory=<desired-memory>,ephemeral-storage=<desired-ephemeral> --containers=mongodb -n <backup-restore-namespace>
For example:
oc set resources sts mongodb-v2 --limits=cpu=8,memory=3Gi,ephemeral-storage=2Gi --containers=mongodb -n ibm-backup-restore
Kafka broker-pool - scale out replicas
The Kafka broker-pool is managed as a KafkaNodePool resource. Scale its replicas using:
oc patch kafkanodepool broker-pool -n <backup-restore-namespace> --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <desired-replicas>}]'
For example:
oc patch kafkanodepool broker-pool -n ibm-backup-restore --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": 15}]'
backup-service - scale out replicas
oc scale deployment backup-service --replicas=<desired-replicas> -n <backup-restore-namespace>
For example with 100 clusters/1000 jobs (3 replicas):
oc scale deployment backup-service --replicas=3 -n ibm-backup-restore
backuppolicy-deployment - scale out replicas
oc scale deployment backuppolicy-deployment --replicas=<desired-replicas> -n <backup-restore-namespace>
For example with 90-100 clusters (2 replicas):
oc scale deployment backuppolicy-deployment --replicas=2 -n ibm-backup-restore
applicationsvc - scale out replicas
oc scale deployment applicationsvc --replicas=<desired-replicas> -n <backup-restore-namespace>
For example with 80-100 clusters (2 replicas):
oc scale deployment applicationsvc --replicas=2 -n ibm-backup-restore
job-manager - scale out replicas
oc scale deployment job-manager --replicas=<desired-replicas> -n <backup-restore-namespace>
For example with 80-100 clusters (2 replicas):
oc scale deployment job-manager --replicas=2 -n ibm-backup-restore
windowEndTime - update through BackupPolicy CR
The windowEndTime is a field under spec.schedule in the BackupPolicy CR. The value is in HH:MM format.
Note: windowEndTime must be set to a minimum of 4 hours later than the scheduled (cron) start time. This can also be updated from the IBM Fusion user interface by navigating to Backup & Restore > Policies and by editing the required policy.
Update the windowEndTime field (HH:MM format) under spec.schedule. Ensure the value is at least 4 hours after the policy cron schedule time:
oc patch fbp <policy-name> -n <ibm-spectrum-fusion-ns> --type='json' -p='[{"op":"replace","path":"/spec/schedule/windowEndTime","value":"<HH:MM>"}]'
For example with 100 clusters/1000 jobs (window ending at 10:00):
oc patch fbp <policy-name> -n <ibm-spectrum-fusion-ns> --type='json' -p='[{"op":"replace","path":"/spec/schedule/windowEndTime","value":"10:00"}]'
cancelJobAfter - update in job-manager deployment
The cancelJobAfter value is specified in milliseconds and is configured as an environment variable in the job-manager deployment.
Patch the env variable value (index 3 corresponds to cancelJobAfter):
oc patch deployment job-manager -n <backup-restore-namespace> --type='json' -p='[{"op":"replace","path":"/spec/template/spec/containers/0/env/3/value","value":"<milliseconds>"}]'
For example with 80 clusters/800 jobs (10800000 ms = 3 hours):
oc patch deployment job-manager -n ibm-backup-restore --type='json' -p='[{"op":"replace","path":"/spec/template/spec/containers/0/env/3/value","value":"10800000"}]
MaxConcurrentReconciles - update in dp-operator CSV
MaxConcurrentReconciles is configured as an environment variable in the guardian-dp-operator cluster service version.
Patch the value (index 3 corresponds to MaxConcurrentReconciles based on the current CSV):
oc patch csv <guardian-dp-operator-csv-name> -n <backup-restore-namespace> --type='json' -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/1/env/3/value", "value": "<desired-value>"}]'
For example with 80-100 clusters (20 reconciles):
oc patch csv guardian-dp-operator.v2.13.0-20260402145657 -n ibm-backup-restore --type='json' -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/1/env/3/value", "value": "20"}]'