Workload scale customization

Scale your workload to use more or less resources. Scalable resources include memory, CPU, and replicas.

From Maximo® Visual Inspection version 8.9, you can use workload scale customization.

Customization processes

HorizontalPodAutoscaler (HPA)
Defined and enabled for the following components:
  • vision-service-dataset
  • vision-dataset-feature
  • vision-model-conversion
Custom ConfigMap

Enabled with manual editing.

Note:

If you are upgrading Maximo Visual Inspection to version 9.0+, back up the Custom ConfigMaps that you modified. Complete the backup in the Red Hat® OpenShift® web console by selecting the Custom ConfigMaps that have the following naming convention: custom-name.

podTemplates
Enabled by adding the associated component template in the corresponding visualinspectionapp custom resource.

Affinity customization

Configure affinity customization by using the podTemplates specification.

Add the associated component template in the corresponding visualinspectionapp costume resource by refining the following affinity rules:

  • nodeAffinity
  • podAffinity
  • podAntiAffinity
  • tolerations

Example podTemplates specifications

The following parameters are an example of the podTemplates specification with the node affinity rule for the vision-ui component:


spec:
  podTemplates:
    - name: vision-ui
      # Schedule vision-ui pod to run on node matching following expression
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
                matchExpressions:
                  - key: runtimeType
                    operator: In
                    values:
                      - frontend

Storage guidelines

Table 1. Storage guidelines
Configuration Workload type Storage partition PVC ObjectStorage Notes
small
  • Training workloads
  • Approximately 50 models
  • Approximately 8,000 images

Models: 25%

Data: 75% training or inspection

100 GB Disabled Enable data retention policy
medium
  • Training and continuous workloads
  • Approximately 250 models
  • Approximately 90,000 training images
PVC
  • Models: 25%
  • Data: 75% training
ObjectStorage
  • Data: 100% inspection
500 GB Enabled Enable data retention policy
large
  • Training and continuous inspection workloads
  • Approximately 500 models
  • Approximately 190,000+ training images
PVC
  • Models: 25%
  • Data: 75% training
ObjectStorage
  • Data: 100% inspection
1 TB Enabled Enable data retention policy

Components and supported workload customizations

The following table lists the components and their supported customizations:

Component Supported customization Notes
vision-dnn-service
  • Resources
  • Affinity
  • Tolerations
Not applicable
vision-edge-dle
  • Custom ConfigMap
  • Resources
  • Affinity
  • Tolerations
Not applicable
vision-preprocessing
  • Affinity
  • Tolerations
Not applicable
vision-service
  • Custom ConfigMap
  • Resources
  • Affinity
  • Tolerations
Not applicable
vision-service
  • Custom ConfigMap
  • Resources
  • Affinity
  • Tolerations
Not applicable
vision-service-dataset
  • Custom ConfigMap
  • Resources
  • Affinity
  • Tolerations
  • HPA managed
Not applicable
vision-ui
  • Custom ConfigMap
  • Resources
  • Affinity
  • Tolerations
Not applicable
vision-edgeman
  • Custom ConfigMap
  • Resources
  • Affinity
  • Tolerations
Not applicable
vision-video-microservice
  • Resources
  • Affinity
  • Tolerations
Not applicable
vision-dataset-feature
  • Custom ConfigMap
  • Resources
  • Affinity
  • Tolerations
  • HPA
Not applicable
vision-dataset-summarization
  • Resources
  • Affinity
  • Tolerations
Not applicable
vision-model-conversion
  • Resources
  • Affinity
  • Tolerations
  • HPA
Not applicable

The following components do not need scaling and affinity:

  • vision-user agent, small workload
  • vision-taskanaly, small workload
  • vision-usage-hourly, cronjob

vision-dnn-service

The following parameters are used with the vision-dnn-service component customization:

Workload
Used when training models.
Resources
By default, there is no request or limits.
Example podTemplates specifications
Use the following custom/sample_affinity_training.yaml of node affinity for picking a specific GPU type for training:


spec:
  podTemplates:
    - name: vision-dnn-service
      # Schedule training workloads to run on node matching following expression
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
                matchExpressions:
                  - key: nvidia.com/gpu.product
                    operator: In
                    values:
                      - NVIDIA-H100-NVL

vision-edge-dle

The following parameters are used with the vision-edge-dle component customization:

Workload
Used when deploying models.
Custom ConfigMap files
custom-dle-config
Resources
By default, there is no request or limits.
Example podTemplates specifications
Use the following custom/sample_affinity_inference.yaml of node affinity for picking a specific GPU type for training:


spec:
  podTemplates:
    - name: vision-edge-dle
      # Schedule inference workloads to run on node matching following expression
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
                matchExpressions:
                  - key: nvidia.com/gpu.product
                    operator: In
                    values:
                      - NVIDIA-L40S

vision-preprocessing

The following parameters are used with the vision-preprocessing component customization:

Workload
Used when augmenting data sets.
Resources
By default, there is no request or limits.

vision-service

The following parameters are used with the vision-service component customization:

Workload
Handles the workloads of data set and inspection uploads.
Custom ConfigMap files
custom-service-config, custom-service-jvm-config
Resources
Parameters for the template type.

The following table lists the customization for the vision-service

Table 2. vision-service customization
Configuration Workload Resources
small
  • Single user
  • Single workload
  • No inspection workload

The following parameters are used for the small_template:


spec:
  podTemplates:
  - containers:
    - name: vision-service
      resources:
        limits:
          cpu: "5"
          memory: 10Gi
        requests:
          cpu: "1"
          memory: 2Gi
default
  • Multiple users
  • Concurrent workloads
  • Approximately 200 devices are sending 1 image per minute

The following parameters are used for the default_template:


spec:
  podTemplates:
  - containers:
    - name: vision-service
      resources:
        limits:
          cpu: "20"
          memory: 25Gi
        requests:
          cpu: "2"
          memory: 2Gi
Note: When the resources are updated, update the Java™ virtual machine (JVM) options in the custom-service-jvm-config configuration.

vision-service-dataset

Note: The vision-service-dataset component customization is supported from Maximo Visual Inspection version 9.0.

The following parameters are used with the vision-service-dataset component customization:

Workload
Handles data set related workloads including inspection result uploads, working with data sets, and the images associated with a specific dataset.
Custom ConfigMap files

The vision-service-dataset component is an extension of the vision-service component. Therefore, these components share some Custom ConfigMaps and provide the API service for Maximo Visual Inspection. Many of the workload customizations apply to both components.

The following Custom ConfigMap files apply to both the vision-service and vision-service-dataset components:

  • custom-service-config
  • custom-service-logging-config

The following Custom ConfigMap file applies only to the vision-service-dataset component:

  • custom-svc-dataset-jvm-config
Resources
Parameters for the template type.

The following table lists the customization for the vision-service-dataset

Table 3. vision-service-dataset customization
Configuration Workload Resources
default
  • Multiple user
  • Multiple workload
  • 100 or fewer devices are sending 1 image per minute

The following resource parameters are used for the default_template:


spec:
  podTemplates:
  - containers:
    - name: vision-service-dataset
      resources:
        limits:
          cpu: "4"
          memory: 8Gi
        requests:
          cpu: "500"
          memory: 2Gi
The following HPA parameters are used for the default_template:

spec:
  minReplicas: 1
  maxReplicas: 2
  targetCPUUtilizationPercentage: 85

large
  • Multiple users
  • Concurrent workloads
  • Approximately 200 devices are sending 1 image per minute

The following parameters are used for the large_template:


spec:
  podTemplates:
  - containers:
    - name: vision-service-dataset
      resources:
        limits:
          cpu: "4"
          memory: 8Gi
        requests:
          cpu: "2"
          memory: 4Gi
The following HPA parameters are used for the large_template:

spec:
  minReplicas: 2
  maxReplicas: 4
  targetCPUUtilizationPercentage: 85

Note: The main differences between the default_template and the large_template are the HPA settings and the minimum CPU request values. When the minimum CPU value is increased, more of the vision-service-dataset component can be consumed before another component replica is created by the HPA. The HPA replicas are doubled for the large_template.
Note: When the maximum memory limit is increased, update the Java virtual machine (JVM) options in the custom-svc-dataset-jvm-config configuration. This update ensures that the JVM can consume more memory.

vision-ui

The following parameters are used with the vision-ui customization:

Workloads
Used for a front-end user session.
Custom ConfigMap file
custom-ui-config
Resources
Parameters for the template type.

The following table lists the customization for the vision-ui:

Table 4. vision-ui customization
Configuration Workload type Resources
default Multiple users

The following parameters are used for the default_template:


spec:
  podTemplates:
  - containers:
    - name: vision-ui
      resources:
        limits:
          cpu: "1"
          memory: 1024Mi
        requests:
          cpu: "0.5"
          memory: 256Mi
Note: NGINX Rate Limiting and other NGINX configurations can be customized in the custom-ui-config file.

vision-edgeman

The following parameters are used with the vision-edgeman customization:

Workload
Manages Maximo Visual Inspection Edge devices.
Custom ConfigMap file
custom-edgeman-config
Resources
Parameters for the template type.

The following table lists the customization for the vision-edgeman:

Table 5. vision-edgeman customization
Configuration Workload size Resources
small 0–20 edge devices

The following parameters are used for the small_template:


spec:
  podTemplates:
  - containers:
    - name: vision-edgeman
      resources:
        limits:
          cpu: "0.5"
          memory: 256Mi
        requests:
          cpu: "0.25"
          memory: 128Mi

default 20–100 edge devices

The following parameters are used for the default_template:


spec:
  podTemplates:
  - containers:
    - name: vision-edgeman
      resources:
        limits:
          cpu: "1"
          memory: 1Gi
        requests:
          cpu: "0.5"
          memory: 512Mi
large 100–200 edge devices

The following parameters are used for the large_template:


spec:
  podTemplates:
  - containers:
    - name: vision-edgeman
      resources:
        limits:
          cpu: "2"
          memory: 2Gi
        requests:
          cpu: "1"
          memory: 1Gi

video-microservice

The following parameters are used with the vision-video-microservice customization:

Workload
Video operations, such as frame extraction, video object detection, and coreML conversion.
Resources
Parameters for the template type.

The following table lists the customization for the vision-video-microservice:

The following table lists the customization for the vision-video-microservice:

Table 6. vision-video-microservice customization
Configuration Workload type Resources
small
  • Single user
  • No video operations
  • No Core ML conversion

The following parameters are used for the small_template:


spec:
  podTemplates:
  - containers:
    - name: vision-video-microservice
      resources:
        limits:
          cpu: "0.25"
          memory: 1Gi
        requests:
          cpu: "0.1"
          memory: 512Mi
default
  • Single user
  • Handling concurrent video (640x360) operations

The following parameters are used for the default_template:


spec:
  podTemplates:
  - containers:
    - name: vision-video-microservice
      resources:
        limits:
          cpu: "20"
          memory: 12Gi
        requests:
          cpu: "1"
          memory: 2Gi

vision-dataset-feature

The following parameters are used with the vision-dataset-feature customization:

Workload
Data sets are prepared for grouping using the Group images feature.
HPA name
<instanceID>-dataset-feature
Custom ConfigMap file
custom-datasetfeature-config
Resources
Parameters for the template type.

The following table lists the customization for the dataset-feature:

Table 7. dataset-feature customization
Configuration Workload type HPA Custom ConfigMap Memory pod limit
default
  • Single user
  • Preparing 10,000 or fewer high-resolution images
  • minReplicas: 1
  • maxReplicas: 2
WRITE_INTERVAL: 10

The following parameters are used for the default_template:


spec:
  podTemplates:
  - containers:
    - name: vision-dataset-feature
      resources:
        limits:
          cpu: "2"
          memory: 10Gi
        requests:
          cpu: "1"
          memory: 1Gi
medium
  • Single user
  • Preparing 60,000 or fewer high-resolution images
  • minReplicas: 1
  • maxReplicas: 2
WRITE_INTERVAL: 60

The following parameters are used for the medium_template:


spec:
  podTemplates:
  - containers:
    - name: vision-dataset-feature
      resources:
        limits:
          cpu: "6"
          memory: 20Gi
        requests:
          cpu: "1"
          memory: 1Gi
large
  • Single users
  • Preparing 120,000 or fewer high-resolution imagess
  • minReplicas: 1
  • maxReplicas: 2

WRITE_INTERVAL: 120

The following parameters are used for the large_template:


spec:
  podTemplates:
  - containers:
    - name: vision-dataset-feature
      resources:
        limits:
          cpu: "10"
          memory: 30Gi
        requests:
          cpu: "1"
          memory: 1Gi
Note: For more concurrent usage, increase the minReplicas and maxReplicas values.

vision-dataset-summarization

Workload
Groups that are generated by using the Group images feature.

The following table lists the customization for the vision-dataset-summarization:

Table 8. vision-dataset-summarization customization
Configuration Workload type Resources
small
  • Single user
  • Grouping 10,000 or fewer high-resolution images

The following parameters are used for the default_template:


spec:
  podTemplates:
  - containers:
    - name: vision-dataset-summarization
      resources:
        limits:
          cpu: "1"
          memory: 1Gi
        requests:
          cpu: "0.1"
          memory: 128Mi
default
  • Single user
  • Grouping 120,000 or fewer high-resolution images

The following parameters are used for the large_template:


spec:
  podTemplates:
  - containers:
    - name: vision-dataset-summarization
      resources:
        limits:
          cpu: "4"
          memory: 4Gi
        requests:
          cpu: "1"
          memory: 1Gi

vision-model-conversion

The following parameters are used with the vision-model-conversion customization:

Workload
Training YOLOv3 and Tiny YOLOv3 models. TFLite assets are generated as part of the training.
HPA name
<instanceID>-model-conversion

The following table lists the customization for the vision-model-conversion:

Table 9. vision-model-conversion customization
Configuration Workload type HPA
default
  • Single user
  • Training one model at a time
  • minReplicas: 1
  • maxReplicas: 1
medium
  • Multiple users
  • Training several models in parallel
  • minReplicas: 2
  • maxReplicas: 8
Note: The conversion workloads are coupled with training workloads. Match the minReplicas and maxReplicas values to the number of GPUs in the cluster.