Guidelines for configuring custom libraries

Resilience evaluation process

During a resilience assessment, Concert processes operational data in multiple stages before resilience scores and resilience actions are generated.

Configuring requirement definitions

Requirement definitions identify the operational capability or runtime condition that Concert evaluates during resilience assessments.

Requirement descriptions should provide sufficient operational context to support accurate resilience evaluations and resilience action generation.

• Vague requirement descriptions can result in incomplete remediation recommendations
• Incomplete operational context can reduce action accuracy for critical operational conditions

Requirement categories affect how Concert groups operational risks and prioritizes resilience actions during evaluations.

Requirement evaluation behaviour is controlled by the criteria_type configuration.

Supported evaluation methods can include:

• snippet_ref
• pre_computed
• Default evaluation functions

The selected evaluation method determines how metric values are evaluated during resilience assessments.

For example:

• A runtime availability requirement can evaluate whether application instances remain operational during deployment failures.
• A Kubernetes configuration requirement can evaluate whether CPU and memory limits are configured for containers.
• A Java runtime requirement can evaluate unsupported runtime versions or insecure runtime configurations.

The following example shows a requirement definition in a custom library configuration:

{
  "name": "message_throughput",
  "category": "Performance",
  "criteria_type": "snippet_ref",
  "expected_score": "85",
  "rating_thresholds": "50,70,85,95",
  "input_data_keys": "avg_messages_per_second, peak_message_rate"
}

This configuration defines:

• The operational capability being evaluated
• The expected resilience score
• Metric values used during evaluation
• The operational metrics required for scoring and resilience action generation
• The evaluation method used during assessments

Configuring metric definitions

Metric definitions specify the operational data that Concert evaluates for each requirement.

Metrics are configured in the input_data_keys.json file and define:

• Metric identifiers
• Aggregation behavior
• Supported data types
• Operational data sources
• Facts-to-metric conversion behavior

Metric definitions should include enough operational context to support accurate evaluations and remediation actions.

• A container compliance metric can evaluate the percentage of containers missing CPU limits.
• A metric can aggregate NanoCpus values by using the latest value received during assessments.
• A metric can convert missing container counts into percentage-based evaluation values.

The following example shows how metric thresholds can be defined in the input_data_keys.json configuration file for a custom library:

[
  {
    "name": "pct_containers_missing_cpu_limit",
    "data_type": "double",
    "unit": "i18n:pct_unit",
    "description": "i18n:pct_containers_missing_cpu_limit_desc_key",
    "label": "i18n:pct_containers_missing_cpu_limit_label",
    "origin": "i18n:container_scanner_origin_label",
    "metadata": {
      "sources": [
        {
          "source": "docker_inspect",
          "type": "Container",
          "search_key": "HostConfig",
          "fact_field": "NanoCpus",
          "fact_aggregation": "latest"
        }
      ]
    },
    "facts_to_metric_conversion": "missing_count_percentage"
  }
]

Configuring fact aggregation functions

Fact aggregation functions evaluate multiple received values for the same operational fact field within an assessment period. Aggregation determines whether Concert uses the sum, average, latest value, minimum value, or maximum value before metric evaluation occurs. Aggregation does not define how data is collected. It defines how multiple received values for the same operational fact field are processed during evaluation.

• Error count values received multiple times for the same deployment during an assessment period can be summed.
• Response time values received for the same runtime instance during an assessment period can be averaged.
• CPU utilization values received for the same container during an assessment period can use the latest reported value.
• Availability status values received for the same application instance during an assessment period can use the minimum reported value.

Aggregation functions should be selected carefully because incorrect aggregation behavior can result in inaccurate resilience evaluations or incomplete resilience actions.

Configuring facts-to-metric conversion functions

Facts-to-metric conversion functions transform aggregated operational facts into metric values that Concert evaluates during resilience assessments. Conversion configurations are defined by using the facts_to_metric_conversion field in metric definitions.

• Boolean operational states can be converted into percentage-based metrics.
• Runtime health conditions can be converted into numerical resilience scores.
• Operational status values can be converted into threshold-based evaluation metrics.
• Misconfiguration compliance results can be converted into percentage-based compliance metrics.

Incorrect conversion configurations can result in inaccurate resilience scores, unsupported metric evaluations, or incomplete remediation actions.

The following table describes commonly used facts_to_metric_conversion functions and how they transform operational facts into metric values during resilience evaluations.

Configuring resilience thresholds

Validating resilience evaluations and resilience actions

Regular validation helps maintain accurate resilience evaluations, resilience scores, and operational remediation actions across environments