Dynamic recipe

Use this section to create a dynamic recipe and guidelines on how to create it.

Dynamic recipe provides a capability to handle the backup and restore of applications with multiple sub-components dynamically. With a dynamic recipe, you can define a parent recipe that automatically discovers child recipes that provide extra workflow information, making your recipes more dynamic.
Some of the important considerations you should follow when you create a dynamic recipe:
  • Make sure that child recipes correctly reference the parent recipe by using labels.
  • You can define new workflows apart from reserved backup and restore workflows.
  • If a child recipe uses the same workflow as the parent with a higher priority value, then the child recipe runs first.
  • If a child recipe shares the same workflow and priority value as its parent, then the execution order is random but they run sequentially.

    The priority value can range from -2,147,483,648 to 1,000,000,000.

    For example:
    • A workflow with priority -2,147,483,648 contain the lowest precedence.
    • A workflow with priority 1,000,000,000 contain the highest precedence.

Creating a parent recipe

In addition to the standard backup and restore workflows, you can define the new workflows in the dynamic recipe. These workflows enable dynamic behavior during the backup and restore processes.
Note: Parent and child can have same hook or group name but during recipe merge process child hook name is going to be prefixed automatically with child recipe name separated by period.
For example:
apiVersion: spp-data-protection.isf.ibm.com/v1alpha1
kind: Recipe
metadata:
  name: parent-recipe
  namespace: ibm-spectrum-fusion-ns
spec:
  ...
  hooks:
  ...
  workflows:
  - name: pre-hook
    priority: 4
    sequence:
    - hook: mysql-pod-exec/flush-tables-with-read-lock
  - name: post-hook
    priority: 4
    sequence:
    - hook: mysql-deployment-check/replicasReady
  - name: backup
    sequence:
    - group: mysql-resources
    - workflow: pre-hook
    - group: mysql-volumes
  - name: restore
    sequence:
    - group: mysql-volumes
    - group: mysql-resources
    - workflow: post-hook
Make sure that the names can be any valid string or similar to other workflow names like backup and restore. In this scenario, pre-hook is used in the backup workflow and post-hook is used in the restore workflow.

If a child recipe defines the same workflow, then it dynamically retrieves and runs.

Creating a child recipe

Once a recipe linked to a another recipe through these labels dp.isf.ibm.com/parent-recipe and dp.isf.ibm.com/parent-recipe-namespace, then parent-child relationship gets created.

For example:
apiVersion: spp-data-protection.isf.ibm.com/v1alpha1
kind: Recipe
metadata:
  labels:
    dp.isf.ibm.com/parent-recipe: parent-recipe
    dp.isf.ibm.com/parent-recipe-namespace: ibm-spectrum-fusion-ns
  name: child-1
  namespace: ibm-spectrum-fusion-ns
spec:
  appType: mysql-child-1 
  groups:
  ...
  hooks:
  ...
  workflows:
  - name: pre-hook
    priority: 5000 
    sequence:
    - group: cm-child-1
    - hook: mysql-pod-exec-child-1/create-temp-file-pre
  - name: post-hook
    priority: 5000
    sequence:
    - hook: mysql-pod-exec-child-1/create-temp-file-post 
    - group: cm-child-1

Adding new workflows with priorities

A child recipe can define workflows that are present in the parent recipe.
Adding a new workflow
For example, add a new workflow named demo-workflow with priority 1 in the parent recipe. 
workflows:
  - name: demo-workflow
    priority: 1
    sequence:
      - hook: demo-hook/echo-test
Now, you can define the same workflow in the child recipe. The execution behavior depends on the assigned priority:
  • High priority - Runs before the parent workflow
  • Low priority - Runs after the parent workflow
  • Same priority - Run order is arbitary (either the parent or child might run first).
Child workflow with higher priority
For example, the child recipe defines a demo-workflow with priority 5, then it runs before the parent workflow.
workflows:
  - name: demo-workflow
    priority: 5
    sequence:
      - hook: demo-child-hook/echo-child-test

Adding new workflows without a sequence

A parent recipe can define workflows that have no sequence. This allows child recipes to use these workflows and define their own sequences.
Defining a workflow in the parent recipe
For example, add a new workflow named demo-workflow in the parent recipe with no sequence and priority 0:
workflows:
  - name: demo-workflow
    priority: 0
    sequence: []
Defining a workflow in the child recipe
Now, the child recipe can define the same workflow and specify its own sequence:
workflows:
- name: demo-workflow
priority: 5
sequence:
- hook: demo-child-hook/echo-child-test1
- hook: demo-child-hook/echo-child-test2
Important:
  • If the parent has only one child and parent workflow sequence is empty, then priority does not matter, and the child workflow executes as defined.
  • If the parent has multiple children defining the same workflow, execution occurs according to the assigned priorities.

Pre-recipe inventory workflow (Optional)

This is an optional workflow for the parent recipe, run before the backup workflow.
  • It allows an application to define and run its own custom exec hook in the parent recipe before any child recipe is merged into parent.
  • Upon successful execution, IBM Fusion continues processing and takes the child recipe inventory for dynamic recipe execution.
  • The main objective is to construct child recipes that reflect the current state of the application in the cluster.
For example:
apiVersion: spp-data-protection.isf.ibm.com/v1alpha1
kind: Recipe
metadata:
  name: parent-recipe
  namespace: ibm-spectrum-fusion-ns
spec:
  appType: mysql-ns
  groups:
  ...
  hooks:
  - name: my-child-recipes
    type: exec
    ...
    - name: "manage"
...
  workflows:
  - name: prerecipeinventory
    sequence:
    - hook: my-child-recipes/manage
  ...

Parallel backup hook execution

Running sequences in parallel within dynamic recipe workflows can significantly speed up the overall process. It is useful for workflows that involve tasks that depend heavily on I/O operations. If any one sequence fails, then the entire job aborts, but it waits for any currently running sequences in other parallel blocks to finish first.
Note: Undo operations are still performed sequentially, but in the reverse order of workflow execution.

Parallel hook execution can be requested in a IBM Fusion recipe by assigning the same priority to the workflow step in each recipe.

Key considerations
  • Parallel execution is available for both backup and restore workflows.
  • Parallel execution only can be used with dynamic recipes (when there is a parent recipe and at least one child recipe).
  • If there are multiple recipes, consider running them simultaneously to increase efficiency.
    • Workflows with same priority run in parallel only if all sequences are hooks.
    • Workflows with same priority run sequentially if they have at least one group sequence within the workflow.
    • Workflows with different priorities run sequentially.
    • Workflows with mixed priorities: For example, there is a shared workflow that is defined in three recipes, two have the same priority and one is a different priority. In this particular case, the workflows with identical priorities run in parallel and the recipe with the different priority runs sequentially, before or after depending on the relative priorities.
  • Number of parallel workers are controlled by flag hook-parallel-workers in guardian-configmap (ibm-backup-restore namespace). The default value is 1 and can be set to higher values with maximum of 35. Minimum value 0 have special meaning - internally translates to number of usable CPUs.

Parallel restore hook execution

IBM Fusion's parallel restore mechanism follows the same approach as its backup process. During restoration, the system dynamically merges saved parent-child recipes at runtime to build the final restore workflow. If a child recipe is not eligible for restoration to an alternate namespace, the system skips it during the merge. The status field in the Fusion Restore Custom Resource (CR) records details of any skipped recipes.

To perform a parallel restore with modified recovery steps (distinct from the saved parent recipes), create a custom Restore CR. Specify the new recipe name and namespace in the spec field.
Key considerations

Parallel restore adheres to all the key considerations outlined for backup. Additionally, it possesses specific capabilities that further contribute to successful application recovery.

  • The system excludes child recipes that are labeled with dp.isf.ibm.com/alternate-namespace-restore=false when you target alternate namespaces.
  • To introduce slight variations in restore steps, you can specify a new recipe name and namespace in the custom Restore CR. This existing capability is valuable especially during parallel restores.
Sample custom Restore CR
apiVersion: data-protection.isf.ibm.com/v1alpha1
kind: Restore
metadata:
  name: <custom-restore-name>
  namespace: <custom-restore-namespace>
spec:
  recipe:
    name: <new-parent-recipe-name>
    namespace: <new-parent-recipe-namespace>

Parallel backup and restore can significantly optimize data protection times when processing Fusion recipes, which have multiple activities that can be executed in parallel.