Using the canvas to define a flow

Edit online

Configure a flow in IBM® Optim Archive by using the flow builder canvas to define the sequential set of nodes (connection, access definition, subset, mask, and target) that make up the end-to-end process of preparing and provisioning data.

Configuring an IBM Optim Archive flow involves selecting the data to be extracted, applying optional transformations, and defining how and where the data will be exported.

Who can complete this task?

Administrators or users can create flows. Standard users can only see and work with the flows that they created. Administrators can see and work with all flows in the system, regardless of who created them.

Building a flow

To build a new IBM Optim Archive flow, or edit an existing one:

  1. From the navigation menu, click Workflow icon Workflow > Flow builder.
    • Click New flow to start a new flow builder canvas.
      or
    • Load an existing flow from the My flows panel.
      or
    • Click Import to load a previously exported flow from a valid JSON flow definition. For more information, see Importing and exporting a flow.
      or
    • Click Node templates to view templates saved from existing nodes such as access definitions, subsets, masking policies, or targets.
  2. Create new nodes or drag and drop existing node configurations from the Node templates panel onto the canvas.
    Nodes must be added to the flow builder canvas in the following order:
    1. Access definition

    The definition of the specific data to be extracted from the database connection. Access definitions are based on specific source connection nodes.

    When you add an access definition node, it will automatically pull in any required source nodes.

    For information about defining an access definition, see Creating an access definition.

    2. Subset (optional)

    A subset of data from the access definition that you want to include in your extracted data.

    For information about defining a subset, see Subsetting data.

    3. Masking (optional)

    The set of masking policies that will be applied to the data. Masking ensures that personally identifiable information (PII), and other confidential data are transformed while preserving data integrity and usability.

    For information about defining masking policies, see Masking data.

    4. Target

    The final target location of the extracted and exported data, such as a target database connection, archive, or extract.

    For information about defining a target, see Defining a target.

    Note: After you drop a node onto the canvas, the previous node will become locked. This means that it cannot be removed from the canvas unless its dependent nodes are removed first.
  3. Click on any node in the canvas to edit it as needed.
    Note: When you edit or create a node, the new node must be compatible with the existing nodes in the flow. For example, an access definition node must contain all tables and columns referenced by downstream nodes such as subset and masking nodes. For details about node validation, see Node validation.
  4. To save progress on your flow at any time, click Save. Saving a flow does not run it as a job; it only saves it as a draft flow.
  5. After building your flow and defining all of your nodes, you are ready to run your flow. A running flow job extracts data from the connection based on the access definition, applies subsetting and masking, and then exports it to the target.

    To run the flow, click Run flow.

Node validation

When you edit a node or create one from a node template, the node must be compatible with the existing nodes in the flow. This validation is in place to prevent incompatible nodes from being added to a flow.

For access definition nodes
IBM Optim validates to ensure that all tables and columns in downstream nodes are defined in the access definition node. This is a strict rule. All data in the flow must be defined in the access definition.
  • If any tables or columns are missing from the access definition, then the node cannot be added and you will see an error message.
  • If all tables and columns are included in the access definition, then the node is added or updated.
For subset and masking nodes
IBM Optim validates to ensure that at least one table or column from the node is included in the access definition node.
  • If at least one table or column in the node is valid, then the node will be added to the flow. If some tables or columns are missing from the access definition, they will be filtered out of the subset or masking node and IBM Optim displays inline warning notifications. This smart filtering enables subset and masking nodes to automatically adapt to the access definition.
  • If no tables or columns are valid, then the node cannot be added and you will see an error message.

For more information about node templates, see Creating and working with template nodes.