Applying masking policies

Edit online

Use masking policies in IBM® Optim Data Privacy to define how sensitive fields get transformed during test data provisioning.

Note: Optionally, you can apply advanced data masking and de-identification by using the IBM Optim Data Privacy advanced data masking API. For more information, see Advanced data masking by using the IBM Optim Data Privacy API.
Who can complete this task?

Administrators or users can create masking policies. Standard users can see and work only with the masking policies that they created. Administrators can see and work with all masking policies in the system, regardless of who created them.

Before you begin

You must have a valid access definition node with source tables selected.

Review the masking policy reference topics in the Masking policy reference section.

Applying masking policies by using the flow builder

You can apply masking policies to a flow by adding a Mask node to the flow builder canvas. Mask nodes must follow an Access definition or Subset node in a flow.

Figure 1. Masking is the optional fourth step in the flow of data in IBM Optim
Data flows from the source, through the access definition, then undergoes subsetting and masking before going to the target

To add masking to a flow:

  1. From the navigation menu, click Workflow icon Workflow > Flow builder, then open a flow builder canvas:
    • Click New flow to start a new flow builder canvas.
      or
    • Load an existing flow from the My flows panel.
      or
    • Click Import to load a previously exported flow from a valid JSON flow definition. For more information, see Importing and exporting a flow.
      or
    • Click Node templates to view templates saved from existing nodes such as access definitions, subsets, masking policies, or targets.
  2. Add an access definition node to your flow builder canvas, if one does not already exist. For more information, see Creating an access definition.
  3. Optionally, create a subset node. For more information, see Applying subsetting conditions.
  4. Connect a mask node to the previous access definition or subset node in your flow. You can either create a new masking definition or add a predefined masking definition from your node templates.
    • To create a new masking definition, click on the circle on the side of the previous node in the flow, then drag the connector line to an open position on your canvas. Select New masking definition.
    • To add a masking definition based on an existing node template, drag a mask template from the Node templates panel and drop it onto the canvas. For more information, see Creating and working with template nodes.
  5. Define your mask node details:
    1. Provide a name and description for the masking definition, then click Next.
    2. Click Add policy to open the policy configuration panel.
    3. In the Assign columns section, select one or more table columns to apply the masking policy to. Expand the tables to view available columns, then select the checkboxes for the columns that you want to mask.
    4. In the Add policy details section, select the policy type by choosing either Predefined policy or Standard policy.

      For details about the various masking policy types, see Masking policy reference.

    5. Configure the policy based on your selection:
      • For predefined policies: Select a predefined policy from the option list (such as Redact Policy, Social Security Number Policy, National Identification Policy, Credit Card Policy, or Email Policy) and configure any available parameters.
      • For standard policies: Choose a Data category that defines the type of data to mask (such as Address, Credit Card Information, Device And Network Address, Email, Financial Information, or Healthcare Information). Select a Data format type that provides more detail specific to the category, and finally choose a Data processor type that defines how the masking definition transforms the data. For more information about data processor types, see Choosing a processor.
    6. Click Add to add the policy to your masking definition.
    7. To define multiple masking rules, click Add policy again and repeat the configuration process. Add as many policies as you need to create your masking definition.
    8. Click Next to proceed to the review step.
    9. Review the masking definition that you configured, then click Create.
  6. Click Save to save the flow.
  7. Finish building the flow by adding more nodes if you need them. For more information about creating flows, see Building an IBM Optim Data Privacy flow.

Applying masking policies by using the API

You can define masking policies by using the API. The API supports both standard policies and predefined policies, with different payload structures for each type.

  1. For API usage, ensure that the following prerequisites are in place:
    • You have access to the appropriate masking categories and types (retrievable using the API).
    • You have a valid access token (<accessToken>). For more information, see Retrieving access tokens for API usage.

To apply masking policies to a flow by using the API:

  1. Send a POST or PUT request to https://VM_HOSTNAME:PORT/optim/v1/flow using the required headers and a JSON payload that defines the masking configuration.
    Headers
    
Authorization: Bearer <accessToken>
Content-Type: application/json
    Masking node template
    {
  "id": "<UUID for masking definition node>",
  "type": "execution_node",
  "op": "transform_definition",
  "app_data": {
    "name": "<name for masking definition>",
    "mask_tables": [
      {
        "source_table_id": "<UUID for table entry from access definition>",
        "table_name": "<table name>",
        "database": "<database name>",
        "schema": "<schema name>"
        "columns": [
          {
            "column_name_source": "<source column name>",
            "column_name_target": "<target column name>",
            "masking_properties": {
              "magen_function": {
                "config": {},
                "name": "<magen function name>",
                "type": "<magen function type>"
              },
              "masking_definition_id": "<UUID for masking definition ID>"
            }
          }
        ]
      }
    ]
  }
}
    Sample request body for standard policy

    This job masks the first_name column in the customers table with the USFirstName masking format. The RedactProcessor type performs the masking.

    {
  "id": "302e581d-56dd-47dc-9bc2-465c78691a76",
  "version": "3.0",
  "app_data": {
    "optim": {
      "version": "2.0.0"
    }
  },
  "doc_type": "pipeline",
  "pipelines": [
    {
      "id": "369d625b-ca3e-45a5-bbd0-08d41c1bc8b8",
      "name": "F2",
      "nodes": [
        {
          "id": "a3761753-58cd-40fe-a449-668b0669aa89",
          "op": "connection_profile",
          "type": "binding",
          "inputs": [],
          "outputs": [],
          "connection": {
            "ref": "f4dc15d1-25ee-4a20-800c-caa0d439e451",
            "name": "PgSQL"
          },
          "parameters": {
            "connection_type": "source"
          }
        },
        {
          "id": "e4d89905-6952-4e17-b929-3e0917a14091",
          "op": "access_definition",
          "type": "execution_node",
          "app_data": {
            "source_tables": [
              {
                "id": "a79a9b95-6e11-4a15-a32e-919ca935228c",
                "schema": "public",
                "database": "optim_test",
                "table_name": "customers",
                "connection_profile": "f4dc15d1-25ee-4a20-800c-caa0d439e451"
              }
            ]
          }
        },
        {
          "id": "e0e35e2b-af57-4b66-b967-6ef40791ba57",
          "op": "subset_definition",
          "type": "execution_node",
          "app_data": {
            "subset_tables": [
              {
                "schema": "public",
                "database": "optim_test",
                "table_name": "customers",
                "source_table_id": "a79a9b95-6e11-4a15-a32e-919ca935228c",
                "subset_parameters": [
                  {
                    "type": "where_clause",                    
                    "structured_conditions": {
                        "condition": "AND",
                        "clauses": [
                          {
                            "value": 1000,
                            "operator": "=",
                            "table_name": "customers",
                            "column_name": "customer_id",
                            "schema_name": "public",
                            "table_alias": ""
                          },
                          {
                            "value": "'EMMA'",
                            "operator": "<>",
                            "table_name": "customers",
                            "column_name": "first_name",
                            "schema_name": "public",
                            "table_alias": ""
                          }
                        ]
                    }
                  }
                ]
              }
            ]
          }
        },
        {
          "id": "ab5b6b4f-7a91-47a4-b6a1-4cdad3c13396",
          "op": "connection_profile",
          "type": "binding",
          "inputs": [],
          "outputs": [],
          "connection": {
            "ref": "6a7b49eb-4de8-4390-9431-91bfe08fd0b6",
            "name": "DB2"
          },
          "parameters": {
            "connection_type": "target"
          }
        },
        {
          "id": "cf7e971c-a3c3-4643-a4ba-5231dd4c7000",
          "op": "insert_definition",
          "type": "execution_node",
          "app_data": {
            "table_map": [
              {
                "id": "64febd6b-8952-45a2-af49-05c0caec24eb",
                "source": {
                  "schema": "public",
                  "database": "optim_test",
                  "table_name": "customers",
                  "source_table_id": "a79a9b95-6e11-4a15-a32e-919ca935228c",
                  "connection_profile": "f4dc15d1-25ee-4a20-800c-caa0d439e451"
                },
                "target": {
                  "schema": "demo_target",
                  "database": "testdb",
                  "table_name": "customers",
                  "connection_profile": "6a7b49eb-4de8-4390-9431-91bfe08fd0b6"
                }
              }
            ],
            "insert_parameters": {
              "relational_integrity": {
                "conflict_resolution": "merge"
              }
            }
          }
        },
        {
          "id": "daa01f77-aea9-4d9f-a4af-cc4f662b30ac",
          "op": "transform_definition",
          "type": "execution_node",
          "app_data": {
            "mask_tables": [
              {
                "schema": "public",
                "columns": [
                  {
                    "column_name_source": "first_name",
                    "column_name_target": "first_name",
                    "masking_properties": {
                      "magen_function": {
                        "name": "USFirstName",
                        "type": "RedactProcessor",
                        "config": {}
                      },
                      "masking_definition_id": "982dae75-af7e-457b-9f7f-acda2431df10"
                    }
                  }
                ],
                "database": "optim_test",
                "table_name": "customers",
                "source_table_id": "a79a9b95-6e11-4a15-a32e-919ca935228c"
              }
            ]
          }
        }
      ],
      "app_data": {
        "ui_data": {
          "comments": [],
          "worksheet_type": "subset"
        }
      },
      "runtime_ref": "optim-runtime:2.0.0"
    }
  ],
  "parameters": {},
  "json_schema": "https://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json",
  "primary_pipeline": "369d625b-ca3e-45a5-bbd0-08d41c1bc8b8"
}
    Sample request body for predefined policy

    This example shows a transform definition using a predefined policy. The masking_policy_id in the masking_properties section points to a policy object in the mask_policies array. Each predefined policy is a self-contained Elyra JSON structure with unique parameters.

    Note: Predefined policies have different templates that require manual editing in the JSON. You can retrieve the appropriate template for each predefined policy type by using the template API.
    {
  "id": "a63b7875-9717-4b8a-aa98-a18526c1c91e",
  "op": "transform_definition",
  "type": "execution_node",
  "app_data": {
    "mask_policies": [
      {
        "id": "12fb6599-9d1e-45a9-9d78-ce6208b8a89e",
        "name": "redact-policy",
        "type": "optim_policy",
        "policy": {
          "id": "12fb6599-9d1e-45a9-9d78-ce6208b8a89e",
          "version": "3.0",
          "doc_type": "pipeline",
          "json_schema": "https://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json",
          "primary_pipeline": "dcd8cacf-d973-48cd-8126-fecdc34030b1",
          "parameters": {},
          "app_data": {
            "optim_tdm": {
              "version": "1.1.0"
            }
          },
          "pipelines": [
            {
              "id": "dcd8cacf-d973-48cd-8126-fecdc34030b1",
              "name": "Redact Policy",
              "runtime_ref": "optim-runtime:1.0.0",
              "app_data": {
                "expressions": [],
                "formats": [],
                "policy_metadata": {
                  "length": 6
                },
                "ui_data": {
                  "summary": "charMask: @, length: 6, prefix: p"
                }
              },
              "nodes": [
                {
                  "id": "b5d4ca80-8d7b-483c-9646-eb2142764d88",
                  "type": "execution_node",
                  "op": "processor_definition",
                  "app_data": {
                    "originalId": "redact",
                    "processor": {
                      "id": "Redactor",
                      "type": "RedactProcessor",
                      "root": true,
                      "loggable": true,
                      "configuration": {
                        "charMask": "@",
                        "prefix": "p",
                        "suffix": "",
                        "preProcessorConfiguration": {
                          "javaScript": "function process(processor,payload,metaData,state,magenLogger){var value=String(payload),length=String(metaData.get(\"length\")),number=Number(length);if(isNaN(number)||0===number)return metaData.put(\"trail\",\"\"),value;if(number>value.length){for(var padding=[],i=0;i<number-value.length;i++)padding.push(\"x\");return metaData.put(\"trail\",\"\"),value+padding.join(\"\")}return metaData.put(\"trail\",value.substring(number)),value.substring(0,number)}"
                        },
                        "postProcessorConfiguration": {
                          "javaScript": "function process(processor,payload,metaData,state,magenLogger){var trail=String(metaData.get(\"trail\")),value=String(payload);return\"\"===trail?value:value+trail}"
                        }
                      }
                    },
                    "ui_data": {
                      "label": "Redact",
                      "x_pos": 19,
                      "y_pos": 2,
                      "ui_parameters": {
                        "title": "Redact",
                        "subtitle": "Processor",
                        "color": "#6929C4",
                        "image": "connect"
                      }
                    }
                  },
                  "inputs": [
                    {
                      "id": "processor_port_in",
                      "app_data": {
                        "ui_data": {
                          "label": "Processor Input Port",
                          "cardinality": {
                            "min": 0,
                            "max": 100
                          }
                        }
                      }
                    }
                  ],
                  "outputs": [
                    {
                      "id": "processor_port_out",
                      "app_data": {
                        "ui_data": {
                          "label": "Processor Output Port",
                          "cardinality": {
                            "min": 0,
                            "max": 100
                          }
                        }
                      }
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    ],
    "mask_tables": [
      {
        "database": "archon_test",
        "schema": "archon_src",
        "table_name": "DA_CUSTOMERS_CA",
        "source_table_id": "49f37fad-3c3b-4949-aa8e-0cc7f8c2e2c6",
        "columns": [
          {
            "column_name_source": "CUSTNAME",
            "column_name_target": "CUSTNAME",
            "masking_properties": {
              "masking_definition_id": "7dc6b7da-733b-4288-9646-acbb0ecea123",
              "masking_policy_id": "12fb6599-9d1e-45a9-9d78-ce6208b8a89e",
              "magen_function": {
                "name": "redact-policy",
                "type": "12fb6599-9d1e-45a9-9d78-ce6208b8a89e",
                "config": {}
              }
            }
          }
        ]
      }
    ],
    "name": "mask-len-6",
    "ui_data": {
      "label": "mask-len-6",
      "x_pos": 54,
      "y_pos": 245.8125,
      "ui_parameters": {
        "title": "mask-len-6",
        "subtitle": "Masking definition",
        "color": "#b28600",
        "image": "transform_code",
        "isPoliciesLoading": false,
        "policies": ["redact-policy"]
      }
    }
  },
  "inputs": [
    {
      "id": "transform_port_in",
      "app_data": {
        "ui_data": {
          "label": "",
          "cardinality": {
            "min": 0,
            "max": 1
          }
        }
      },
      "links": [
        {
          "id": "a3c66e51-ca4a-460d-b337-af628b025daa-to-a63b7875-9717-4b8a-aa98-a18526c1c91e-link",
          "node_id_ref": "a3c66e51-ca4a-460d-b337-af628b025daa",
          "port_id_ref": "access_definition_port_out"
        }
      ]
    }
  ],
  "outputs": [
    {
      "id": "transform_port_out",
      "app_data": {
        "ui_data": {
          "label": "Output Port",
          "cardinality": {
            "min": 0,
            "max": 1
          }
        }
      }
    }
  ]
}
  2. To update an existing masking configuration, send a PUT request to https://VM_HOSTNAME:PORT/optim/v1/flow with the updated masking rules in the request body.
  3. To define masking rules for more tables or columns, extend the mask_tables array with the relevant schema, table name, and column-level masking properties.

What to do next

  • Preview the masked data to validate the applied policies before running the flow. For more information, see Previewing masked data.
  • For standard policies with the custom data category, you can define custom masking formats to control the output format of masked data. For more information, see Defining custom data masking formats.