Simplifying data mapping and data transformation with AI-powered suggestions

Mapping assist uses a pre-trained algorithm and mapping histories to suggest mappings for the fields in the nodes that you add when building a flow. Mapping suggestions are generated as soon as you add a node to a flow, and these suggestions identify output from previous nodes in the flow, which could possibly serve as input for the fields in the current node.

Above the filter bar, suggestions of the best possible matches (top suggestions) are indicated with a count (N suggestions), which identifies the total number of fields that will be populated with mappings. These top suggestions are estimated to have an 80% or higher level of accuracy or relevance, and, with a click, can be automatically inserted in inline preview mode into fields for review before being accepted.

In any fields that remain unpopulated, you can click to check for suggested mappings. If found, these mappings are shown in a Suggested mappings list, which displays up to five suggestions with assigned percentage matches with a 30% threshold. For each mapping, the field hierarchy identifies the origin of the content, such as the application or node name, action, and object; for example, Salesforce/Retrieve contacts/Contacts, Request/Request body parameters/Object and For each: Parsed CSV row.

If Mapping Assist is enabled, when you add a node to a flow, you'll see a Generating suggestions message (briefly) above the filter bar.

After all the potential mappings are discovered, the top N suggestions are shown next to a View suggestions link. (If multiple mappings with matching ratings of 80% or higher are discovered for the same field, those mappings are excluded from the top suggestions count to enable you to choose the best match. You can manually inspect these mappings and make a choice as described later.)

To auto-populate the fields with the top suggestions, complete the following steps:

1. Click View suggestions to switch to inline preview mode, which inserts a preview of the suggested mappings into the matching fields. The remaining unpopulated fields are shown in a disabled state.
   
2. Examine each of the preview mappings to verify that the mapping is suitable for your requirements.
   • If unsuitable, you can clear the mapping from any field by clicking the Clear suggestion icon (x), and will be able to manually populate the field later with your preferred choice.
     

     Tip: You can hover over each mapping and then check the tooltip to verify that the inserted data from the specified node is your preferred match.

     

     When you clear a suggestion, the mapping is removed from the field, which also then becomes disabled.

   • To exit from the inline preview mode without accepting any of the top suggestions, click Cancel.
     

     The initial view of top suggestions is displayed again.

     
3. To accept the preview mappings (and exit the inline preview mode), click Apply N suggestions, where N represents the count. Those fields are populated with the top suggestions and the remaining fields remain blank.
   
   Note: If multiple mappings with matching ratings of 80% or higher are discovered for the same field, that field will not be auto-populated. You will instead be expected to manually inspect such mappings to determine the best match for the data that you want to map, as described later.

If you want to replace the mapping for a top suggestion after it's inserted in a field, you can do so as follows:

1. Delete the mapping from the field. (When you delete the mapping, notice that its top suggestions count (N suggestions) is redisplayed, and you can auto-populate the field again by clicking View suggestions and Apply N suggestions. If you choose to add your own preferred mapping as described in the following steps, the count is removed and View suggestions is disabled.)
2. Remove the focus from the field and then click within the field again to display the Suggested mappings list. Alternatively, click the Insert a mapping icon to open the Available mappings list and then click Suggested mappings. Then choose another mapping as follows:
   1. Inspect the mappings and then choose any one that you prefer. (You can view the percentage matches of these mappings by hovering over the rating icon .)
      
   2. If you would prefer to choose from a wider range of mappings, click All mappings to switch to the Available mappings list, which displays all mappings from all the previous nodes in the flow. Expand the sections to locate and select your required mapping.
      Tip: From the Available mappings list, you can switch back to the Suggested mappings list by clicking Suggested mappings.
      

In any remaining fields that were not automatically populated, you can manually insert mappings into each field as follows:

1. Click within the field.
2. If suggested mappings are available for this field, the Suggested mappings list is immediately displayed. These mappings fall into two categories, and you can make a selection as follows:
   • Multiple mappings with the same percentage match: Inspect any such mappings to decide on the best match and then click the mapping that you prefer.
   • Mappings with individual percentage matches of 30% or higher: Inspect such mappings to decide on the best match and then click the mapping that you prefer.

   If you prefer to choose from a wider range of mappings, you can click All mappings to switch to the Available mappings list.

3. If no suggested mappings are discovered for this field, click the Insert a mapping icon to open the Available mappings list and then select your required mapping from one of the sections.

If no top suggestions are discovered for an action or node, or if suggested mappings are not discovered for any of the fields, the initial Generating suggestions message is replaced by a grayed-out View suggestions link when the discovery process completes.

For either of these cases, you must manually select mappings by clicking within the fields:

• If suggested mappings are available, you can use the Suggested mappings list to inspect and select mappings.
• If no suggested mappings were found, you can use the Insert a mapping icon to choose a mapping from the Available mappings list. Note that the Suggested mappings link will be grayed out in this list.
  

Incremental learning uses AI modeling techniques to periodically analyze the source-to-target mapping preferences in an App Connect Designer instance and memorize patterns that can be used to offer helpful suggestions for previously untried source-to-target mappings.

A base AI model is supplied for your App Connect Designer instance, and every 15 days or once a month (as configured for your instance), mapping preferences are collated as training data and used to retrain the model. Only simple mappings such as one-to-one field mappings between nodes are included in the training data. Complex mappings are discarded. The iterative training is used to incrementally build a personalized model for your Designer instance over time. As new mapping insights are learnt, fine-tuning occurs to improve the degree of accuracy of the suggestions that are offered when you build flows.

When mapping fields in a flow, if your source and target data formats or drop-down values do not match, you can use the transformation generator to help you construct JSONata expressions for your mappings.

The following transformations are supported:

• Pattern transformation:

  Use this transformation to change the format of source data that is passed to a target application if the source and target data formats do not match.

  For example, if a source application requires dates to be in the format DD/MM/YYYY, but the target application requires MM/DD/YYYY, you can provide sample data of the source and target formats, and then use the transformation generator to generate a JSONata expression for the "source" mapping in the target field. Similarly, if a source application requires names to be in the format Firstname Surname and the target application requires Firstname SURNAME, the transformation generator can construct a JSONata expression by using samples that you provide.

• Enum transformation:

  Use this transformation to define how to map source and target drop-down values that do not match, to ensure that appropriate values are passed to the target application.

  For example, if the source and target applications offer varying values for a Priority drop-down list as shown in the following table, you could choose to map the source and target values as follows: High:Urgent, Medium:Normal, and Low:Low. You can then use the transformation generator to generate a JSONata expression for the "source" mapping in the target field.

  Source application	Target application
  High Medium Low	Urgent Normal Low

If you are creating a flow and have applied top suggestions to auto-populate the fields for an action with mappings from previous nodes, you can manually insert mappings in the remaining fields and then apply transformations to the populated mappings as required.

Completing pattern transformations

If a text field is populated with a single mapping from a top suggestion or from the Suggested mappings or Available mappings list, but the data format of the "source" mapping does not match the data format of the target field that is populated, you can generate a JSONata expression to define how the source data should be presented in the target application.

To map the source and target data formats, complete the following steps in the version of Designer that you are using:

• Applicable to App Connect Designer 12.0.5.0-r1-lts or later:
  1. Click the relevant "source" mapping within the target field and then click Generate transformation from the pop-up menu.
     

     The "Generate transformation for SourceFieldName" panel opens with five blank source and corresponding target fields that you can use for mapping data formats. When this panel opens, an attempt is made to discover data from existing records in the source and target applications, so that this data can be used to populate the source and the target fields with examples. A Populate using retrieved data link is displayed above the left (source) column if data is discovered in the source application for the connected account.

     
  2. Provide examples of source and target data formats as follows:
     • If data is discovered in the source application (as indicated by the Populate using retrieved data link), click this link to populate the fields in the left column with a random sampling of unique examples from the source application. If fewer than five examples are discovered, manually specify similar examples in any source fields that are left blank.
       
       • If data is additionally discovered in the target application, this data is analyzed to identify various formats; for example, lowercase or uppercase formats. The source examples in the left column are then used to generate target examples for the discovered formats. Up to five formats can be shown within individual tabs (numbered 1 to 5) above the right (target) column. A Copy from source tab is also shown as the last tab. You can either click a numbered tab to display generated target examples for a specific format, or you can click the Copy from source tab to add a copy of the source examples as target examples. You can also edit the examples if required.
         
       • If no data is discovered in the target application, the fields in the right column remain blank, but a Copy from source link is displayed. You can click this link to add a copy of the source examples as target examples, and can edit the values as required.
         
       
       
     • If no data is discovered in the source and target applications, a Populate using retrieved data link is not displayed. A Copy from source link is shown as initially disabled.
       

       In the left column, manually specify at least five examples for the source field. (The Copy from source link becomes enabled after you add the first entry.) Next click Copy from source to add a copy of the source examples as target examples, and then edit the target examples if required. If you prefer not to copy the source examples, you can instead just manually specify target examples in the preferred format.

     Tip:

     If you decide to manually specify examples:

     • Ensure that an identical data format is used for all the source field examples. Similarly ensure that an identical data format is used for all the target field examples.
     • Ensure that there is a completed example for each source/target pair of fields.
     • Ensure that each example is of the expected data type. For example, use a date format if the source and target fields are used to present dates.

     To add more examples, click Add row. You can also delete examples, but must ensure that at least five completed source/target pairs are provided in order to enable the Generate transformation link.

  3. Click Generate transformation to generate a JSONata expression that represents your selections.
     
  4. Click Insert transformation to insert the JSONata expression into the target field.
     
• Applicable to App Connect Designer 12.0.4.0-r2 or earlier:
  1. Click the relevant "source" mapping within the target field and then click Generate transformation from the pop-up menu.
     

     The "Generate transformation for SourceFieldName" panel opens with five blank source and corresponding target fields that you can use for mapping data formats.

     
  2. In the left column, provide at least five examples for the source field by typing in values, and then add examples in the right column to show the preferred format for the target field.
     • Ensure that an identical data format (or pattern) is used for all the source field examples.
     • Ensure that an identical data format is used for all the target field examples. To complete a target example, copy and paste the corresponding source format example and then edit the format as required.
     • Ensure that there is a completed example for each source/target pair of fields.
     • Ensure that each example is of the expected data type. For example, use a date format if the source and target fields are used to present dates.
     • To add more examples, click Add row. You can also delete examples, but must ensure that at least five completed source/target pairs are provided in order to enable the Generate transformation link.
  3. Click Generate transformation to generate a JSONata expression that represents your selections.
     
  4. Click Insert transformation to insert the JSONata expression into the target field.
     

Completing enum transformations

If a drop-down field is populated with a single mapping from a top suggestion or from the Suggested mappings or Available mappings list, and the drop-down list values for the inserted "source" mapping do not match the drop-down list values for the target field that is populated, the following notification is displayed under the target field:

SourceApplication has different values for SourceFieldName

To map the drop-down values for the source and target fields, complete the following steps:

1. Click the Generate transformation link to open the "Generate transformation for SourceFieldName" panel. The drop-down values for the source field are listed in left column, and the right column contains corresponding drop-down lists that are populated with target values.
   
2. Use the target drop-down lists to map an appropriate target value to each source value.
3. Click Generate transformation to generate a JSONata expression that represents your selections.
   
4. Click Insert transformation to insert the JSONata expression into the target field.
   

Disclaimer: You are responsible for any example data (including any personal information) used for, and any decision to proceed with, any automated flows which are suggested for your convenience when IBM App Connect is used. IBM has no responsibility for any such automated flows and IBM warranties and support will not apply to them; they are used at your risk. IBM might periodically modify the underlying learning models in App Connect through updates, fixes, or patches in order to improve App Connect performance.