Topic
4 replies Latest Post - ‏2013-11-14T14:18:50Z by smithha
phuyn
phuyn
2 Posts
ACCEPTED ANSWER

Pinned topic In IA 9, are functions used in defining rules user-definable?

‏2013-11-12T19:12:13Z |

In IA 9.1, the built-in functions that can be used to define rules are fairly limited. For example, given an input string I want to check that it is a comma-separated list of code values that are valid (e.g., from a reference table of code values). Can I write a function that "splits" the input string with commas and checks if all returned values are in a reference table?

  • RobertDickson
    RobertDickson
    66 Posts
    ACCEPTED ANSWER

    Re: In IA 9, are functions used in defining rules user-definable?

    ‏2013-11-12T23:49:18Z  in response to phuyn

    Hi,

    While there are quite a few rules that can be used, they are more along the 'validation' or 'test' side, and not the 'transformation' side.  There are functions available when you create an output column, but I don't see ones that will give you exactly what you need.  For your use case, you may want to create a database view that splits the input the way you want.  There is no way I know of to create user-defined functions.

    IBM support may be able to offer a better answer.

    Regards,

    Robert

    • smithha
      smithha
      162 Posts
      ACCEPTED ANSWER

      Re: In IA 9, are functions used in defining rules user-definable?

      ‏2013-11-13T12:59:58Z  in response to RobertDickson

      Depending on the exact requirements, you might be able to use the Matches_Regex check and put together a regular expression to check the strings for valid values.  That's as close as you can get to user-defined functions at present.  It would not support actual parsing of the data though and if you need that either a database view as Robert noted or use of a tool with strong parsing capabilities such as QualityStage would be the best options.

      Harald

      • phuyn
        phuyn
        2 Posts
        ACCEPTED ANSWER

        Re: In IA 9, are functions used in defining rules user-definable?

        ‏2013-11-13T23:17:07Z  in response to smithha

        Thanks Robert and Harald for your reply. My preference is to have the IBM tool provide the solution, instead of using the source database. The obvious benefit then is to keep all the assets in one place so that when the source platform changes, we don't have to recreate all those assets. Currently we don't have QualityStage, but before recommending the purchase of QualityStage, I want to make sure that it can give us the expected extensibility. Regular expression is good but its power is limited. Would QualityStage allow us to define rules with ANY logic we want?

        • smithha
          smithha
          162 Posts
          ACCEPTED ANSWER

          Re: In IA 9, are functions used in defining rules user-definable?

          ‏2013-11-14T14:18:50Z  in response to phuyn

          Hi,

          QualityStage provides robust capability and rules to support:

          • Parsing/standardization of data
          • Deduplication/matching/linking of data
          • Survivorship/consolidation of data

          Parsing of a comma-separated list of values is readily handled with the Standardization rules, and you can build parsing/standardization rules for extremely complex data domains such as addresses, products, description text, etc. where the parsing is much less obvious.  It can incorporate standardization or cleansing of subtle data issues through use of lookup tables and classifications of codes including fuzzy comparison to address subtle spelling issues.  The standardization logic does not validate in and of itself. 

          You can use additional stages for data transformations with a broad array of functions to handle more of the validation side, or you can embed Information Analyzer data rules directly in QualityStage job flows to handle validation functions once you've parsed and standardized the data.

          Harald