In IA 9.1, the built-in functions that can be used to define rules are fairly limited. For example, given an input string I want to check that it is a comma-separated list of code values that are valid (e.g., from a reference table of code values). Can I write a function that "splits" the input string with commas and checks if all returned values are in a reference table?
RobertDickson 0600009JMM137 Posts
Re: In IA 9, are functions used in defining rules user-definable?2013-11-12T23:49:18ZThis is the accepted answer. This is the accepted answer.
While there are quite a few rules that can be used, they are more along the 'validation' or 'test' side, and not the 'transformation' side. There are functions available when you create an output column, but I don't see ones that will give you exactly what you need. For your use case, you may want to create a database view that splits the input the way you want. There is no way I know of to create user-defined functions.
IBM support may be able to offer a better answer.
smithha 110000PAKN161 Posts
Re: In IA 9, are functions used in defining rules user-definable?2013-11-13T12:59:58ZThis is the accepted answer. This is the accepted answer.
- RobertDickson 0600009JMM
Depending on the exact requirements, you might be able to use the Matches_Regex check and put together a regular expression to check the strings for valid values. That's as close as you can get to user-defined functions at present. It would not support actual parsing of the data though and if you need that either a database view as Robert noted or use of a tool with strong parsing capabilities such as QualityStage would be the best options.
phuyn 270006T5D12 Posts
Re: In IA 9, are functions used in defining rules user-definable?2013-11-13T23:17:07ZThis is the accepted answer. This is the accepted answer.
- smithha 110000PAKN
Thanks Robert and Harald for your reply. My preference is to have the IBM tool provide the solution, instead of using the source database. The obvious benefit then is to keep all the assets in one place so that when the source platform changes, we don't have to recreate all those assets. Currently we don't have QualityStage, but before recommending the purchase of QualityStage, I want to make sure that it can give us the expected extensibility. Regular expression is good but its power is limited. Would QualityStage allow us to define rules with ANY logic we want?
smithha 110000PAKN161 Posts
Re: In IA 9, are functions used in defining rules user-definable?2013-11-14T14:18:50ZThis is the accepted answer. This is the accepted answer.
- phuyn 270006T5D1
QualityStage provides robust capability and rules to support:
- Parsing/standardization of data
- Deduplication/matching/linking of data
- Survivorship/consolidation of data
Parsing of a comma-separated list of values is readily handled with the Standardization rules, and you can build parsing/standardization rules for extremely complex data domains such as addresses, products, description text, etc. where the parsing is much less obvious. It can incorporate standardization or cleansing of subtle data issues through use of lookup tables and classifications of codes including fuzzy comparison to address subtle spelling issues. The standardization logic does not validate in and of itself.
You can use additional stages for data transformations with a broad array of functions to handle more of the validation side, or you can embed Information Analyzer data rules directly in QualityStage job flows to handle validation functions once you've parsed and standardized the data.