Example: Creating a custom enrichment manually

You can create a custom enrichment manually by creating and importing a field type that includes the enrichment. In this example, you create custom date and currency converters.

About this task

The designer provides default converters for field values like dates or currency. In some instances, you might need to customize the way this enrichment work for the values that are included in your typical documents. For example:

You can use the date converter that is provided by default on the sys.Date field type to convert a date in the format %Y-%m-%d in the US/Eastern time zone to the same format in Coordinated Universal Time (UTC). However, a custom date converter is needed if the date in the document is in a different format or belongs to a different time zone.
Similarly, you can use the currency converter that is provided by default on the sys.CurrencyCodeObjType object type to convert the seven predefined currency symbols to a required currency code and then check for a match at the start of the value. A custom currency converter is needed if the currency code in the document is different from the pre-defined seven code or if the currency code is present at the end of the text instead of the start.

To accommodate a situation like the described examples, do the following high-level steps to add the necessary converter or formatter to the field:

Define a JSON file for a new Field Type with the required set of converters or formatters.
Import the JSON file to create the new Field Type.
Create a new field, or update an existing field to belong to the Field Type that you imported. From the enrichment settings, choose the new converter or formatter that you defined for the field type.

Instead of creating a new Field Type with just the converter or formatter that is required, it might be useful to extend an existing provided Field Type and re-add all its existing converters, formatter, and validators so that the new Field Type preserves the implementations from the default Field Type. To understand the JSON structure of an existing object type, look at the caDefinitions.json file in the project folder of your downloaded archive.

Procedure

Create the new JSON file, based on a copy of the JSON file for an existing field type that includes the new converter:
1. Make a copy of a similar field type JSON file from the downloaded archive.
2. Remove all GUID values, for example objectTypeId, converterId, formatterId, validatorId, and extractorId.
  This prevents a conflict with the field types that are already established in the project.
3. Add the new segment in the relevant section.
  - To add a new converter, add a new entry into the converterList array.
  - To add a new formatter, add a new entry into the formatterList array.
  For this example, two new converters are added.
  New date converter:
```
{
    "name": "Custom Date Converter",
    "symbolicName": "CustomDateConverter",
    "description": "Custom Date Converter that converts required date seen as - %A, %B %d, %Y at %I:%M:%S %p",
    "tool": "dateconverter",
    "format": "%A, %B %d, %Y at %I:%M:%S %p",
    "timezone": "America/New_York"
}
```
  New currency converter:
```
{
    "name": "Custom Currency Converter",
    "symbolicName": "CustomCurrencyConverter",
    "description": "Custom Currency Converter that converts code at the start of the value to corresponding symbol.",
    "tool": "map",
    "wordMap": {
        "USD": "$",
        "CAD": "Can$",
        "AUD": "AU$",
        "GBP": "£",
        "EUD": "€"
    },
    "positionMatch": "^"
}
```
  The following parameters apply:
  name
  
  Describes the name of the converter.
  
  symbolicName
  
  Describes a unique name for the converter under the object type. Only alphabets and numerics are allowed.
  
  description
  
  Text that describes the converter.
  
  tool
  
  Describes the type of converter.
  
  Use the dateconverter value for the custom date converter.
  
  Use the map value for the custom currency code converter.
  
  format
  
  Applies for and specifies the format in which the text for date is seen in the document, based on python formatting symbols. For details, see the Python Datetime page.
  
  timezone
  
  Applies for the dateconverter value only and specifies the time zone of the date in the document. Supports all possible time zones. For more information, see the pytz-time-zones.py GitHub page.
  
  wordMap
  
  Applies only for map converters. Specifies the dictionary in this case:
  
  The dictionary's key specifies the match that is needed on the value of the key-value pair.
  
  And the dictionary's value specifies that the new text that should replace the matched text in the value of the key-value pair.
  
  For example, "USD": "$" means that when there is a match of characters USD in the value, replace them with $ when the converter is applied.
  
  positionMatch
  
  Applies only for map converters. Specifies how the dictionary's key in the wordMap value might be matched in the value of the key-value pair. Allowed values are ^, $, * , representing that the match is either at the start OR at the end OR anywhere in the value of the key-value pair. This field is optional and when it is not specified, the map converter does a full match of the key that is specified in the wordMap value.
4. Ensure that the file has the following components at the top level.
  - The jsonSchemaVer parameter is set to 3.
  - The objectTypeList array is defined, including the object type to be created. For example, if you create both the object types in a single JSON, this list has two entries.
  The example creates Custom Date as a subtype of sys.Date and Custom CurrencyCode Object Type as a subtype of CurrencyCode Object Type. The subtype field refers to the respective parent.
Import the new field type into the project:
1. Click the Enrich tab, then click Field types and enrichments.
2. Click Import library, and select the JSON file that you created.
  A message confirms that the import was successful. A new entry displays in the Field type libraries list with the name for the new field types, for example, Custom library.
3. Click the Custom library to see the two object types as defined in the JSON file.
4. Edit each of the object types and verify that the converters exist as specified in the JSON.
Create a new field type with the required converter:
1. Create a new Field for the relevant document type.
2. Select the newly imported object type as the Field type for the field.
3. In Value settings > Value format, select Edit > Converters > Add converter.
4. Click Select existing.
5. From the drop-down list, select the custom converter that you created.
  After you add your converter, you can use the Test panel to verify how the input field value is modified by the enrichment.
6. Save the field.

Results

When documents of this type are processed at run time, the value that is extracted for this field is converted to the required date or currency format.

Example

The following example illustrates the format and contents for a Field Type JSON file:

{
    "jsonSchemaVer": 3,
    "objectTypeList": [
        {
            "name": "Custom Date",
            "symbolicName": "CustomDate",
            "description": "Custom Date",
            "scope": "custom",
            "subtypeOf": "sys.Date",
            "cardinality": "single",
            "delimiter": "",
            "aliasList": [],
            "valueFormats": {
                "text": {
                    "extractorSequence": null,
                    "converterSequence": null,
                    "formatterSequence": null,
                    "converterList": [
                        {
                            "name": "Date Converter",
                            "symbolicName": "DateConverter",
                            "description": "Date Converter",
                            "tool": "dateconverter",
                            "format": "%Y-%m-%d",
                            "timezone": "US/Eastern"
                        },
                        {
                            "name": "Custom Date Converter",
                            "symbolicName": "CustomDateConverter",
                            "description": "Custom Date Converter that converts required date seen as - %A, %B %d, %Y at %I:%M:%S %p",
                            "tool": "dateconverter",
                            "format": "%A, %B %d, %Y at %I:%M:%S %p",
                            "timezone": "America/New_York"
                        }
                    ],
                    "formatterList": [
                        {
                            "name": "Clean up text",
                            "symbolicName": "TextCleanupFormatter",
                            "description": "Text Cleanup Formatter",
                            "tool": "textcleanupformatter"
                        },
                        {
                            "name": "Remove extra spaces",
                            "symbolicName": "WhitespaceFormatter",
                            "description": "Remove all extra spaces",
                            "tool": "whitespaceformatter"
                        }
                    ]
                }
            },
            "mandatory": false,
            "sensitive": false,
            "validatorList": [
                {
                    "name": "Datatype Mismatch Validator",
                    "symbolicName": "DatatypeMismatchValidator",
                    "description": "Datatype Mismatch Validator",
                    "tool": "datatypemismatchvalidator",
                    "severity": "Error",
                    "failureReason": "Value is not of the supported date format (YYYY-MM-DD)"
                },
                {
                    "name": "Low Confidence Validator",
                    "symbolicName": "LowConfidenceValidator",
                    "description": "Low Confidence Validator",
                    "tool": "confidencevalidator",
                    "threshold": 80,
                    "severity": "Warning",
                    "failureReason": "Value confidence lower than the the specified threshold"
                },
                {
                    "name": "Required Value Validator",
                    "symbolicName": "RequiredValueValidator",
                    "description": "Required Value Validator",
                    "tool": "requiredvaluevalidator",
                    "severity": "Error",
                    "failureReason": "Required value is missing"
                },
                {
                    "name": "Near to Current Date Validator",
                    "symbolicName": "NearToCurrentDateValidator",
                    "description": "Near To Current Date Validator",
                    "tool": "neartocurrentdatevalidator",
                    "numDays": 0,
                    "condition": ">=",
                    "severity": "Warning",
                    "failureReason": "Value should have Number of days (>, >=, <, <=) to the current server date"
                },
                {
                    "name": "Date Value Check Validator",
                    "symbolicName": "DateValueCheckValidator",
                    "description": "dateValueCheckValidator",
                    "tool": "datevaluecheckvalidator",
                    "operator": ">=",
                    "operand": "1970-01-01",
                    "severity": "Warning",
                    "failureReason": "Date should be (==, !=, >, >=, <, <=) to the operand"
                },
                {
                    "name": "Date Value Range Validator",
                    "symbolicName": "DateValueRangeValidator",
                    "description": "Date Value Range Validator",
                    "tool": "datevaluerangevalidator",
                    "minimumDateValue": "1970-01-01",
                    "maximumDateValue": "9999-12-31",
                    "severity": "Warning",
                    "failureReason": "Date should be between the specified minimum and maximum date value"
                }
            ]
        },
        {
            "name": "Custom CurrencyCode Object Type",
            "symbolicName": "CustomCurrencyCodeObjType",
            "description": "Custom Currency code object type",
            "scope": "custom",
            "subtypeOf": "sys.CurrencyCodeObjType",
            "cardinality": "single",
            "delimiter": "",
            "aliasList": [],
            "valueFormats": {
                "text": {
                    "extractorSequence": null,
                    "converterSequence": null,
                    "formatterSequence": null,
                    "converterList": [
                        {
                            "name": "Currency Converter",
                            "symbolicName": "CurrencyConverter",
                            "description": "Currency Converter",
                            "tool": "map",
                            "wordMap": {
                                "C$": "CAD",
                                "Can$": "CAD",
                                "AU$": "AUD",
                                "A$": "AUD",
                                "£": "GBP",
                                "€": "EUD",
                                "$": "USD"
                            },
                            "positionMatch": "^"
                        },
                        {
                            "name": "Custom Currency Converter",
                            "symbolicName": "CustomCurrencyConverter",
                            "description": "Custom Currency Converter that converts code at the start of the value to corresponding symbol.",
                            "tool": "map",
                            "wordMap": {
                                "USD": "$",
                                "CAD": "Can$",
                                "AUD": "AU$",
                                "GBP": "£",
                                "EUD": "€"
                            },
                            "positionMatch": "^"
                        }
                    ],
                    "formatterList": [
                        {
                            "name": "Remove punctuation",
                            "symbolicName": "RemovePunctuation",
                            "description": "Remove Punctuation",
                            "tool": "regexformatter",
                            "stringToFind": "'.?!,;:-[]{}()",
                            "replaceWith": "",
                            "positionMatch": "*"
                        },
                        {
                            "name": "Clean up text",
                            "symbolicName": "TextCleanupFormatter",
                            "description": "Text Cleanup Formatter",
                            "tool": "textcleanupformatter"
                        },
                        {
                            "name": "Remove extra spaces",
                            "symbolicName": "WhitespaceFormatter",
                            "description": "Remove all extra spaces",
                            "tool": "whitespaceformatter"
                        }
                    ]
                }
            },
            "mandatory": false,
            "sensitive": false,
            "validatorList": [
                {
                    "name": "Datatype Mismatch Validator",
                    "symbolicName": "DatatypeMismatchValidator",
                    "description": "Datatype Mismatch Validator",
                    "tool": "datatypemismatchvalidator",
                    "severity": "Error",
                    "failureReason": "Value is not a valid 3 letter ISO 4217 standard currency code"
                },
                {
                    "name": "Low Confidence Validator",
                    "symbolicName": "LowConfidenceValidator",
                    "description": "Low Confidence Validator",
                    "tool": "confidencevalidator",
                    "threshold": 80,
                    "severity": "Warning",
                    "failureReason": "Value confidence lower than the the specified threshold"
                },
                {
                    "name": "Required Value Validator",
                    "symbolicName": "RequiredValueValidator",
                    "description": "Required Value Validator",
                    "tool": "requiredvaluevalidator",
                    "severity": "Error",
                    "failureReason": "Required value is missing"
                },
                {
                    "name": "Value Length Validator",
                    "symbolicName": "ValueLengthValidator",
                    "description": "Value Length Validator",
                    "tool": "valuelengthvalidator",
                    "minimumLength": 3,
                    "maximumLength": 3,
                    "severity": "Warning",
                    "failureReason": "Value length should be between the specified minimum and maximum length"
                }
            ]
        }
    ]
}