Configuring Ontology

What is Ontology?

Ontology is a formal representation of the properties and relations between the data and entities. In Content Analyzer, the ontology refers to the definition and properties of document classes. The Ontology Builder is a no-code design tool where you can train the system to classify documents and identify key values that are important to you.

Click the Ontology icon at the left navigation bar to begin.

Regular backup of ontology

You must export the ontology periodically to back up your data for future reuse. Back up through Content Analyzer's Ontology Management Tool UI by clicking Export Ontology. You can also reference how to perform export of the ontology through the Content Analyzer API interface. Click API, select the Documentation tab for usage details. You can also try it out via SwaggerUI tab, and click Get /ontology. The Ontology data is returned in JSON format.

Configuring Ontology

Document class: Content Analyzer provides sample document classes, such as "Invoice" and "Power of Attorney". Document classes are the types of documents that you would like to classify. Each document class definition consists of titles, classification keys, key classes, and headings. Click Add Document Class to add a document class.

Titles: Common titles are found on a document class. Content Analyzer looks for titles that are defined (in bold, larger text, etc.) to classify documents. For example, accident reports might have titles such as Accident Summary and Police Report. To add a document title, click Add Document Title. You can write multiple words in a title. When the exact title is found in the document, the confidence score is higher in the document classification.

Classification: Common and unique keywords and words are found on a document class. For example, accident reports might have entities such as police, accident, and injury. Click Add Classification Word to add classification words. Enter only a single word in the classification. The more classification keys that are found in the document, the higher the confidence score is in the document classification.

Keys

A Key-Value Pair (KVP) is a set of two linked data items: a Key is a unique identifier for data. Value is associated data for the Key.

You need to add aliases to the key. For example, claim has the same semantic meaning as Claim ID, or Claim Num. This situation is called Semantic normalization, which maps the written variations of text labels within a document to consistent keys of the same meaning.

Aliases can be added in two ways: in the Ontology Builder and in the Documents detail page by clicking Learn. To add a symbolic name for a key (such as Claim No), click Add Key Class. Then, click the key class that was created. To add an alias (such as Claim Number to that key), click Add Key Alias. This step can be repeated multiple times to add different aliases for the key. You can enter multiple words in aliases.

Key Aliases allow users to extract TableLineItems, as they are used to identify column headers of a table in the TableList, under which Content Analyzer identifies TableLineItems.

The KVPs are extracted and included in the final JSON output.

Headings: If you have larger sections of text that you want to extract, you can specify headings. This process is similar to Key-Value Pairs. The name of the heading is Key. One or more paragraphs of text under the headings is Value. For example, Risk Snapshot is the heading (Key) and the paragraph of text, after the heading, is the Value. The output of headings extraction is stored in the JSON Output. The headings are not used to classify a document.

Patterns

Patterns are regular expressions that can be associated with a Key Class to help identify and extract keys and values. A regular expression is a sequence of characters that define a search pattern. The use of regular expression patterns is optional in Content Analyzer. Regular expression patterns can provide Content Analyzer extra information to potentially improve the accuracy in extracting the correct keys and values.

A Key Class can have zero or more patterns that are associated with it. Content Analyzer provides a set of sample regular expression patterns in GitHub (https://github.com/ibm-ecm/content-analyzer-regex) that you can copy and modify, with or without customization, in your Key Class definitions. You can also create custom patterns that are common in your documents.

Content Analyzer uses the Python syntax for regular expressions. For more information, see https://docs.python.org/3/library/re.html.

You can use patterns for Key Aliases or values. A particular pattern is used one time per document class, in the same way that Key Aliases are traditionally configured.

Key Alias patterns:
For a specific key, if the possible Key Aliases can differ, you can define multiple Key Aliases. You can also specify a single 'key alias pattern' that can match all the variations of your Key Aliases. For example, if the Key Alias can be Email or Email Address, you can create two separate Key Aliases or create just one Key Alias pattern that matches the variances in the key.
Value patterns:
For a specific key, if the potential values follow a rule that can be expressed in a regular expression, you can specify a 'value pattern'. This pattern can match all the variations of your values. For example, the expected value for a Start Date key might be in a date format. You can create a regular expression pattern for 'US Date' and then associate the value pattern of 'US Date' to your Key Class.
Value pattern with optional Key Alias:
Sometimes you want to extract a value that does not have a corresponding key in the document, but you know the pattern of the value. You can define the value pattern and denote that the value might be anywhere in the document without attaching to a Key Alias. This designation allows for the presence of a Key Alias to be optional. For example, you want to extract the employee ID number, which can be described with a regular expression pattern. However, some documents show the employee number with a key Employee ID:, while other documents show the employee number without a corresponding key. You can specify the pattern to be a 'Value pattern with optional Key Alias’ and be able to extract the employee ID number in both types of documents.

KVP detection and pattern matching

Content Analyzer's search for Key-Value Pairs that uses regular expression patterns is independent of other KVP functions.

If you do not define any patterns for a Key Class, Content Analyzer uses only user-defined Key Aliases to find Key-Value Pairs for that Key Class.
If you define a Key Alias, Content Analyzer does not use any Value patterns that might be defined.
If you do define patterns for a Key Class, to ensure that Content Analyzer has sufficient information to search and find KVPs:
- If you define a Value pattern, you must also define a Key Alias pattern.
- If you define a Key Alias pattern, you must also define a Value pattern.
- If you define a Value pattern with optional Key Alias, defining a Key Alias pattern is optional.