ApplicationObjects actions

Batch
- Document
- - Page
- - - Field

A Batch node can have a child Document, Page or Field. Documents can also have child Documents, Pages and Fields. Pages can only have child Pages, and Fields. Fields can only have child fields.

Page objects are usually a child of a Document and are named and arranged in a way that is meaningful for a specific application. One exception is the page type "Other". "Other" is a common page type in Datacap applications. As documents are ingested to create a batch, they are assigned the initial type of "Other". This type is used as a starting point for processing documents. Unlike other page types, the Other page is directly attached to the Batch node. Other application specific page types are typically attached to a Document object to group one or more pages or page types together in a meaningful way.

After files are ingested and assigned a type of Other, the next step is to identify and process the files. The "Other" pages are typically processed then assigned new page types that define a more detailed hierarchy that groups pages together under a Document.

Each page type can have a unique set of fields that indicate areas on the page where data will be extracted. A page typically defines the kind of page, such as an "Invoice", the fields define the type of data expected on a page such as Date, Vendor Name, Address, Invoice number, and other data expected on an Invoice.

It is also possible for a page to have different locations of data based on the document although the same page type is used. For example, this is commonly performed in the APT application where invoices are processed, while each invoice has the same types of data such as names, addresses, dates, line items, the location of this data can be different based on the fingerprint of the page. For example, in one invoice the date may be on the top left, another invoice may have the date in the top right, and still another may place the date somewhere else on the page. This allows an application to process the same type of page consistently although the pages and location of data on each page is different.

While there is only one Batch node, there can be multiple nodes of all of the other types. An application specific hierarchy can consist of any combination of these objects that defines the documents, pages and fields required to process a set of files as defined by specific use cases.

B MyApplication
- P Other
- D Invoice
- - P MainPage
- - - - F Date
- - - - F Invoice Number
- - - - F Invoice Amount
- - - - F Line Items
- - - - - F Quantity
- - - - - F Description
- - - - - F Price
- - P TrailingPage
- D Mortgage
- - P Agreement
- - - F Signature
- - - F Date

Each DCO node can have a "property" / "variable" that holds meta data about the node. These are defined directly on the node. For example, a page will have a variable that contains the name of the file associated with the page. There are some sets of variables that are standard however any number of additional user required variables can be created at any time.

A DCO node can contain any number of variables or children. While objects typically have a specific type of child, there are some exceptions. For example, a Page or a Field can be attached directly to a Batch node. Likewise, a Field node can in turn have additional children fields, as is required for line item processing.

Each application has a "Setup DCO" and a "Runtime DCO". A Setup DCO is the static hierarchy that defines the object layout of the application. It is called the "Setup DCO" because it is defined at application setup / design time. It defines one of all of the possible types of nodes that will be processed. The setup DCO is also used to define the relationship of each application object to the application rules that will be run on each object. For example rules can be attached to the "MainPage" object that will run on each object that has been identified as a MainPage. The same rules or different sets of rules can run on the TrailingPage.

The Runtime DCO is the dynamic version of the object hierarchy that directly reflects a specific batch that is being processed and contains all of the currently instantiated objects. For example, if there are 10 pages in the batch, there will be 10 page objects existing in the Runtime DCO. The types of these pages may be mixed. Perhaps there are 3 MainPages, 9 TrailingPages, all children of a set of 3 Invoice document objects and 1 Agreement page under a Mortgage document object. The runtime DCO will contain the values of each of the variables contained in each object node. At runtime, DCO nodes are created for the pages, documents and fields within the batch to reflect the unique files and data within the batch.

The primary focus of the Application Object action library is on manipulation of these DCO nodes. Every action library will read, act on, and can also update an object. The Application Object library is more general for updating, organizing and deleting DCO nodes themselves.

Access of variables and nodes is case sensitive but it is not recommended to use names and types for objects that differ only in case.

User variables can be configured in the setup DCO. Any object that has a variable preconfigured will automatically have that value unless it is changed at runtime. For example, a DCO node may be configured with a variable value so all objects of that type, will initially have the value set in the setup DCO, then specific nodes can have that variable updated at runtime, as required by specific application use cases.

Predefined DCO Variables

DCO Status

Each DCO has a status property. The status indicates if the data associated with the node is valid, invalid or indicates a particular state or problem. Verify panels are the primary place the status is used although actions may set the status or make logic decisions based on the status of an object.

The status is updated at run time depending on the object type and the rules and actions of the task. Some actions may update the status of a DCO object. It is also possible for a failed ruleset to automatically change the status of an object to fail, if the Status Preserve option is disabled.

Some common status values are shown in this table with their conventional meanings. Commonly used status values are -1, 0, 1, 2, 49. Other status are less common and may or may not be used as listed depending on the configuration and uses cases for the application.

DCO Type and Page Identification

Images and files are ingested using the scan or virtual scan task. During the ingestion, the pages are initially assigned the page type of "Other". This is a special page type that does not have any document structure and uses the batch as a parent. This is the starting point for processing of ingested files.

The next goal is to change the page type from "Other" to another meaningful page type. A DCO type cannot contain double quotes. Once assigned a meaningful type, the pages can be processed and manipulated based on the use cases for the application. While page identification is the next goal, it is not necessarily the next direct processing step. The next step can vary from application to application.

For example, if images are ingested as PDFs, word documents, emails or other types of formats, one of the use cases may be to recognize these pages so the text within them can be analyzed, verified and stored. In this situation, before performing page ID, these kinds of documents are typically converted into TIFF images. The final result is that the original source file, such as a PDF, is still in the batch, but now there are individual images for each page from within the parent file. Similar work is typically performed on emails or other types of documents that become a parent for multiple images, one new image for each page.

Another common step prior to page ID is to perform image enhancements on all or some of the images. If recognition is going to be performed and the image source isn't a nice clean document, such as a Word document, then the image may need some image adjustments. While leaving out many details from this description, typically operations like deskew, automatic rotation, line removal and other enhancements are performed to create clean images to provide to recognition. The result would be a clean document that has proper orientation and is not skewed. It is also sometimes useful to remove any lines from the document in this step.

The actual page ID process can be performed in a number of ways and varies based on specific use cases. One approach would be to perform fingerprinting after image enhancement. Fingerprint matching directly assigns a type ID to the page, changing it from "Other" to the correct type. The fingerprinting actions will look at a page, and the known fingerprints and match the page to the correct fingerprint, so now the page type is determined along with the locations of any fields on the page.

Using barcodes or patch codes are another technique for determining the type of a page and then assigning the page type with the SetPageType action. This could be a bar code directly on the page to identify, or a separator page that is scanned before the current page that indicates the page type of the following page.

Other types of identification can include performing recognition and identifying a page by specific text within the page that the application knows belongs to a specific type and then directly assigning the page type. If a page is blank, that may be assigned a different page type or possibly deleted or used as a separator.

Datacap provides a number of different kinds of actions that can be used to identify the page type and then assign it. Again, what is appropriate depends on the application use cases. Custom actions could even be used to use specific logic to figure out a page type and then assign it to the page. A page could also be provided to a classification engine, which will determine the correct page type and fingerprint.

Some applications use manual page identification with the Flex ID panel to have operators set page types by reviewing the document, or may simply supplement manual identification only for a smaller set of pages that could not be automatically identified. The actions are the tools to perform the identification process, how to arrange them to achieve the required page identification use cases is specific to an application. Most applications use some common techniques as described.

Once the Page ID rules have set new page types for each of the "Other" pages, then the "Create Documents" ruleset can be called which will automatically organize all of the pages into the appropriate page hierarchy, grouping pages together under a document node, or multiple document nodes, as defined by the application. Once page identification is done and the new document structure has been created, then processing of the batch can continue, but now that the page types are identified, then specific rules, and different rules, can be run on each object based on the DCO node level and the type.

After creating documents, the CreateFields action in the ApplicationObjects library will generate the field objects for a page. Initially these fields exist with no position information. The position information is then determined by loading the zones for the associated fingerprint or using actions to dynamically find zone locations using rules at runtime.

Once all the pages are identified and documents created, subsequent actions that have been defined on the objects will run to perform the specific operations required for each type of object within the DCO hierarchy.

Field Confidence

A field represents a specific type of data.. While it is not required, a field usually also has position information that identifies the location on the associated parent page object where the data exists. This position information is a representation of the pixel area on the page from where the text was recognized in a format of: left, top, right, bottom.

Field objects have the ability to store recognized text that is associated with the field. For example, if the field represents a date, the position would indicate the location of the text and the field's value would be the date, such as "01/01/2020".

Each character within the field's value also has an associated confidence level. The confidence level represents if the recognition engine thinks the value that was read is accurate or could have been misread. The confidence level is represented as a number between 1 to 10, where 1 means the engine is very unsure that the character is correct and 10 means the engine is very sure the number is correct. Typically a confidence value of 8, 9 or 10 are considered "high" confidence and that the engine was probably correct in reading the text. This default value is adjustable. If adjusted, it could be dropped to 7 or 6. It is rarely raised to 9 or 10.

Each of the characters has this confidence level. If any one of the characters has a confidence lower than the minimum confidence, it is highlighted to the verification operator when viewing the fields in the verify panel. The operator can visually confirm or correct any misread values.

There are other ways to confirm characters are correct, including mechanisms to automatically lower or raise confidence values based on one or more conditions. The ValidationsAndTextAdjustments action library is one action library that can perform automatic verification. Other mechanisms such as pattern matching or voting can also automatically adjust the confidence of characters within a field.

Note that in some situations, a character's confidence may be represented in a range from 0-9. Be sure to understand the correct range when working with confidence values.

Smart Parameters

Parameters that use an "@" notation, such as "@X.xxxx", "@P.xxxx", "@STRING()", "@PILOT()", etc., are known as "smart parameters". The data passed as a parameter to an action will be evaluated at runtime. For example, "@X" is notation for accessing the current object. "@P" accesses the page object, which could be the current object, or a parent object. Being evaluated at runtime, allows recognized data or other batch specific data to be passed to actions or stored as metadata. A smart parameter such as @DATE could be used to name export files based on the current date. Similar smart parameters will retrieve the current batch ID, current batch directory, current operator ID, and more. Smart parameters are a very powerful mechanism of getting data from other areas of the application and copying it into a new location or passing it as a parameter to an action. It is recommended that the smart parameter documentation is reviewed to understand the full capabilities available. The top-level help of the ValidationAndTextAdjustments action library also has information about smart parameters. Some characters such as +, \ and . also have special smart parameter meaning and may be interpreted as a smart parameter. This can be controlled using smart parameter syntax.