Identifying simple pattern classes

The subfield classes 1 to 9, and -1 to -9 are used to parse individual words of an ? string.

The number 1 represents the first word, the number 2 represents the second, the number –1 represents the last word, and the number –2 represents the next to last word. If the word referenced does not exist, the pattern does not match. If you are processing company names and only wanted the first word, a company name like WILLIAMS BIG SUPER CELL COMPANY matches to the following patterns (assume COMPANY is in the classifications (.CLS) as a type C).

You can combine single alphabetic classes (+) with subfield classes. For example, in a series of consecutive unknown tokens like CHERRY HILL SANDS, the following pattern causes the following match:

+ | -1

The + matches to the word CHERRY and the –1 matches to SANDS. The operand [1] is CHERRY and operand [2] is SANDS.

When matching to a pattern, you can specify a range of words.

The format is as follows:

(beg:end)

Examples are:

If you have the address 123 - A B Main St, you can use the following pattern:

^ | - | (1:2)
COPY [3] {HouseNumberSuffix}
RETYPE [2] 0
RETYPE [3] 0

This pattern results in A and B being moved to the {HouseNumberSuffix} (house number suffix) field. This pattern also retypes A and B to NULL tokens (and similarly retyping the hyphen) to remove them from consideration by any further patterns in the file.

You can combine the universal ( **) class with other operands to restrict the tokens grabbed by the class. The universal class can be null, signifying no tokens.

The class ** matches all tokens. For example, if you use a pattern of **, you match 123 MAIN ST and 123 MAIN ST, LOS ANGELES, CA 90016, and so on. The following pattern matches to all tokens before the type (which can be no tokens) and the type:

** | T

Thus, 123 N MAPLE AVE matches with operand [1] being 123 N MAPLE and operand [2] being AVE.

The universal class can be null. No tokens are required to precede the type. AVENUE also matches this pattern with operand [1] being NULL.

In the following pattern, the ** refers to all tokens between the numeric and the street type:

^ | ** | T

In the example, the class ^ (caret) and the type (T) class define the start and end of the ** class. The class ** can contain numbers that are in addition to the class ^ but not any additional street type tokens.

You can specify a range of tokens for an ** operand. For example, the following pattern matches a numeric followed by at least two nonstreet-type tokens followed by a street type:

^ | ** (1:2) | T

Operand [2] consists of exactly two nonstreet-type tokens. This matches 123 CHERRY TREE DR, but not 123 ELM DR. Only one token follows the number. You can specify ranges from a single token, such as (1:1), to all the tokens, such as (1:–1).

The pattern **(1:1) results in much slower processing time than the equivalent & to match to any single token. However, you do not want to use & in a pattern with **, such as ** | &, because the first token encountered is used by &. Value checks or appropriate conditions that are applied by using & with ** can make sense. For example:

** | & = "123", "ABC"

No conditional values or expressions are permitted for operands with **.

The $ specifier does not match any real token, but denotes the end of the pattern.

A pattern condition without the $ specifier can represent a portion of the field, such as the city, state, and postal code information in a U.S. address. For example, Littleton, MA 01460, and LITTLETON MA 01460-6245 match the following pattern:

? | S | ^

However, the hyphen (-) and the ZIP+4, 01460-6245, are not part of the match. To include the postal code, 01460-6245, as part of the match condition, use the pattern as follows:

? | S | ^ | - | ^ | $ 

Any input data following the postal code is not part of the match.

You use positioning specifiers to modify the placement of the pattern matching.

For the patterns documented so far, the pattern had to match the first token in the field. For example, the following pattern matches MAPLE AVE and CHERRY HILL RD, but does not match 123 MAPLE AVE, because a number is the first token:

? | T

You can use floating specifiers to scan the input field for a particular pattern. The asterisk (*) is a positioning specifier and means that the pattern is searched, from left to right, until there is a match or the entire pattern is scanned. You can use the asterisk (*) to indicate that the class immediately following is a floating class.

If you have apartment numbers in the address, to simplify your data, you might want to scan for, process, and retype the apartment numbers to NULL so that basic patterns can process the core address. For example, addresses such as 123 MAIN ST APT 34 and 770 KING ST FL 3 RM 101 contain a basic street address with additional information. The following pattern searches for the unit and floor information, populates the appropriate dictionary fields, and removes the unit and floor information from further processing. U is the class for unit and F is the class for floor.

*U | ^ 
COPY_A [1] {UnitType}
COPY [2] {UnitValue}
RETYPE [1] 0 
RETYPE [2] 0


*F | ^ 
COPY_A [1] {FloorType}
COPY [2] {FloorValue}
RETYPE [1] 0 
RETYPE [2] 0

Retyping the tokens to NULL removes the tokens from consideration by any patterns later in the Pattern-Action file. You prevent recounting all combinations of possibilities. Now, the data to be processed is 123 MAIN ST and 770 KING ST. Both entries have the pattern: ^ | ? | T.

Processing a portion of the data by using the floating specifier simplifies the two input fields and makes the input pattern the same. The standardization task is made easier.

Floating positioning specifiers operate by scanning a token until a match is found. If all operands match, the pattern matches. If the operands do not match, the scanner advances one token to the right and repeats the process. This is like moving a template across the input string. If the template matches, the process is done. Otherwise, the template advances to the next token.

The asterisk must be followed by a class. For example, the following operands are valid with a floating positioning specifier followed by a standard class:

* U
* ?
* ^

*^ | ? | *T

Operand [1] is 123. Operand [2] is CHERRY HILL. Operand [3] is RD. NORTH is classified as a directional (D) so it is not included in the unknown string (?).

The reverse floating positioning specifier, indicated by a number sign (#), is similar to the floating positioning specifier (*) except that scanning proceeds from right to left instead of from left to right.

You can use this specifier to search for items that appear at the end of a field, such as postal code, state, and apartment designations.

The reverse floating positioning specifier must only appear in the first operand of a pattern, since it is used to position the pattern. For example, if you wanted to find a postal code and you have given the state name the class S, the following pattern scans from right to left for a state name followed by a number:

#S | ^

If you have an input string CALIFORNIA 45 PRODUCTS, PHOENIX ARIZONA 12345 DEPT 45, the right to left scan positions the pattern to the ARIZONA. The number following causes a match to the pattern.

If no match is found, scanning continues to the left until a state followed by a number is found. If you are limited to the standard left-right floating positioning specifier (*S | ^), the CALIFORNIA 45 is incorrectly interpreted as a state name and postal code.

The fixed position specifier is positioned at a particular operand in the input string.

Sometimes it is necessary to position the pattern matching at a particular operand in the input string. This is handled by the %n fixed position specifier. Examples of the fixed position specifier are:

The positions can be qualified by following the %n with a token type. Some examples are:

You can use the fixed position specifier (%) in only two ways:

• As the first operand of a pattern
• As the first and second operands of a pattern

The following pattern is allowed and matches the second numeric token as operand [1] and the third leading alpha token that follows as operand [2]:

%2^ | %3<

The fixed position specifier treats each token according to its class. The following examples illustrate how to use the fixed position specifier for the input field:

John Doe 
123 Martin Luther St 
Salt Lake

The position specifier does not continue scanning if a pattern fails to match (unlike * and #).

Assuming the input value S is classified as a D for direction, the following pattern matches the 789 S in the string 123 A 456 B 789 S:

%3^ | D

That same pattern does not match 123 A 456 B 789 C 124 S because the third number (789) is not followed by a direction.

The exclamation point (!) is used to indicate NOT.

The following pattern language syntax shows how to use the negative to specify matching:

The following example matches to SUITE 3, APT GROUND but not to SUITE CIRCLE because CIRCLE is classified as a street type (T):

*U | !T

The phrase RT 123 can be considered to be a street name only if there is no unknown word following, such as RT 123 MAPLE AVE. You can use the following pattern to create a non-match to the unknown word:

*T | ^ | !?

This pattern matches to RT 123, but not to RT 123 MAPLE AVE because an unknown alphabetic character follows the numeric operand.

You can combine the negation class with the floating class (*) only at the beginning of a pattern. For example, when processing street addresses, you might want to expand ST to SAINT where appropriate.

For example, change 123 ST CHARLES ST to 123 SAINT CHARLES ST, but do not convert 123 MAIN ST REAR APT to 123 MAIN SAINT REAR APT. You can use the following pattern and action set:

*!? | S | +
RETYPE [2] ? "SAINT"

The previous example requires that no unknown class precede the value ST because tokens with this value have their own class of S.