Grooper.Core.DataPattern

Directions=N|S|W|E
Suffix=road|street|boulevard|circle

This lexicon defines variables named @Direction and @Suffix which can be used in the regular expression:

(@Directions) [a-z]+ (@Suffix)

At run time, this expands to the following regular expression:

(N|S|W|E) [a-z]+ (road|street|boulevard|circle)

• LeastCost: Matches the least cost (highest confidence) result. If the confidence of two results are equal, then the longer result will be matched. If both results also have the same length, then the item appearing first in the text flow will be matched. For example, let’s say you have a pattern [0-9]{5,6} and you have a value of 10000O. You should get a hit on 10000 at 100% and a hit on 100000 (depending on your Fuzzy Match Weightings) at a lower percentage. In LeastCost mode, you will only capture the 10000 because it has a higher confidence than converting the O to a 0.
• BestValue: Matches the longest result which is above the minimum confidence. If two overlapping results have the same length, then the result with the highest confidence will be matched. If both results have equal confidence, then the result appearing first in the text flow will be matched. For example, let’s say you have a pattern [0-9]{5,6} and you have a value of 10000O. You should get a hit on 10000 at 100% and a hit on 100000 (depending on your Fuzzy Match Weightings) at a lower percentage. In BestValue mode, if it is above the minimum confidence, your return value will be 100000 because it is the longer of the 2 values.

• RegEx: Normal regular expression mode. Finds all instances which are a 100% match for the regular expression. RegEx mode supports the full syntax and feature set of Microsoft .Net Framework Regular Expressions.
• FuzzyRegEx: Fuzzy regular expression mode. Finds all instances which match the regular expression to a specific percentage of similarity. FuzzyRegEx mode supports most of the syntax and features of RegEx mode, with a handful of exceptions noted below.

  Processing time for FuzzyRegEx is proportional to the perplexity of the regular expression. Perplexity is the number of possible permutations of the pattern, which in turn defines the number of passes which must be made through the content.

  For example, A{1,2}B{1,2}C{1,2} has a perplexity of 2 * 2 * 2 = 8. (ABC, ABCC, ABBC, ABBCC, AABC, AABCC, AABBC, AABBCC). If we allow each element to range from {1,4}, the perplexity is 4 * 4 * 4 = 64. The expression [0-9]{4} (miles|kilometers) has a perplexity of 2. If we change it to [0-9]{1,5} (miles|kilometers), the perplexity becomes 10.

  Note that FuzzyRegEx supports an option which is unavailable in RegEx. (?r) will turn on required mode, and (?-r) will turn it off. At the start of a FuzzyRegEx, required mode always defaults to off. Once turned on, required mode will stay on until it is turned off. This mechanism can be used, for example, to require the start of a new line. The syntax to accomplish this would be be (?r)\n(?-r).

  The following regular expression features are NOT supported in FuzzyRegEx mode:

  • Quantifiers : + and *. Also, the 'few times as possible' construct (i.e. \w*?) is not supported.
  • Character Escapes : \a \b \e \nnn \cX \cx \unnnn.
  • Character Classes : \p{name} abd \P{name}
  • Grouping Constructs : Only basic named and unnamed group constructs are supported
  • Other: Anchors other than ^ and $, Backreference Constructs, and Alternation Constructs are unsupported.

• FuzzyList: Fuzzy list mode. Finds all instances which match an entry in the vocabulary. In this mode, no Value Pattern is required, since the values to be matched are specified as lexicon entries. To configure fuzzy list mode, edit the Lookup Options as follows: (1) define one or more vocabulary entries; and (2) specify the required match percentage. The extractor will return all instances which match an entry in the lexicon to the specified degree of similarity.

• Input: The quick brown fox jumped over the log.
• Pattern: \w+
• Output - nGram Size 1: The, quick, brown, fox, jumped, over, the, log
• Output - nGram Size 3: The quick brown, quick brown fox, brown fox jumped, fox jumped over, jumped over the, over the log.

• nGrams - Applies to all components of the nGram.
• nGram1 - Applies to component 1 of the nGram.
• nGram2 - Applies to component 2 of the nGram.
• nGram3 - Applies to component 3 of the nGram.
• nGram4 - Applies to component 4 of the nGram.
• nGram5 - Applies to component 5 of the nGram.

The output format can contain (a) literal characters and (b) placeholders for groups captured in the regular expression. Placeholders take the general form {GroupName}, and can be expanded to include a typecast and format {GroupName:TypeCast:FormatSpecifier}. Examples:

• {LastName}, {FirstName} - Outputs 'Smith, John' in a case where the value of LastName is 'Smith' and the value of FirstName is 'John'.
• {ItemNo:Integer:0000} - Outputs '0192' in a case where the value of 'ItemNo' is '192'.

Valid typecasts include DateTime, Decimal, Double, Integer, and String. If an extracted value cannot be converted to the specified type, the value will be excluded from the output. Two special typecasts are provided to assist with translation of values captured with the @Number and @Alpha variables. A typecast of 'Number' will convert all alpha characters which resemble numbers to their numeric equivalents. A typecase of 'Alpha' will perform the exact inverse of this operation, converting all numeric characters which resemble alpha characters to their alpha equivalents.

The GroupName must reference a named group defined within the regular expression, be limited to the [0-9A-Z_] character set, and it's length cannot exceed 64.

A valid .Net format specifier for the type indicated in the typecast. Please see the following links for complete documentation:

• Standard DateTime Formats
• Custom DateTime Formats
• Standard Numeric Formats
• Custom Numeric Formats

Commonly-Used Format Strings

Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday

Referencing this lexicon will define a variable named @Weekdays which can be used in the regular expression:

(@Weekdays), June \d+

At run time, this expands to the following regular expression:

(Sunday|Monday|Tuesday|Wednesday|Thursday|Friday|Saturday), June \d+

• [0-9]{5} will find all numeric values with a length of 5 characters.
• [0-9]{5,8} will find all numeric values with a length of 5 to 8 characters.
• [A-Z]{3,12} will find all alpha values with a length of 3 to 12 characters.
• [0-9A-Z]{6} will find all alphanumeric values with a length of 6.