Managing advanced rules projects and extractors

Defining a list

When using standardized terms such as names of departments in your organization, sales priorities, congressional committees, or military ranks to extract values, you can identify the needed terms in a list.

1. Click on the canvas toolbar.
2. Specify a name for the dictionary. You can extend the name with a description. Lengthy text might be truncated in the display but you can hover over the dictionary to read the full description.
3. On the Settings tab under Extractor Properties, enter each term manually or load a list from a file. The maximum length for each entry is 1024 characters.
   • To specify the terms manually, click , enter the term, and press Enter. Enter one term per row.
   • To load terms from a UTF-8 text file with each term on a separate line, . If an error is displayed during the import, ensure that the file does not contain unprintable characters such as Line Feed or Backspace.

Defining a mapping table

When analyzing text that contains synonyms, alternate spellings for a term, or encoded values, you can define a mapping table to map terms in the text to a desired term.

A mapping table is analogous to a lookup table or a key-value pair table. Examples of instances in which you might define a mapping table include matching:

• Nicknames like Maggy and Peggy for Margaret
• Abbreviations and alternate forms of an organization name such as I.B.M., IBM Deutschland and International Business Machines for the commonly used name IBM
• 404 to File Not Found when translating HTML error codes in a log file
• The area code 408 to California

For best performance, use long or complex mapping tables in a second step, after the text extraction step.

1. Click on the canvas toolbar.
2. Specify a name for the dictionary. You can extend the name with a description. Lengthy text is truncated but is displayed when you hover over the dictionary.
3. On the Settings tab under Extractor Properties, click Map Terms.
4. Complete the two-column table manually or by loading from a file. The first column contains the terms to search for in the text and the second column lists the values to generate in the results. The maximum length for an entry is 1024 characters. If a term is repeated in the first column, only the first mapped value is used.
   • To specify the terms manually, click , enter the term and its mapped value, and press Enter.
   • To load terms from an ASCII or UTF-8 text file with each term and its mapped value, separated by a single comma, on a separate line, click . If an error is displayed during the import, ensure that the file does not contain unprintable characters such as Line Feed or Backspace.

Regular expression examples

The following are examples of regular Java expressions that might be used in specific instances. In the pattern descriptions, A represents a character and 9, a digit. For more information about Java syntax, see Class Pattern Javadoc.

• To select text that includes any ordinal number of one or more digits followed by th, st, nd, or rd (21st, 2nd, 3rd, and so forth), specify:

  \d+(st|nd|rd|th)
  

• To select text that includes US Social Security numbers formatted as 999-99-9999, specify:

  \d{3}\-\d{2}\-\d{4}
  

• To select all text that includes a Canadian postal code formatted as A9A-9A9, A9A 9A9 or A9A9A9 (for example, K1G 3K9, V5g-4X3, and x2H3m5), specify:

  [a-zA-Z][0-9][a-zA-Z](-|)[0-9][a-zA-Z][0-9]
  

• To select all text that includes a United Kingdom postal code formatted as A9 9AA, A99 9AA, AA99 9AA, A9A 9AA, or a9d AA9A 9AA (for example, M1 1AE, B33 8TH, DN55 1PT, W1A 0AX, CR2 6XH, and SW1A 2AA), specify:

  [A-Z]{1,2}[1-9][0-9]?[A-Z]?\s[0-9][A-Z]{2,}|GIR 0AA
  

• To select all text that includes number plates for private and commercial vehicles in India formatted as AA-99-AA-9999, AA-99-AAA-9999 and, for states where the 0 is optional, AA-9-AA-9999 with separators that may be hyphens, spaces or not present (for example, TN-86-AF-1199, WB 06 F 5971, and DL4CAF4943), specify:

  [A-Z]{2}(-| |)\d{1,2}(-| |)[A-Z]{1,3}(-| |)\d{4} (M1 1AE, B33 8TH, DN55 1PT, W1A 0AX, CR2 6XH, SW1A 2AA)
  

Sequence pattern example

To select references to military personnel:

1. Create a dictionary called Military Ranks that includes terms such as Warrant Officer, Sergeant, and Lieutenant.

2. Drag the Person extractor onto the canvas following the Military Ranks dictionary to indicate that the new sequence finds ranks then names. The dark blue line after the Military Ranks extractor indicates placement of the Person extractor in the text pattern.

   

   When you release the mouse button, a sequence object is displayed on the canvas.

3. Click the sequence object on the canvas and, under Extractor Properties, open the General tab. Rename the sequence object as Military Personnel.

4. The final extractor rule is displayed on the canvas.

   

Adding proximity rules

Proximity rules to specify the maximum number of tokens that might occur between the desired terms. In text, each word or character is generally referred to as a token. The sentence "The CEO announced the earnings last week, missing analyst estimates by 5%." contains 15 tokens: 11 words separated by spaces, 3 special characters, and 1 number.

You might want to locate references to red brick houses, knowing that phrases such as house made of red brick, red house of brick and the brick on the house reflected red might appear in the text you are analyzing. Proximity rules can ensure that these phrases are extracted while excluding sentences or phrases that coincidentally include the target terms, for example, "The red car passed a house near the brick yard".

1. Right-click an extractor on the canvas and under one of the Add submenus, click Proximity Rule.
2. In the format minimum-maximum, specify the minimum and maximum number of words, special characters, or values to allow between terms. By default, the proximity pattern is set to match the number of tokens between the two numbers you specify in your range. If the desired proximity does not vary, you can change the setting. For example, enter 0-2.
3. Select the new proximity rule. The Extractor Properties opens to the Settings tab where you can change the number range or convert to an exact number of tokens.

Example 1:

1. Create a dictionary called Clerical title that includes terms such as Rabbi, Father, and Archbishop.

2. Drag the Person extractor to right side of the Clerical title dictionary so that the new sequence will find titles followed by names. When you release the mouse button, a sequence object is displayed on the canvas.

3. Right-click Clerical title and click Add After > Proximity Rule. To capture terms such as Archbishop of Canterbury, Robert Runcie, specify the minimum and maximum number of tokens between words, in this case 0-5.

4. Click the sequence on the canvas and then under Extractor Properties, open the General tab if it doesn't open automatically. Rename the sequence Clerics.

   

Example 2:

As a second example, select tweets that refer to Twitter names of industry analysts with a big data term. To accomplish this, create two dictionaries, one of Twitter usernames of analysts and a second of big data terms and combine them on the workspace canvas with a proximity of one to 25 tokens.

![](images/analyst.jpg "Analyst Twitter extractor")

Union example

The term Parties in contracts can refer to two individuals, two organizations, or an individual and an organization. In order to extract information about the parties, you must define two extractors, Party1 and Party2, to select either an individual or an organization.

1. Drag both the Person and the Organization extractors from the Extractor pane onto the canvas.

2. On the canvas, select each extractor in turn and on the Output tab of the Extractor Properties, rename the output columns so that the column definitions are the same for both.

   

3. Drag the Organization extractor to the drop zone below the Person extractor.

   

4. Select Union 1 on the canvas and name it Party 1 on the General tab of Extractor Properties.

5. Copy Union 1 to create a second union and name it Party 2.

6. To look for the parties in the contracts, combine the two unions in a sequence as shown here. Between the two unions in the sequence is the literal and.

   

7. Name the new sequence Legal Parties.

Renaming a column in the results display

1. On the canvas, right-click the extractor that generated the results and click Edit Output.
2. From the column menu, select Rename or simply double click the column.
3. Enter the new column name to be displayed in the results.

Adding a string column

1. On the canvas, right-click the extractor that generated the results and click Edit Output.
2. Click the Manage Columns menu in the left column of the table.
3. Click New Column.

Adding a transformed output column

You can transform the extractor output, for example, converting it all to lowercase. Do these transformations on the Output tab of the Extractor Properties pane.

1. On the canvas, right-click the extractor that generated the results and click Edit Output.
2. Click the drop-down menu in the header of the column that you want to transform and select the type of transformation that you want to do.

If you apply Trim, New Column From Single Column, or New Column From Two Columns to a selected column and then subsequently select Convert to String or Convert to Lowercase String, you will not get yet another new column. The string transformations will be applied instead to the new column that was generated from the previous Trim or New Column choices.

Hiding a column from the results display

You can hide output columns on the Output tab of the Extractor Properties pane.

1. On the canvas, right-click the extractor that generated the results and click Edit Output.
2. Click the Manage Columns menu in the left column of the table.
3. Clear the check boxes for the columns you want to remove from the results display. These columns are hidden from the results, although the content is still extracted.

Deleting a column from the results display

1. On the canvas, right-click the extractor that generated the results and click Edit Output.
2. Click the Manage Columns menu in the left column of the table.
3. Click Delete Column and select the check boxes for the columns you want to remove from the results display.

Removing duplicate and overlapping results

An extractor sometimes generates multiple rows for the same text because the text matches more than one dictionary entry. To remove or consolidate the duplicate entries, you must provide consolidation rules.

1. On the canvas, right-click the extractor that generated the results.
2. In the Extractor Properties pane, click Output.
3. Select Manage overlapping matches.
4. From the Output column list, select the column that is causing the extractor to generate the duplicate output.
5. From the Method list, select one of the following values:
   • Contained Within to keep the longest result.
   • Not Contained Within to keep the shortest result.
   • Contains But Not Equal to keep unique results of the same length.
   • Exact Match to keep one instance of each result.
   • Left to Right to keep the longest result, with the greatest number of terms from left - to right.

As an example, you are designing an extractor for documents that include references to military personnel. After you create and run a dictionary containing a list of military ranks, such as Chief Warrant Officer, Warrant Officer, and Officer, you see the following results for Chief Warrant Officer John Doe:

• Chief Warrant Officer John Doe
• Warrant Officer John Doe
• Officer John Doe

The terms Warrant Officer and Officer are found in the longer term Chief Warrant Officer, causing Chief Warrant Officer John Doe to satisfy all three dictionary entries and generate three rows of results. To retain the longest term, which represents the most significant rank, you specify the consolidation settings as follows.

1. From Output column select Military Rank
2. From Method select Contained Within

Your results now include only the row Chief Warrant Officer John Doe.

Filtering extraneous results

An extractor sometimes produces unwanted results even though the pattern is a match. These unwanted results can often be recognized by other text near the match in the document.

To further restrict matches, create a filter.

1. On the canvas, right-click the extractor that generated the results and click Edit Output. The Output tab in Extractor Properties opens.
2. Choose to Include or Exclude results. Include is the default setting.
   • An include filter is a condition that must be met for a result to be included.
   • An exclude filter is a condition that must not be met for a result to be included.
3. From the first filter drop-down list, select the column that you want to apply the filter to.
4. In the next drop-down list, select the type of filter. Depending on the filter, the subsequent drop-down lists reflect the appropriate options.

5. Once you complete your filter options, execute the extractor again to see how the results have changed.

Your Military Ranks extractor might produce a match for text Chief Warrant Officer John Doe, but you do not want to include results that have the word except preceding the match.

1. Create a dictionary with the term except and any other terms that you might want to use to exclude results.
2. Select Military Ranks from the canvas and open the Output tab under Extractor Properties.
3. Click New Filter.
4. Select Exclude and choose a column.
5. Select range and occurs after.
6. Select your dictionary that includes except.
7. Select the column and between 0 to 2 tokens.

This filter excludes any matches that have the word except within 0-2 tokens before a match.

![](images/samplefilter.png "Sample filter")