Home
Description
Publications

Available Resources
Text Acknowledgements
Related links


Events


CLaRK System

CLaRK System Online Manual


Bulgarian dialects'
electronic archive


Pragramme in Computational Linguistics



 

 

 

 

 

 

 

Menu Definitions

Tokenizers

XML considers the content of each text element a whole string that is unacceptable for corpus processing. For this reason, it is required for the word-forms, punctuation and other tokens in the text to be distinguished. In order to solve this problem, the CLaRK System supports a user-defined hierarchy of tokenizers. At the very basic level users can define a tokenizer (Primitive) in terms of a set of token types. In this basic tokenizer each type is defined by a set of UNICODE symbols. Above basic level there are tokenizers (Complex) for which the token types are defined as regular expressions over the tokens of some other tokenizer, the so called Parent tokenizer.

Here is the Tokenizer Manager tool which shows all the tokenizers saved in the system, aa well as some of their characteristics:

  • Name - each tokenizer has a unique name
  • Type - shows whether the tokenizer is Primitive or Complex
  • Parent - for Complex tokenizers - the name of the Parent tokenizer
  • Compile - shows the Compilation status. Only the compiled tokenizers can be used in other tools.

The user can create a new tokenizer; edit, compile, or remove an existing one, load tokenizers from file out of the system or save tokenizer(s) as an external file.

  • New

    In order to create a new tokenizer (Complex or Primitive), the user must press the New button.

    Each row in the table represents one tokenizer category. The first column presents the category name. The content of the second column depends on the type of the tokenizer. The column contains a category value (all the symbols in the category) if the tokenizer is "primitive", or regular expression for a "non-primitive" tokenizer.

    • Primitive

      If the tokenizer is primitive, the user must select the Primitive check box.

      When defining the category value for a primitive tokenizer, the user should be aware of the following rules:

      1. The characters are quoted in single or double quotations, or referenced by a number. Example "." or ";" or 'k' or 32 (Space)
      2. If the user wants to write more than one symbol for a category, he/she should separate the symbols by a comma. Example: "a","b","c",...
      3. If the user wants to define a range of symbols, the starting and ending symbols must be connected with a dash. Example : "A"-"Z".
      4. Each character cannot have more than one category
      5. A category can be defined on more than one row. It is interpreted as a conjunction of expressions. For example:



        The tokenizer tool will interpret this lines as LAT "'a'-'z','A'-'Z'".
    • Complex

      When defining a complex tokenizer, the user should follow the rules below:

      1. A parent tokenizer must be selected.
      2. Each category Expression must be a valid regular expression.
      3. Two categories cannot have a common token.

      Here is an example of a complex tokenizer:

      The parent of the tokenizer is the "Mixed" primitive tokenizer shown above. This complex tokenizer uses the categories "LAT" and "CYR" from the parent tokenizer in order to define the new categories "LATw" and "CYRw".

      Each complex tokenizer must be compiled in order to be used.

  • Edit

    The user can edit a tokenizer pressing the "Edit" button. The tokenizer selected in the table will be opened for editing. The user can add, remove and reorder rows on the menu which shows up by right-clicking over a row in the table. The parent of a tokenizer is set when the tokenizer is created and can be changed by the user by pressing the Change Parent button.

    For each primitive tokenizer the user can define the sort order of the categories by clicking the Sort Order button. This sort order is used by the other tools in the system when they compare tokens. For example:

    The user can reorder the categories by selecting a row with left mouse click and pressing button - move the row or on the menu which shows up by right clicking over a row in the table.

    Also for each category of the primitive tokenizer a normalization of the symbols can be defined. This normalization is applied when the tokens are compared. The usual normalization is the conversion of the capital letters into small ones, but in the system the user can define any correspondence of the symbols. This can be done by right clicking on the category line in the table. For instance, the following dialog will appear if we select normalize for the "LAT" category of Mixed Tokenizer:

    For each symbol of the category the user can select a corresponding normalized symbol. The "New Category" combo box determines the new category that the symbols will receive after the normalization. This category can coincide with the original category of the symbols before the normalization or it can be completely different.

  • Remove - removes the selected tokenizer from the table;
  • Compile - compiles a non-primitive tokenizer;

    The user can compile a complex tokenizer with the "compile" button. Pressing this button causes not only compiling but also saving the tokenizer. Some possible error messages after compilation: "Ambiguous Categories" - two tokenizer categories recognize the same token from the input word. This error can occur even if the categories belong to different tokenizers in the hierarchy; "Category not defined" - the category name used in the value of one of the tokenizer categories is not defined in the tokenizers above this tokenizer in the hierarchy. When compiling a tokenizer all the tokenizers above and under it in the hierarchy are compiled. This means that a change in one tokenizer can produce an error during the compilation of another tokenizer. The user must be very careful with the error messages. Also it is recommended to keep all tokenizers compiled.

  • Load Tokenizer - loads a tokenizer from a file;
  • Save Tokenizer - saves a tokenizer as a file;
  • Exit - closes the dialog window.

Filters

This menu item starts the filter editor. In order to browse the filters in the system, the user can rely on the "Filter" combo box at the top of the dialog. The user can add token categories from different tokenizers or add XPath expressions to filter element nodes. A filter defines the way of removing tokens and/or elements from the content of a given element when some tool processes its content. Usually filters are used in connection with grammars. When the grammar is applied, it is applied to the content of an element. The content is processed before the grammar is applied. The processing includes tokenization of the text in the content and conversion of elements to list of tokens. The result is a list of tokens which is the input for the grammar. Very often some of the tokens in this list make no sense for the grammar. Such are space tokens, some special symbols, some special elements (an element for parenthetical expression, for instance). In order to escape these non-meaningful tokens, they can be filtered out from the grammar input in advance. This is the purpose of the filters in the system.

"Token Types" is a list of token categories, that will be filtered. The user can take categories from tokenizers in the system ("Choose From" list on the left side of the dialog) and add them to the list of filtered token categories with the arrow ("=>") button.

In order to filter an element the user has to specify an XPath expression. This XPath expression is evaluated on each element in the content which is filtered and if it is evaluated as true or returns a non-empty list of nodes, then the element is filtered out of the content. The addition of the XPath expression is done by pressing "add XPath" button. The new XPath expression will be added to the "Expression" list.

To remove a token category or an XPath from one of the lists, the user must select the line and press the "Remove" button under the table.

The user has to save the filter after editing by the Save button. The user can remove a filter by the Remove button.


Element Features

The Element Features is used for assigning information to the elements of a DTD.

The user can add the following information:

  1. A tokenizer for DTD. This tokenizer will be used to tokenize (when needed) the text content of all the element for which there is no a specific tokenizer set. This tokenizer can be overwritten. Thus it is used by default. For instance, if no tokenizer is defined for the element, the grammar and sort engines will look for the DTD default. After compiling DTD, the "Default" primitive tokenizer is set as default tokenizer.
  2. For each element in the DTD, a tokenizer can be set. This tokenizer overwrites the default DTD tokenizer. It is used for tokenization of the element text data by the Grammar and Sort engines and some other tools.
  3. The user can state that the content of the element is treated as a number.
  4. The user can define an XPath Value (Key) for each DTD element.
  5. The user can define the order over the DTD elements. This option is used for sorting purposes.

In order to select the default tokenizer for the DTD the user must select an item in the "Default Tokenizer" combo box. The user can select a tokenizer for each element of the DTD in the "Tokenizer" column of the table by clicking on a table cell. The check boxes in the "Number" column are used by the sort tool to determine whether the content of the current element can be treated as a number. For example, pages and price can be treated as numbers in the comparison of two books. The values in the "Element Value" column are used by the Grammar Engine to define the value of the element nodes. (For defining value check the Edit Grammar)

The addition (creation) of a new Element feature can be performed by pressing the Add button. The removal of elements can be done by selecting the corresponding row and pressing the Remove button. The OK button closes the dialog and updates the changes (if any). The Cancel button closes the dialog without saving the changes.

The order of the elements in DTD can be defined in the sort table shown when the user presses the "Sort Order" button. When comparing two elements, the position in the sort table defines their order. The user can change the position of elements by dragging their rows to correct positions or using the context menu opened when the user right clicks on sort table row. Here is an example:

Attribute Features

The Attribute Features is used to add information to the attributes of elements from a DTD or user elements.

The addition (creation) of a new Attribute feature can be performed by pressing the Add button. The removal of attributes can be done by selecting the corresponding row and pressing the Remove button. The OK button closes the dialog and updates the changes (if any). The Cancel button closes the dialog without saving the changes

The "Element" column of the table contains the elements which have attributes. One and the same element can appear several times because it can have more than one attribute. The "Attribute" column has the name of the attribute for which the attribute features are defined. "Is Enumeration " column shows whether the attribute is Enumeration or not. In the "Tokenizer" column the user can select different tokenizers for each attribute. The sort tool uses information from the "Number" column to select the way of comparing the value of the attributes (As plain text or as a number).

An additional feature is the order of enumerated attribute values. The attributes with enumerated values have "Yes" string at the "Is Enumeration" column. In order to sort the enumerated values the user must click on an attribute with enumeration value and to click the "Sort Values" button. Example:

The values are sorted in ascending order. The user can change the order by dragging the rows or by pressing the right mouse button.


XPath Macros

The XPath Macros gives the user possibility to name XPath Expressions. It contains a macro name and an XPath expression. It is designed for the general usage. The XPath Macros can be parts of XPath expressions and thus they can be used in each place where XPath is used. Within an XPath expression a macro is denoted by '%' (percent sign), followed by the macro name, i.e. %macro_name. A macro citation itself also represents an XPath expression (CLaRK System extension).
In the picture below you is shown what the XPath Macros editor looks like:


Keys

The Keys are a means for naming of XPath expressions and some specific information important for some of the tools in the system. The key names can be unique arbitrary strings. Here is what the XPath Key manager window looks like:

Each row in the table above represents one XPath Key. The content of the table cannot be modified directly form here. In order to modify a key entry, the user must select the corresponding row in the table and then press the Edit button. A new dialog window appears where the key specifications can be changed. The addition (creation) of a new XPath Key can be done by pressing the Add button. The removal of keys can be done by selecting the corresponding row and pressing the Remove button. The Exit button closes the dialog and updates the changes (if any).

There are several types of keys. Each type has different additional options and usage:

  • Grammar Key - This key is designed for usage with the Grammar tool, especially in the Element Value option. It has two more additional fields: normalization on/off and a specification of a tokenizer. A description of the usage of this key can be found in the description of menu option Apply Grammar. Here is what the Grammar XPath Key editor window looks like:


  • Sort Key - This key is designed for usage with the Sort tool. It contains settings specific for its usage:order descending/ascending; reverse sorting; removing leading/ending white spacing (trim); enabling number interpretation, enabling normalization; tokenizer specification. All these options and their usage are described in menu option Sort.


  • Table Sort Key - This key is designed for usage in the Concordance tool. It is used for specifying the sort options for the different table columns' content. This key has one more option than the Sort Key options: Prefix. This option specifies for which table column this key is defined to be used. Here is what the editor window looks like:



Shortcuts

This tool allows user to customize keyboard shortcuts. There is a possibility to add a new shortcut to a command or an existing query; to edit or remove an existing one (by selection from the table with all shortcuts):

When the user creates or edits Action a dialog appears:

  • Action Items

    The user can assign shortcuts to any action event - for tree, menu or for attribute area ("Action Type" combo box) in the system. Action must be selected from the table with actions for the current type.

  • Queries

    The user can assign shortcuts for queries saved in the system.

  • By Modifiers and Key Code the user sets the appropriate key combination for the selected action or query.