Grammar Tool Demo

The goal of this demo is to show how the grammar tool of the CLaRK system can be used to find and mark-up sequence of data from XML documents.

The regular grammars in CLaRK System work over token and/or element values generated from the content of an XML document and they incorporate their results back in the document as XML mark-up.

The tokens are determined by the corresponding tokenizer.

Before used in the grammar, each XML element is converted into a list of textual items. This list is call element value for the XML element. The element values are defined with the help of XPath keys, which determine the important information for each element.

In the grammars, the token and element values are described by token and element descriptions. These descriptions could contain wildcard symbols and variables. The variables are shared among the token descriptions within a regular expression and can be used for the treatment of phenomena like agreement.

Each grammar is a set of rules. Each rule consists of three regular expressions and mark-up. The three regular expressions corresponds to the item for which we are searching in the document (Regular Expression column), the left context of found string (Left Context Regular Expression), the right context of found string (Right Context Regular Expression). The left and the right regular expressions could be empty if no constraints over the context is imposed. The return mark-up is a well-formed XML fragment which is substituted for the found string. In fact the return mark-up encodes the category of the rule.

Here is the list of the token and element descriptions:

  1. "token" -> describes the token itself. This description can be matched to the token itself and nothing else.
  2. $TokenCategory -> describes all tokens of the category TokenCategory. In the grammar input this description is matched against exactly one token of this category.
  3. Wildcard Symbols: #, @, % -> describe substrings of a given token. # - describes a substring of arbitrary length from 0 to infinity, @ - describes a substring of arbitrary length from 0 or 1, % - describes a substring of arbitrary length one. Here are some examples:
  4. Variables: &V -> describes some substring in a token when initialised for first time, then matches to the same substring in the same token, or some next tokens. The scope of a variable is one grammar rule. The variable can be used in the return mark-up and in this case the value of the variable is copied in the return mark-up. Each variable consists of the symbol & followed by a single Latin letter. Each variable has positive and/or negative constraints over the possible values. They are given by lists of token descriptions. Here are some examples:
  5. Complex token descriptions. The user can combine the above descriptions in one token description. Some examples: "lov%#", "Vp&N&G#&P"
  6. Element description is an regular expression in angle brackets: "<" RegularExpression ">". Here the RegularExpression is over token descriptions which is matched against the element value. Examples:

Demo 1: Grammar that marks tokens

The goal here is to mark all meaningful tokens - words, punctuation, numbers etc. in the text using tokenizer categories.

The document used for this demo is: Standart20030524.tag.

The tokenizer is MixedWord tokenizer which is defined within the System. If you like to see the token types defined within this tokenizer you have to select Tokenizers item from menu Definitions.

In order to run the demo you have to perform the following steps:

  1. Check whether the document: Standart20030524.tag is loaded and saved in the system. If it is not, then load it as it is described in Import XML Tool Demo
  2. Open the document in the editor.
  3. Open the dialog of the Grammar Manager tool from the menu item Tools.
  4. Create new grammar named tokenize - the grammar editor dialog is opened.
  5. The goal of the grammar is to mark all tokens from the text - words, numbers, punctuation and white spaces. Punctuation will be marked with <pt> tag, white spaces with <whitespace> tag and all other tokens will be marked with <tok> tag. The MixedWord Tokenizer separates text in such tokens, but it divides some tokens that must be a whole.

    For each token we have a different rule in the grammar:

  6. The grammar in this case has to be:
  7. In order to use the grammar save and compile it, and then return to the Grammar Manager dialog from Exit button.
  8. Applying the grammar over the document:

The above query is saved in document tokenize.gram.que in the demo directory.

Demo 2: Grammar for tense recognition

The goal here is to create a grammar that will be used in concordance to view verbal tenses in the text.

The document used for this demo is: Standart20030524.tag.

The tokenizer is MixedWord tokenizer which is defined within the System. If you like to see the token types defined within this tokenizer you have to select Tokenizers item from menu Definitions.

In order to run the demo you have to perform the following steps:

  1. Check whether the document: Standart20030524.tag is loaded and saved in the system. If it is not, then load it as it is described in Import XML Tool Demo
  2. Open the document in the editor.
  3. Open the dialog of the Grammar Manager tool from the menu item Tools.
  4. Create new grammar named tense - the grammar editor dialog is opened.
  5. This grammar will be used from concordance tool that is why found items will not be marked.

    For each tense we have a different rule in the grammar:

  6. The grammar in this case has to be:
  7. The goal here is not to be precise. This grammar can be used for observation in the text and can be a base more explicit grammars.

  8. In order to use the grammar save and compile it, and then return to the Grammar Manager dialog from Exit button.
  9. Applying the grammar over the document:

The above query is saved in document tense.gram.que in the demo directory.

Demo 3: Grammars that add morphological information to tokens

The goal here is add morphological information to tokens - all possible analyses for a current word.

Two grammars are demonstrated. The difference between them is only in the Return mark-up. Grammars work over the result file from Demo1.

The document used for this demo is: Standart20030524.tag.

No tokenizer is needed because grammar works over the whole string in tok elements.

In order to run the demo you have to perform the following steps:

  1. Check whether the document: Standart20030524.tag - the result from Demo1 is loaded and saved in the system. If it is not, then load it as it is described in Import XML Tool Demo
  2. Open the document in the editor.
  3. Open the dialog of the Grammar Manager tool from the menu item Tools.
  4. Load the two grammars dict-attr.grm and dict-element.grm in the Grammar Manager:
  5. User can see grammars when selects each and presses Edit button.

    What grammars do is to mark each token in appropriate way and add information with all possible morphological analyses for that token.

    In dict-element.grm recognized words are marked with <w> tag.For each word we encode the wordform in <ph> tag, the appropriate morpho-syntactic information from the dictionary is encoded as two elements: <aa> element, which contains a list of morpho-syntactic tags for the wordform separated by a semicolon, and <ta> element, which contains the actual morpho-syntactic tag for this use of the wordform.

    The grammar in this case is :

    In dict-attr.grm recognized words are marked with <w> tag.For each wordform the appropriate morpho-syntactic information from the dictionary is encoded as two attributes: aa attribute, which contains a list of morpho-syntactic tags for the wordform separated by a semicolon, and ana element, which contains the actual morpho-syntactic tag for this use of the wordform.

    The grammar in this case is :

  6. Applying the grammar over the document - query for dict-attr.grm grammar will be described. The query for dict-element.grm is the same, only the grammar must be changed :

The above query is saved in document Eng_Dict_Attribute in the demo directory.

The query of dict-element.grm is saved in document Eng_Dict_Element in the demo directory.

Demo 4 : Grammar recognizing sentence delimitation

The goal here is to delimitate sentence boundaries.

The document used for this demo is:Standart20030525gram.tag.

The tokenizer is Uptok tokenizer which separates words with capital first letter from words with small letters. This is important for named entity recognition and sentence boundary. The tokenizer can be loaded within the System from Tokenizers item from menu Definitions.

In order to run the demo you have to perform the following steps:

  1. Check whether the tokenizer Uptok is loaded and saved in the system. If it is not, then load it as it is described in Tokenizer Tool Demo
  2. Check whether the document:Standart20030525gram.tag is loaded and saved in the system. If it is not, then load it as it is described in Import XML Tool Demo
  3. Open the document in the editor.
  4. Open the dialog of the Grammar Manager tool from the menu item Tools.
  5. Load the grammar in the system from File I/O, Load grammar from file- choose sentence.gram. If you like to see the grammar rules within this grammar you have to select Edit button Grammar Manager Dialog.
  6. Grammar determines a sentence as a sequence of tokens – words, numbers and punctuation (we refer to the content of <w>, <tok> and <pt> elements with Element Values). As sentence first element we have a capitalized word ($LATwc)or tok element, which is a number ($NUMBER+). The sentence ends with punctuation – full stop, question mark or exclamation mark. The sentence can be followed by another sentence or it can be the last one in a paragraph ($$) – these features are matched by the right context of grammar. Sometimes the sentence can be a direct speech – in this case it starts with a dash.

  7. The grammar in this case has to be:
  8. Applying the grammar over the document:

The above query is saved in document sentence.gram.que in the current directory.