The goal of this demo is to show how the grammar tool of the CLaRK system can be used to find and mark-up sequence of data from XML documents.
The regular grammars in CLaRK System work over token and/or element values generated from the content of an XML document and they incorporate their results back in the document as XML mark-up.
The tokens are determined by the corresponding tokenizer.
Before used in the grammar, each XML element is converted into a list of textual items. This list is call element value for the XML element. The element values are defined with the help of XPath keys, which determine the important information for each element.
In the grammars, the token and element values are described by token and element descriptions. These descriptions could contain wildcard symbols and variables. The variables are shared among the token descriptions within a regular expression and can be used for the treatment of phenomena like agreement.
Each grammar is a set of rules. Each rule consists of three regular expressions and mark-up. The three regular expressions corresponds to the item for which we are searching in the document (Regular Expression column), the left context of found string (Left Context Regular Expression), the right context of found string (Right Context Regular Expression). The left and the right regular expressions could be empty if no constraints over the context is imposed. The return mark-up is a well-formed XML fragment which is substituted for the found string. In fact the return mark-up encodes the category of the rule.
Here is the list of the token and element descriptions:
"token" -> describes the token itself. This description can be matched to the token itself and
nothing else.$TokenCategory -> describes all tokens of the category TokenCategory. In the
grammar input this description is matched against exactly one token of this category.#, @, % -> describe substrings of a given
token. # - describes a substring of arbitrary length from 0 to infinity, @ - describes
a substring of arbitrary length from 0 or 1, % - describes a substring of arbitrary length one. Here
are some examples:"lov#": matches exactly one token which starts with "lov": "lov", "love", "loves", and many
others."lo#ve": matches exactly one token which starts with "lo" and ends with "ve": "love",
"locative", "locomotive" etc."%og":matches exactly one token which ends with "og": "bog", "dog", "fog", "jog" etc."do":matches exactly one token which starts with "do": "do", "doe", "dog", "don" and many others.
&V -> describes some substring in a token when initialised for first time, then
matches to the same substring in the same token, or some next tokens. The scope of a variable is one grammar
rule. The variable can be used in the return mark-up and in this case the value of the variable is copied in the return
mark-up. Each variable consists of the symbol & followed by a single Latin letter. Each variable
has positive and/or negative constraints over the possible values. They are given by lists of token
descriptions. Here are some examples:<w>: matches exactly one w element.<$TokenCategory> : matches exactly one element, which element value is a token
description with category TokenCategory<"token"> : matches exactly one element, which element value is token itself and
nothing else. Token can contain wildcard symbols and variables.<<N>> : matches exactly one element, which element value is element
N. The goal here is to mark all meaningful tokens - words, punctuation, numbers etc. in the text using tokenizer categories.
The document used for this demo is:
Standart20030524.tag.
The tokenizer is MixedWord tokenizer which is defined within the System. If you like to see
the token types defined within this tokenizer you have to select Tokenizers item from menu
Definitions.
In order to run the demo you have to perform the following steps:
Standart20030524.tag is loaded and saved in the system. If it is not, then load it
as it is described in Import XML Tool DemoTools.tokenize - the grammar editor dialog is opened.The goal of the grammar is to mark all tokens from the text - words, numbers, punctuation and white spaces.
Punctuation will be marked with <pt> tag, white spaces with <whitespace>
tag and all other tokens will be marked with <tok> tag. The MixedWord Tokenizer separates
text in such tokens, but it divides some tokens that must be a whole.
For each token we have a different rule in the grammar:
LATw - the Regular Expression(RE) of grammar rule will be $LATw.MP's will be
interpreted as MP ,' and s. So the rule must unite this three tokens
- the RE will be $LATw,"'",$LATwThe two rules for words and possessive forms differ in "'",$LATw and they can be united:
$LATw,("'",$LATw)?.
$NUMBER+ ($NUMBER i a Token Category that recognizes digits)$NUMBER+,$LATw
Exit button.Grammar Manager pressing Apply button or from Apply Grammar menu item.Grammar select the tokenize grammar.Apply to write the following XPath expression :
//text/descendant-or-self::p | //head | //trailer which select all the parents of a textual element within
the document.Tokenizer choose
MixedWord - which token Categories are used in the grammar.Normalize check box is not selected because the tokenizer makes no difference between
capital and small letters.Multiple Apply.
Then a new part of the dialog appears in which you have to select the documents from the corresponding
document group via the button Add Documents.Options and then Overwrite
radio button in order to have the result from the grammar in the same document. Then return to the main
dialog.
The above query is saved in document tokenize.gram.que in the demo directory.
The goal here is to create a grammar that will be used in concordance to view verbal tenses in the text.
The document used for this demo is: Standart20030524.tag.
The tokenizer is MixedWord tokenizer which is defined within the System. If you like to see
the token types defined within this tokenizer you have to select Tokenizers item from menu
Definitions.
In order to run the demo you have to perform the following steps:
Standart20030524.tag is loaded and saved in the system. If it is
not, then load it as it is described in Import XML Tool DemoTools.tense - the grammar editor dialog is opened.This grammar will be used from concordance tool that is why found items will not be marked.
For each tense we have a different rule in the grammar:
("was"|"were"),"not"?,"#". ("has"|"have"|"had"),"#"("am"|"is"|"are"),"#" ("has"|"have"|"had"),"been","#"."will","not"?,"#"
The goal here is not to be precise. This grammar can be used for observation in the text and can be a base more explicit grammars.
Exit button.Grammar Manager pressing
Apply button or from Apply Grammar menu item.Grammar select the tense grammar.Apply to write the following XPath expression : //text//p | //head
which select all the parents of a textual element that may have usage of tenses within the document.Tokenizer choose MixedWord - that separates text into
words.NO_SPACE filter is
used. It is described in Filter Tool Demo.Normalize check box can be selected because in the text there can be words with capitalized
letters.Multiple Apply.
Then a new part of the dialog appears in which you have to select the documents from the corresponding
document group via the button Add Documents.Options and then Overwrite
radio button in order to have the result from the grammar in the same document. Then return to the main dialog.
The above query is saved in document tense.gram.que in the demo directory.
The goal here is add morphological information to tokens - all possible analyses for a current word.
Two grammars are demonstrated. The difference between them is only in the Return mark-up. Grammars work over the result file from Demo1.
The document used for this demo is: Standart20030524.tag.
No tokenizer is needed because grammar works over the whole string in tok elements.
In order to run the demo you have to perform the following steps:
Standart20030524.tag - the result from Demo1 is loaded and
saved in the system. If it is not, then load it as it is described in Import XML
Tool DemoTools.dict-attr.grm and dict-element.grm in the Grammar
Manager:File I/O button at the bottom of the dialog and click over Load grammar from
file - a standard File chooser appears.dict-attr.grm and press Load from file
button.User can see grammars when selects each and presses Edit button.
What grammars do is to mark each token in appropriate way and add information with all possible morphological analyses for that token.
In dict-element.grm recognized words are marked with <w> tag.For each
word we encode the wordform in <ph> tag, the appropriate morpho-syntactic information
from the dictionary is encoded as two elements: <aa> element, which contains a list of
morpho-syntactic tags for the wordform separated by a semicolon, and <ta> element, which
contains the actual morpho-syntactic tag for this use of the wordform.
The grammar in this case is :

In dict-attr.grm recognized words are marked with <w> tag.For each
wordform the appropriate morpho-syntactic information from the dictionary is encoded as two attributes:
aa attribute, which contains a list of morpho-syntactic tags for the wordform separated by a
semicolon, and ana element, which contains the actual morpho-syntactic tag for this use of the
wordform.
The grammar in this case is :

dict-attr.grm grammar will be described. The query for dict-element.grm is the same, only the grammar must be changed :Grammar Manager pressing
Apply button or from Apply Grammar menu item.Grammar select the dict-attr.grm grammar.Apply to write the following XPath expression : //tok which
select all tok element within the document.Tokenizer choose No Tokenizer.Simple radio button as a Settings Mode.Normalize check box must be selected because in order not to have difference between capital
and small letters.Multiple Apply. Then
a new part of the dialog appears in which you have to select the documents from the corresponding document
group via the button Add Documents.Options and then Overwrite
radio button in order to have the result from the grammar in the same document. Then return to the main dialog.
w tag and morphological information added.The above query is saved in document Eng_Dict_Attribute in the demo directory.
The query of dict-element.grm is saved in document Eng_Dict_Element in
the demo directory.
The goal here is to delimitate sentence boundaries.
The document used for this demo is:Standart20030525gram.tag.
The tokenizer is Uptok tokenizer which separates words with capital first letter from words
with small letters. This is important for named entity recognition and sentence boundary. The tokenizer can be
loaded within the System from Tokenizers item from menu Definitions.
In order to run the demo you have to perform the following steps:
Uptok is loaded and saved in the system. If it is not, then load it
as it is described in Tokenizer Tool DemoStandart20030525gram.tag is loaded and saved in the system. If it is
not, then load it as it is described in Import XML Tool DemoTools.File I/O, Load grammar from file-
choose sentence.gram. If you like to see the grammar rules within this grammar you have to select Edit button Grammar Manager Dialog.Grammar determines a sentence as a sequence of tokens – words, numbers and punctuation (we refer to the
content of <w>, <tok> and <pt> elements with Element Values). As sentence first element we
have a capitalized word ($LATwc)or tok element, which is a number
($NUMBER+). The sentence ends with punctuation – full stop, question mark or exclamation
mark. The sentence can be followed by another sentence or it can be the last one in a paragraph
($$) – these features are matched by the right context of grammar. Sometimes the sentence can be
a direct speech – in this case it starts with a dash.

Grammar Manager pressing Apply button or from Apply Grammar menu item.Grammar select the sentence grammar.Apply to write the following XPath expression : //text/descendant::p
which select all the paragraphs within the document.Tokenizer choose Uptok - which token Categories are used in
the grammar.NO_SPACE filter is used. It is described in Filter Tool Demo.Normalize check box is not selected because it is important to distinguish capital from small
letters.w, pt and tok
elements, but we are interested of their context, that is why Element Values are used :
w - the text of ph child element is taken : ph/text()tok - the text : text()pt element text child is taken : text().
The above query is saved in document sentence.gram.que in the current
directory.