History
-September 2008: partly outdated - to be updated for API 1.5. +September 2008: updated for Version 1.5.
-October 2007: updated for API 1.2. +October 2007: updated for Version 1.2.
January 2006: first version. @@ -32,20 +78,31 @@ will give some hints how to extend the API. A manual for using the resource grammar is found in
-http://www.cs.chalmers.se/~aarne/GF/lib/resource-1.0/doc/synopsis.html.
+www.cs.chalmers.se/Cs/Research/Language-technology/GF/lib/resource/doc/synopsis.html.
A tutorial on GF, also introducing the idea of resource grammars, is found in
-http://www.cs.chalmers.se/~aarne/GF/doc/tutorial/gf-tutorial2.html.
+www.cs.chalmers.se/Cs/Research/Language-technology/GF/doc/gf-tutorial.html.
-This document concerns the API v. 1.0. You can find the current code in +This document concerns the API v. 1.5, while the current stable release is 1.4. +You can find the code for the stable release in
-http://www.cs.chalmers.se/~aarne/GF/lib/resource-1.0/
+www.cs.chalmers.se/Cs/Research/Language-technology/GF/lib/resource/
+and the next release in +
+
+www.cs.chalmers.se/Cs/Research/Language-technology/GF/lib/next-resource/
+
+It is recommended to build new grammars to match the next release. +
+The library is divided into a bunch of modules, whose dependencies @@ -54,8 +111,11 @@ are given in the following figure.
+Modules of different kinds are distinguished as follows: +
-The solid ellipses show the API as visible to the user of the library. The
-dashed ellipses form the main of the implementation, on which the resource
-grammar programmer has to work with. With the exception of the Paradigms
-module, the visible API modules can be produced mechanically.
+Put in another way:
-
-
-Thus the API consists of a grammar and a lexicon, which is
-provided for test purposes.
+The dashed ellipses form the main parts of the implementation, on which the resource
+grammar programmer has to work with. She also has to work on the Paradigms
+module. The rest of the modules can be produced mechanically from corresponding
+modules for other languages, by just changing the language codes appearing in
+their module headers.
The module structure is rather flat: most modules are direct
parents of Grammar. The idea
-is that you can concentrate on one linguistic aspect at a time, or
+is that the implementors can concentrate on one linguistic aspect at a time, or
also distribute the work among several authors. The module Cat
defines the "glue" that ties the aspects together - a type system
to which all the other modules conform, so that e.g. NP means
the same thing in those modules that use NPs and those that
constructs them.
+For the user of the library, these modules are the most important ones.
+In a typical application, it is enough to open Paradigms and Syntax.
+The module Try combines these two, making it possible to experiment
+with combinations of syntactic and lexical constructors by using the
+cc command in the GF shell. Here are short explanations of each API module:
+
Try: the whole resource library for a language (Paradigms, Syntax,
+ Irreg, and Extra);
+ produced mechanically as a collection of modules
+Syntax: language-independent categories, syntax functions, and structural words;
+ produced mechanically as a collection of modules
+Constructors: language-independent syntax functions and structural words;
+ produced mechanically via functor instantiation
+Paradigms: language-dependent morphological paradigms
+
-The direct parents of the top will be called phrase category modules,
+The immediate parents of Grammar will be called phrase category modules,
since each of them concentrates on a particular phrase category (nouns, verbs,
adjectives, sentences,...). A phrase category module tells
how to construct phrases in that category. You will find out that
@@ -106,9 +190,10 @@ one of a small number of different types). Thus we have
Conjunction: coordination of phrases
Phrase: construction of the major units of text and speech
Text: construction of texts as sequences of phrases
-Idiom: idiomatic phrases such as existentials
+Idiom: idiomatic expressions such as existentials
+
Expressions of each phrase category are constructed in the corresponding
@@ -137,6 +222,7 @@ can skip the lincat definition of a category and use the default
{s : Str} until you need to change it to something else. In
English, for instance, many categories do have this linearization type.
What is lexical and what is syntactic is not as clearcut in GF as in @@ -162,41 +248,42 @@ samples than complete lists. There are two such modules:
The module Structural aims for completeness, and is likely to
be extended in future releases of the resource. The module Lexicon
-gives a "random" list of words, which enable interesting testing of syntax,
-and also a check list for morphology, since those words are likely to include
+gives a "random" list of words, which enables testing the syntax.
+It also provides a check list for morphology, since those words are likely to include
most morphological patterns of the language.
In the case of Lexicon it may come out clearer than anywhere else
in the API that it is impossible to give exact translation equivalents in
-different languages on the level of a resource grammar. In other words,
-application grammars are likely to use the resource in different ways for
+different languages on the level of a resource grammar. This is no problem,
+since application grammars can use the resource in different ways for
different languages.
In addition to the common API, there is room for language-dependent extensions -of the resource. The top level of each languages looks as follows (with English as example): +of the resource. The top level of each languages looks as follows (with German +as example):
- abstract English = Grammar, ExtraEngAbs, DictEngAbs + abstract AllGerAbs = Lang, ExtraGerAbs, IrregGerAbs
-where ExtraEngAbs is a collection of syntactic structures specific to English,
-and DictEngAbs is an English dictionary
-(at the moment, it consists of IrregEngAbs,
-the irregular verbs of English). Each of these language-specific grammars has
+where ExtraGerAbs is a collection of syntactic structures specific to German,
+and IrregGerAbs is a dictionary of irregular words of German
+(at the moment, just verbs). Each of these language-specific grammars has
the potential to grow into a full-scale grammar of the language. These grammar
can also be used as libraries, but the possibility of using functors is lost.
To give a better overview of language-specific structures,
-modules like ExtraEngAbs
+modules like ExtraGerAbs
are built from a language-independent module ExtraAbs
by restricted inheritance:
- abstract ExtraEngAbs = Extra [f,g,...] + abstract ExtraGerAbs = Extra [f,g,...]
Thus any category and function in Extra may be shared by a subset of all
@@ -210,42 +297,15 @@ In a minimal resource grammar implementation, the language-dependent
extensions are just empty modules, but it is good to provide them for
the sake of uniformity.
-Among all categories and functions, a handful are -most important and distinct ones, of which the others are can be -seen as variations. The categories are -
-- Cl ; VP ; V2 ; NP ; CN ; Det ; AP ; --
-The functions are -
-- PredVP : NP -> VP -> Cl ; -- predication - ComplV2 : V2 -> NP -> VP ; -- complementization - DetCN : Det -> CN -> NP ; -- determination - ModCN : AP -> CN -> CN ; -- modification --
-This toy Latin grammar shows in a nutshell how these -rules relate the categories to each other. It is intended to be a -first approximation when designing the parameter system of a new -language. -
--If you want to experiment with a small subset of the resource API first, -try out the module -Syntax -explained in the -GF Tutorial. -
+Some lines in the resource library are suffixed with the comment -```--# notpresent +
++ --# notpresent ++
which is used by a preprocessor to exclude those lines from a reduced version of the full resource. This present-tense-only version is useful for applications in most technical text, since @@ -254,10 +314,14 @@ be useful to exclude those lines in a first version of resource implementation. To compile a grammar with present-tense-only, use
- i -preproc=GF/lib/resource-1.0/mkPresent LangGer.gf + make Present- +
+with resource/Makefile.
+
Unless you are writing an instance of a parametrized implementation @@ -265,7 +329,8 @@ Unless you are writing an instance of a parametrized implementation simplest way is to follow roughly the following procedure. Assume you are building a grammar for the German language. Here are the first steps, which we actually followed ourselves when building the German implementation -of resource v. 1.0. +of resource v. 1.0 at Ubuntu linux. We have slightly modified them to +match resource v. 1.5 and GF v. 3.0.
GF/lib/resource/english, named
@@ -279,6 +344,8 @@ of resource v. 1.0.
Ger and Deu are given, and we pick Ger.
+ (We use the 3-letter codes rather than the more common 2-letter codes,
+ since they will suffice for many more languages!)
*Eng.gf files from english german,
and rename them:
@@ -286,7 +353,10 @@ of resource v. 1.0.
cp ../english/*Eng.gf .
rename 's/Eng/Ger/' *Eng.gf
-
+ If you don't have the rename command, you can use a bash script with mv.
+Eng module references to Ger references
in all files:
@@ -294,7 +364,8 @@ of resource v. 1.0.
sed -i 's/Eng/Ger/g' *Ger.gf
The first line prevents changing the word English, which appears
- here and there in comments, to Gerlish.
+ here and there in comments, to Gerlish. The sed command syntax
+ may vary depending on your operating system.
Eng - verify this by
@@ -327,10 +398,10 @@ of resource v. 1.0.
You will get lots of warnings on missing rules, but the grammar will compile.
-- pg -printer=missing + pg -missingtells you what exactly is missing.
Here is the module structure of LangGer. It has been simplified by leaving out
the majority of the phrase category modules. Each of them has the same dependencies
-as e.g. VerbGer.
+as VerbGer, whose complete dependencies are shown as an example.
-The real work starts now. There are many ways to proceed, the main ones being +The real work starts now. There are many ways to proceed, the most obvious ones being
Phrase and go down to Sentence, then
@@ -373,31 +445,34 @@ test data and enough general view at any point:
lincat N = {s : Number => Case => Str ; g : Gender} ;
we need the parameter types Number, Case, and Gender. The definition
-of Number in common/ParamX works for German, so we
+of Number in common/ParamX
+works for German, so we
use it and just define Case and Gender in ResGer.
-regN in ParadigmsGer. In this way you can
+mkN in ParadigmsGer. In this way you can
already implement a huge amount of nouns correctly in LexiconGer. Actually
-just adding mkN should suffice for every noun - but,
+just adding the worst-case instance of mkN (the one taking the most
+arguments) should suffice for every noun - but,
since it is tedious to use, you
might proceed to the next step before returning to morphology and defining the
-real work horse reg2N.
+real work horse, mkN taking two forms and a gender.
resource directory, by the commands
- i -retain ParadigmsGer - cc regN "Kirche" + > i -retain german/ParadigmsGer + > cc -table mkN "Kirche"
NounGer (DetCN UsePron DetSg SgQuant NoNum NoOrd DefArt IndefArt UseN)and
-StructuralGer (i_Pron every_Det). You also need some categories and
+NounGer (DetCN UsePron DetQuant NumSg DefArt IndefArt UseN) and
+StructuralGer (i_Pron this_Quant). You also need some categories and
parameter types. At this point, it is maybe not possible to find out the final
-linearization types of CN, NP, and Det, but at least you should
+linearization types of CN, NP, Det, and Quant, but at least you should
be able to correctly inflect noun phrases such as every airplane:
- i LangGer.gf
- l -table DetCN every_Det (UseN airplane_N)
+ > i german/LangGer.gf
+ > l -table DetCN every_Det (UseN airplane_N)
Nom: jeder Flugzeug
Acc: jeden Flugzeug
@@ -406,16 +481,16 @@ be able to correctly inflect noun phrases such as every airplane:
CatGer.V, ResGer.VForm, and
-ParadigmsGer.regV. You may choose to exclude notpresent
+ParadigmsGer.mkV. You may choose to exclude notpresent
cases at this point. But anyway, you will be able to inflect a good
number of verbs in Lexicon, such as
-live_V (regV "leven").
+live_V (mkV "leben").
VP and
Cl in CatGer, VerbGer.UseV, and SentenceGer.PredVP.
Even if you have excluded the tenses, you will be able to produce
- i -preproc=mkPresent LangGer.gf
+ > i -preproc=./mkPresent german/LangGer.gf
> l -table PredVP (UsePron i_Pron) (UseV live_V)
Pres Simul Pos Main: ich lebe
@@ -425,22 +500,30 @@ Even if you have excluded the tenses, you will be able to produce
Pres Simul Neg Inv: lebe ich nicht
Pres Simul Neg Sub: ich nicht lebe
+You should also be able to parse:
++ > p -cat=Cl "ich lebe" + PredVP (UsePron i_Pron) (UseV live_V) +-
CatGer.V2 ParadigmsGer.dirV2 VerbGer.ComplV2)
+CatGer.V2 CatGer.VPSlash ParadigmsGer.mkV2 VerbGer.ComplSlash VerbGer.SlashV2a)
are a natural next step, so that you can
-produce ich liebe dich.
+produce ich liebe dich ("I love you").
-CatGer.A ParadigmsGer.regA NounGer.AdjCN AdjectiveGer.PositA)
+CatGer.A ParadigmsGer.mkA NounGer.AdjCN AdjectiveGer.PositA)
will force you to think about strong and weak declensions, so that you can
-correctly inflect my new car, this new car.
+correctly inflect mein neuer Wagen, dieser neue Wagen
+("my new car, this new car").
The following develop-test cycle will @@ -449,14 +532,13 @@ and in later steps where you are more on your own.
NounGer, and uncomment some
- linearization rules (for instance, DefSg, which is
- not too complicated).
+ linearization rules (for instance, DetCN, as above).
CN, NP, N) and the
+CN, NP, N, Det) and the
variations they have. Encode this in the lincats of CatGer.
You may have to define some new parameter types in ResGer.
@@ -467,39 +549,39 @@ and in later steps where you are more on your own.
treebank option
+ preserves the tree
- gr -cat=NP -number=20 -tr | l -table + > gr -cat=NP -number=20 | l -table -treebank-
tree_bank command,
+diff command to compare later
+ linearizations produced from the same list of trees. If you save the trees
+ in a file trees, you can do as follows:
- gr -cat=NP -number=20 | tb -xml | wf NP.tb + > rf -file=trees -tree -lines | l -table -treebank | wf -file=treebank- You can later compared your modified grammar to this treebank by + +
resource/exx-resource.gft. A treebank can be created from this by
+ the Unix command
- rf NP.tb | tb -c + % runghc Make.hs test langs=Ger
You are likely to run this cycle a few times for each linearization rule
-you implement, and some hundreds of times altogether. There are 66 cats and
-458 funs in Lang at the moment; 149 of the funs are outside the two
+you implement, and some hundreds of times altogether. There are roughly
+70 cats and
+600 funs in Lang at the moment; 170 of the funs are outside the two
lexicon modules).
-Here is a live log of the actual process of -building the German implementation of resource API v. 1.0. -It is the basis of the more detailed explanations, which will -follow soon. (You will found out that these explanations involve -a rational reconstruction of the live process! Among other things, the -API was changed during the actual process to make it more intuitive.) -
-
-These modules will be written by you.
+These auxuliary resource modules will be written by you.
ResGer: parameter types and auxiliary operations
@@ -521,38 +603,53 @@ package.
Coordination: operations to deal with lists and coordination
Prelude: general-purpose operations on strings, records,
truth values, etc.
-Predefined: general-purpose operations with hard-coded definitions
+Predef: general-purpose operations with hard-coded definitions
An important decision is what rules to implement in terms of operations in
-ResGer. A golden rule of functional programming says that, whenever
-you find yourself programming by copy and paste, you should write a function
-instead. This indicates that an operation should be created if it is to be
-used at least twice. At the same time, a sound principle of vicinity says that
-it should not require too much browsing to understand what a rule does.
+ResGer. The golden rule of functional programming says:
+
+This rule suggests that an operation should be created if it is to be +used at least twice. At the same time, a sound principle of vicinity says: +
+From these two principles, we have derived the following practice:
ResGer. An example is mkClause,
-used in Sentence, Question, and Relative-
+ it should be created in as an oper in ResGer. An example is mkClause,
+ used in Sentence, Question, and Relative-
Numerals.
-Numerals.
+let definition.
+oper,
+ but rather inlined. However, a let definition may well be in place just
+ to make the readable.
+ Most functions in phrase category modules
+ are implemented in this way.
-This discipline is very different from the one followed in earlier
+This discipline is very different from the one followed in early
versions of the library (up to 0.9). We then valued the principle of
abstraction more than vicinity, creating layers of abstraction for
almost everything. This led in practice to the duplication of almost
all code on the lin and oper levels, and made the code
hard to understand and maintain.
The paradigms needed to implement @@ -565,35 +662,42 @@ variants.
For ease of use, the Paradigms modules follow a certain
naming convention. Thus they for each lexical category, such as N,
-the functions
+the overloaded functions, such as mkN, with the following cases:
mkN, for worst-case construction of N. Its type signature
+N. Its type signature
has the form
mkN : Str -> ... -> Str -> P -> ... -> Q -> N
with as many string and parameter arguments as can ever be needed to
construct an N.
-regN, for the most common cases, with just one string argument:
+- regN : Str -> N + mkN : Str -> N
For the complement-taking variants, such as V2, we provide
-
mkV2, which takes a V and all necessary arguments, such
+
+V and all necessary arguments, such
as case and preposition:
mkV2 : V -> Case -> Str -> V2 ;
-Str and produces a transitive verb with the direct
+ object case:
- dirV2 : V -> V2 ; - -- dirV2 v = mkV2 v accusative [] + mkV2 : Str -> V2 ; ++
+ mkV2 : V -> V2 ;
V2, we provide
The golden rule for the design of paradigms is that
@@ -623,6 +726,7 @@ These constants are defined in terms of parameter types and constructors
in ResGer and MorphoGer, which modules are not
visible to the application grammarian.
An important difference between MorphoGer and
@@ -669,14 +773,15 @@ in her hidden definitions of constants in Paradigms. For instance,
-- mkAdv s = {s = s ; lock_Adv = <>} ;
The lexicon belonging to LangGer consists of two modules:
StructuralGer, structural words, built by directly using
- MorphoGer.
-BasicGer, content words, built by using ParadigmsGer.
+StructuralGer, structural words, built by using both
+ ParadigmsGer and MorphoGer.
+LexiconGer, content words, built by using ParadigmsGer only.
@@ -688,67 +793,31 @@ the coverage of the paradigms gets thereby tested and that the
use of the paradigms in LexiconGer gives a good set of examples for
those who want to build new lexica.
-Detailed implementation tricks -are found in the comments of each module. -
-
-It may be handy to provide a separate module of irregular
+It is useful in most languages to provide a separate module of irregular
verbs and other words which are difficult for a lexicographer
to handle. There are usually a limited number of such words - a
few hundred perhaps. Building such a lexicon separately also
makes it less important to cover everything by the
-worst-case paradigms (mkV etc).
+worst-case variants of the paradigms mkV etc.
You can often find resources such as lists of
irregular verbs on the internet. For instance, the
-Irregular German Verbs
+Irregular German Verb page
+previously found in
+http://www.iee.et.tu-dresden.de/~wernerr/grammar/verben_dt.html
page gives a list of verbs in the
traditional tabular format, which begins as follows:
- backen (du bäckst, er bäckt) backte [buk] gebacken
+ backen (du bäckst, er bäckt) backte [buk] gebacken
befehlen (du befiehlst, er befiehlt; befiehl!) befahl (beföhle; befähle) befohlen
beginnen begann (begönne; begänne) begonnen
beißen biß gebissen
@@ -770,25 +839,47 @@ the table to
When using ready-made word lists, you should think about
-coyright issues. Ideally, all resource grammar material should
-be provided under GNU General Public License.
+coyright issues. All resource grammar material should
+be provided under GNU Lesser General Public License (LGPL).
+
Lexicon extraction from raw text data
This is a cheap technique to build a lexicon of thousands
of words, if text data is available in digital format.
-See the Functional Morphology
+See the Extract Homepage
homepage for details.
-Extending the resource grammar API
+
+Bootstrapping with smart paradigms
+
+This is another cheap technique, where you need as input a list of words with
+part-of-speech marking. You initialize the lexicon by using the one-argument
+mkN etc paradigms, and add forms to those words that do not come out right.
+This procedure is described in the paper
+
+
+A. Ranta.
+How predictable is Finnish morphology? An experiment on lexicon construction.
+In J. Nivre, M. Dahllöf and B. Megyesi (eds),
+Resourceful Language Technology: Festschrift in Honor of Anna Sågvall Hein,
+University of Uppsala,
+2008.
+Available from the series homepage
+
+
+Extending the resource grammar API
Sooner or later it will happen that the resource grammar API
does not suffice for all applications. A common reason is
that it does not include idiomatic expressions in a given language.
The solution then is in the first place to build language-specific
-extension modules. This chapter will deal with this issue (to be completed).
+extension modules, like ExtraGer.
-Writing an instance of parametrized resource grammar implementation
+
+Using parametrized modules
+
+Writing an instance of parametrized resource grammar implementation
Above we have looked at how a resource implementation is built by
the copy and paste method (from English to German), that is, formally
@@ -802,12 +893,12 @@ use parametrized modules. The advantages are
-In this chapter, we will look at an example: adding Italian to -the Romance family (to be completed). Here is a set of +Here is a set of slides on the topic.
-
This is the most demanding form of resource grammar writing.
We do not recommend the method of parametrizing from the
@@ -817,11 +908,60 @@ same family by aprametrization. This means that the copy and
paste method is still used, but at this time the differences
are put into an interface module.
-This chapter will work out an example of how an Estonian grammar -is constructed from the Finnish grammar through parametrization. +This section is relevant for languages using a non-ASCII character set. +
+ ++From version 3.0, GF follows a simple encoding convention: +
++ flags coding = utf8 ; ++ in each source module. +
coding flag
+gfo) and the Portable Grammar Format (pgf)
+ are in UTF-8
++Most current resource grammars use isolatin-1 in the source, but this does +not affect their use in parallel with grammars written in other encodings. +In fact, a grammar can be put up from modules using different codings. +
++Warning. While string literals may contain any characters, identifiers +must be isolatin-1 letters (or digits, underscores, or dashes). This has to +do with the restrictions of the lexer tool that is used. +
+ ++While UTF-8 is well supported by most web browsers, its use in terminals and +text editors may cause disappointment. Many grammarians therefore prefer to +use ASCII transliterations. GF 3.0beta2 provides the following built-in +transliterations: +
+
+New transliterations can be defined in the GF source file
+GF/Text/Transliterations.hs.
+This file also gives instructions on how new ones are added.