These changes are inspired by the gf -cshell implementation of these commands.
The output of the linearize command has been changed to remove superfluous
blank lines and commas, and deliver the result as a list of strings instead of
a single multi-line string. This makes it possible to use -all and pipe the
results to the parse command. This also means that with -treebank -all,
the language tag will be repeated for each result from the same language.
The parse command, when trying to parse with more than one language, would
"forget" other results after a failed parse, and thus not send all
successful parses through the pipe. For example, if English is not the first
language in the grammar,
p "hello" | l
would output nothing, instead of translations of "hello" to all languages,
forcing the user to write
p -lang=Eng "hello" | l
instead, to get the expected result. The cause of this behaviour was in the
function fromParse, which was rather messy, so I assume it is not intentional,
but the result of a programming mistake at some point.
The fromParse function has now been refactored from a big recursive function
into
fromParse opts = foldr (joinPiped . fromParse1 opts) void
where the helper functions fromParse1 deals with a single parse result and
joinPiped combines multiple parse results.
While the abstract Numeral inherits only Cat[Numeral,Digits], some of the
concrete NumeralNNN of Numeral inherited everything from CatNNN.
Normally the compiler outputs a warning message when a concrete syntax
contains superflous lincats, but apparently not when they get included
through inheritance...
This does not seem to cause problems in the Haskell run-time system, but the
C run-time system fails to load PGFs with superflous lincats.
This problems shows up when creating application grammars that inherit Numeral.
The Phrasebook is an example of such a grammar.
Even though the -lang flag was handled in the implementation, it was not
documented, and GF.Command.Interpreter rejects undocumented flags:
option not interpreted: lang
This must be a fairly old bug, so it suggests that the vp command isn't used
much...
pg supports only the -funs, -cats and -langs output modes.
ai IDENTIFIER shows info about a category or a function. ai can not type check
and refine metavariables in expressions.
showType :: Type -> String
categories :: PGF -> [Cat]
But both are implemented as quick hacks: categories is implemented by listing
all functions and taking the target categories from their types. showType uses
ppType copied & modified from PGF.Type, and needs a ppExpr, which is currently
implemented by wrapping showExpr...
TODO: need something correpsonding to PGF.categoryContext.
Options -all and -list use PGF2.linearizeAll, which lists all variants, but
not all forms...
Also, there is no attempt to be compatible with the output from the Haskell
run-rime shell, which produces superfluous blank lines (-all) or
commas (-list), and mixes tagged and untagged lines (-treebank -all).
The print_history command was among the commands implemented in an ad-hoc
way instead of being handled by the command line interpreter, which means
it could not be used in a pipe, as in the example in the help info.
The refactoring in the previous patch made this old bug easy to fix.
Also fixed a bug in the "empty" command, introduced when moving the PGF from
CommandEnv to GFEnv.
TODO: fix the undocumented eh command. A comment in the help info for
print_history, and some commented out old code, suggest that eh means
"execute_history", but at present it does nothing...
+ Generalize the CommandInfo type by parameterizing it on the monad
instead of just the environment.
+ Generalize the commands defined in
GF.Command.{Commands,Commands2,CommonCommands,SourceCommands,HelpCommand}
to work in any monad that supports the needed operations.
+ Liberate GF.Command.Interpreter from the IO monad.
Also, move the current PGF from CommandEnv to GFEnv in
GF.Interactive, making the command interpreter even more generic.
+ Use a state monad to maintain the state of the interpreter in
GF.{Interactive,Interactive2}.
These commands are now implemented as regular commands (i.e. using the
CommandInfo data type) in the new module GF.Command.SourceCommands.
The list of commands exported from GF.Command.Commmands now called pgfCommands
instead of allCommands.
The list allCommands of all commands is now assembled
from sourceCommands, pgfCommands, commonCommands and helpCommand in
GF.Interactive.
Created module GF.Command.CommonCommands with ~250 lines of code for commands
that do not depend on the type of PGF in the environemnt, either because they
don't use the PGF or because they are just documented here and implemented
elsewhere.
TODO: further refactoring so that documentation and implementation of
*all* commands can be kept together.
Some C run-time functionality is now available in the GF shell, by starting
GF with 'gf -cshell' or 'gf -crun'. Only limited functionality is available
when running the shell in these modes:
- You can only import .pgf files, not source files.
- The -retain flag can not be used and the commands that require it to work
are not available.
- Only 18 of the 40 commands available in the usual shell have been
implemented. The 'linearize' and 'parse' commands are the only ones
that call the C run-time system, and they support only a limited set of
options and flags. Use the 'help' commmands for details.
- A new command 'generate_all', that calls PGF2.generateAll, has been added.
Unfortuntaly, using it causes 'segmentation fault'.
This is implemented by adding two new modules: GF.Command.Commands2 and
GF.Interactive2. They are copied and modified versions of GF.Command.Commands
and GF.Interactive, respectively. Code for unimplemented commands and other
code that has not been adapted to the C run-time system has been left in
place, but commented out, pending further work.
+ Move type CommandInfo from GF.Command.Commands to a new module
GF.Commands.CommandInfo and make it independent of the PGF type.
+ Make the module GF.Command.Interpreter independent of the PGF type and
eliminate the import of GF.Command.Commands.
+ Move the implementation of the "help" command to its own module
GF.Command.Help
By adding unlexer=none (or unlexer=id) in requests that output linearizations
(e.g. command=linearize, command=translate), you can leave &+ uninterpreted
instead of gluing the adjacent tokens. This means that the output is left in
a format that can be parsed in a subsequent request.
To implement this consistently, the function linearizeAndBind was replaced
with the function linearizedAndUnlex (but there are a couple of requests
that do not call this function...)
Note that this applies to the Haskell run-time requests only. The C run-time
request (c-linearize, c-translate) always applies the &+ token and the
c-parse request can parse input containing glued tokens.
Becacuse of the new special tokens added to the Symbol type, .gfo and .pgf
files produced with the current version of GF can not always be used with
older versions of GF and the PGF run-time system.
The PGF version number was increased from (2,0) to (2,1). GF can still
read version (2,0) and (1,0), so old PGF files continue to work.
The GFO version was increased from "GF03" to "GF04".
The minibar lets the user choose any category as the start category, but this
can lead to run-time errors for grammars compiled with -optimize-pgf.
The problem can also be observed in the GF shell. For example,
"linearize Fish" (which means that the start category is Kind) fails for
some languages when Foods.pgf is compiled with -optimize-pgf.
When using make -j to compile examples/app or examples/phrasebook, since
the dependencies are not completely specified in the Makefiles, it can
happen that the same file is compiled at the same by more than one process,
resulting in an error when they try to write the same .gfo.tmp file. Adding a
random number to the temporary file name avoids this problem.
The length limit test previously used the URL-encoded UTF-8 representation
of the source sentense. This was needed because of a fixed size buffer in C.
Now that the server is in Haskell, the only reason the length is limited
is to avoid excessive time and space use in the parser, so it is better to
count source characters. This also avoids being too restrictive with
non-European languages.
For further separation of pretty printing concerns from conversion concerns,
the Haskell AST and pretty printer has been moved to its own module,
GF.Haskell, also allowing it to be reused in other places where Haskell
code is generated.
The parse/translate/c-parse/c-translate commands now recognize the option
jsontree=true to augment the returned JSON structure with a field called
"jsontree" next to the field "tree", or "jsontrees" next to "trees",
containing the the returned syntax tree in JSON format (the same format
returned by the abstrjson command, similar to the format returned in the
"brackets" field).
Too see inflection tables, click on a translated sentense, then click on a
blue function name in the textual representation of the abstract syntax tree.
You can now click on a tree to toggle between the abstract syntax tree and
the parse tree.
Also, the implementation now uses the new C run-time requests, e.g.
App14.pgf?command=c-abstrtree&tree=...
so no need to install AppEng.pgf on the server any more.
The trees are drawn by using the Haskell run-time request
AppEng.pgf?command=abstrtree&tree=...
Thus AppEng.pgf has to be installed on the server and in sync with App14.pgf.
This is to avoid loading the huge App14.pgf in the Haskell run-time system, and
is enough since we only need the abstract syntax.
For improved performance, request only one translation from the server
initially. When the user clicks on a translated sentense, request 9 more
translations.
If the enumaration of table parameter values fails during the static
traversal phase, try again in the dynamic computation phase, when the values
of bound variables are known.
This is necessary to properly deal with generic table construction in opers,
like the ones found in prelude/Coordination.gf, e.g.
consTable : (P : PType) -> ... = \P ... -> {s1 = table P {...} ; ... }
GF.Compile.Optimize.mkLinReference can fail and cause this error because
the helper function inside it applies msum to a list that might be empty
(if there is a record type that does not contain a field of type Str).
This means that it can return mzero::Err, i.e.
Bad "error (no reason given)"
which can slip through the top level test that only catches Bad "no string".
Because the prompt included the name of the abstract syntax, the loading
of the PGF was forced even if -retain was used. Even worse,
if an error occured while loading the PGF, it was repeated and caught
every time the prompt was printed, creating an infite loop. The solution
is to not print the name of the abstract syntax when the grammar is
imported with -retain, which is the way things were before anyway.
The commands available in the shell after import -retain are now a superset
of the commands available after import without -retain.
The PGF is created lazily, so there should be no performance penalty if
the PGF isn't needed. If there are errors, they won't be reported until a
command that uses the PGF is entered.
hallgren