This output format converts a GF grammar to a "canonical" GF grammar. A
canonical GF grammar consists of
- one self-contained module for the abstract syntax
- one self-contained module per concrete syntax
The concrete syntax modules contain param, lincat and lin definitions,
everything else has been eliminated by the partial evaluator, including
references to resource library modules and functors. Record types
and tables are retained.
The -output-format canonical_gf option writes canonical GF grammars to a
subdirectory "canonical/". The canonical GF grammars are written as
normal GF ".gf" source files, which can be compiled with GF in the normal way.
The translation to canonical form goes via an AST for canonical GF grammars,
defined in GF.Grammar.Canonical. This is a simple, self-contained format that
doesn't cover everyting in GF (e.g. omitting dependent types and HOAS), but it
is complete enough to translate the Foods and Phrasebook grammars found in
gf-contrib. The AST is based on the GF grammar "GFCanonical" presented here:
https://github.com/GrammaticalFramework/gf-core/issues/30#issuecomment-453556553
The translation of concrete syntax to canonical form is based on the
previously existing translation of concrete syntax to Haskell, implemented
in module GF.Compile.ConcreteToHaskell. This module could now be reimplemented
and simplified significantly by going via the canonical format. Perhaps exports
to other output formats could benefit by going via the canonical format too.
There is also the possibility of completing the GFCanonical grammar
mentioned above and using GF itself to convert canonical GF grammars to
other formats...
Traditionally, GF_LIB_PATH points to something like
`.../share/ghc-8.0.2-x86_64/gf-3.9/lib`
and if you want prelude and alltenses and present, you add a
`--# -path=.:present`
compiler pragma to the top of your .gf file
But if you are developing some kind of application grammar
library or contrib of your own, you might find yourself
repeating your library path at the top of all your .gf files.
After painstakingly maintaining the same library path at the
top of all your .gf files, you might say, let's factor this
out into GF_LIB_PATH.
Then you might then find to your surprise that GF_LIB_PATH
doesn't accept the usual colon:separated:path notation
familiar from, say, unix PATH and MANPATH.
This patch allows you to define
`GF_LIB_PATH=gf-3.9.lib:$HOME/gf-contrib/whatever/lib`
in a more natural way.
If you are an RGL hacker and have your own version of the
RGL tree sitting somewhere, you should be able to have both
paths in the GF_LIB_PATH, for added convenience. This minor
convenience will probably lead to obscure bugs and great
frustration when you find that your changes are mysteriously
not being picked up by GF; so keep this in mind and use it
cautiously.
This caution should probably sit in the documentation
somewhere. A subsequent commit will do that.
If you use zsh, you can do this to quickly build up a big
GF_LIB_PATH:
% gf_lib_path=( $HOME/src/GF/lib/src/{api,abstract,common,english,api/libraryBrowser,prelude,..} )
% typeset -xT GF_LIB_PATH gf_lib_path
* In GHC 8.4.1, the operator <> has become a method of the Semigroup class
and is exported from the Prelude. This is unfortunate, since <> is also
exported from the standard library module Text.PrettyPrint, so in any
module that defines a pretty printer, there is likely to be an ambiguity.
This affects ~18 modules in GF. Solution:
import Prelude hiding (<>)
This works also in older versions of GHC, since GHC does't complain if
you hide something that doesn't exists.
* In GHC 8.4.1, Semigroup has become a superclass of Monoid. This means
that anywhere you define an instance of the Monoid class you also have to
define an instance in the Semigroup class.
This affects Data.Binary.Builder in GF. Solution: conditionally define
a Semigroup instance if compiling with base>=4.11 (ghc>=8.4.1)
+ Generalize the CommandInfo type by parameterizing it on the monad
instead of just the environment.
+ Generalize the commands defined in
GF.Command.{Commands,Commands2,CommonCommands,SourceCommands,HelpCommand}
to work in any monad that supports the needed operations.
+ Liberate GF.Command.Interpreter from the IO monad.
Also, move the current PGF from CommandEnv to GFEnv in
GF.Interactive, making the command interpreter even more generic.
+ Use a state monad to maintain the state of the interpreter in
GF.{Interactive,Interactive2}.
Some C run-time functionality is now available in the GF shell, by starting
GF with 'gf -cshell' or 'gf -crun'. Only limited functionality is available
when running the shell in these modes:
- You can only import .pgf files, not source files.
- The -retain flag can not be used and the commands that require it to work
are not available.
- Only 18 of the 40 commands available in the usual shell have been
implemented. The 'linearize' and 'parse' commands are the only ones
that call the C run-time system, and they support only a limited set of
options and flags. Use the 'help' commmands for details.
- A new command 'generate_all', that calls PGF2.generateAll, has been added.
Unfortuntaly, using it causes 'segmentation fault'.
This is implemented by adding two new modules: GF.Command.Commands2 and
GF.Interactive2. They are copied and modified versions of GF.Command.Commands
and GF.Interactive, respectively. Code for unimplemented commands and other
code that has not been adapted to the C run-time system has been left in
place, but commented out, pending further work.
Because the prompt included the name of the abstract syntax, the loading
of the PGF was forced even if -retain was used. Even worse,
if an error occured while loading the PGF, it was repeated and caught
every time the prompt was printed, creating an infite loop. The solution
is to not print the name of the abstract syntax when the grammar is
imported with -retain, which is the way things were before anyway.
The commands available in the shell after import -retain are now a superset
of the commands available after import without -retain.
The PGF is created lazily, so there should be no performance penalty if
the PGF isn't needed. If there are errors, they won't be reported until a
command that uses the PGF is entered.
2 modules: Name clashes caused by Applicative-Monad change in Prelude
2 modules: Ambiguities caused by Foldable/Traversable in Prelude
2 modules: Backwards incompatible changes in time-1.5 for defaultTimeLocale
9 modules: {-# LANGUAGE FlexibleContexts #-} (because GHC checks inferred types
now, in addition to explicitly given type signatures)
Also silenced warnings about tab characters in source files.
By adding the flag -haskell=variants to the command line, GF will now generate
linearization functions in Haskell that support variants. Variants are
represented as lists in Haskell.
Variants inside pre { ... } expressions are still ignored.
TODO: apply some monad laws to generate more compact code (using an
intermediate representation of the generated Haskell code, instead of
pretty printing directly from the GF code).
The translation is currently good enough to translate all concrete syntaxes
of the Foods and Letter grammars, and some concrete syntaxes of the Phrasebook
grammar (e.g. PhrasebookEng & PhrasebookSpa works, but there are problems with
e.g. PhrasebookSwe and PhrasebookChi)
This functionality is enabled by running
gf -make -output-format=haskell -haskell=concrete ...
TODO:
- variants
- pre { ... }
- eta expansion of linearization functions
- record subtyping can still cause type errors in the Haskell code
in some cases
- reduce code large tables
Renamed appIOE to tryIOE (it is analogous to 'try' in the standard libraries).
Removed unused IOE operations & documented the remaining ones.
Removed/simplified superfluous uses of IOE operations.
Included renamings:
SourceGrammar -> Grammar
SourceModule -> Module
SourceModInfo -> ModuleInfo
emptySourceGrammar -> emptyGrammar
Also introduces a type synonym (which might be good to turn into a newtype):
type ModuleName = Ident
The reason is to make types like the following more self documenting:
type Module = (ModuleName,ModuleInfo)
type QIdent = (ModuleName,Ident)
IOE used to be a monad with extra error handling built on top of the IO monad,
But the IO monad already supports error handling, so this construction was a
superfluous.
The new 'instance ErrorMonad IOE' is defined to preserve the previous error
handling behaviour, i.e. the function 'handle' only catches errors thrown with
'raise' (or 'fail') and not other errors in the IO monad.
* The following modules are no longer used and have been removed completely:
GF.Compile.Compute.ConcreteLazy
GF.Compile.Compute.ConcreteStrict
GF.Compile.Refresh
* The STM monad has been commented out. It was only used in
GF.Compile.SubExpOpt, where could be replaced with a plain State monad,
since no error handling was needed. One of the functions was hardwired to
the Err monad, but did in fact not use error handling, so it was turned
into a pure function.
* The function errVal has been renamed to fromErr (since it is analogous to
fromMaybe).
* Replaced 'fail' with 'raise' and 'return ()' with 'done' in a few places.
* Some additional old code that was already commented out has been removed.
The module src/compiler/GF.hs now serves as a prelimiary compiler API. It just
exports a selection of functions and types from the compiler.
Haddock documentation can be generated with
cabal haddock --hyperlink-source
Also bumbed the version number to 3.6.10.
(1) introduces the module GF.Infra.Concurreny with lifted concurrency
operators (to reduce uses of liftIO) and some additional concurrency
utilities, e.g. a function for sequential logging that is used in
both GF.CompileInParallel and GFServer.
(2) avoids leaving broken .gfo files behind if compilation is aborted.
On my laptop these changes speed up the full build of the RGL and example
grammars with 'cabal build' from ~95s to ~43s and the zero build from ~18s
to ~5s.
The main change is the introduction of the module GF.CompileInParallel that
replaces GF.Compile and the function GF.Compile.ReadFiles.getAllFiles. At
present, it is activated with the new -j flag, and it is only used when
combined with --make or --batch. In addition, to get parallel computations,
you need to add GHC run-time flags, e.g., +RTS -N -A20M -RTS, to the command
line.
The Setup.hs script has been modified to pass the appropriate flags to GF
for parallel compilation when compiling the RGL and example grammars, but you
need a recent version of Cabal for this to work (probably >=1.20).
Some additonal refactoring were made during this work. A new monad is used to
avoid warnings/error messages from different modules to be intertwined when
compiling in parallel, so some functios that were hardiwred to the IO or IOE
monads have been lifted to work in arbitrary monads that are instances in
the appropriate classes.
In particular, the function compileOne has been moved to the new module
GF.CompileOne and its type has been changed from
compileOne :: ... -> CompileEnv -> FilePath -> IOE CompileEnv
to
compileOne :: ... -> SourceGrammar -> FilePath -> IOE OneCompiledModule
making it more suitable for use in a parallel compiler.
GF.Text.Pretty provides the class Pretty and overloaded versions of the pretty
printing combinators in Text.PrettyPrint, allowing pretty printable values to
be used directly instead of first having to convert them to Doc with functions
like text, int, char and ppIdent. Some modules have been converted to use
GF.Text.Pretty, but not all. Precedences could be added to simplify the pretty
printers for terms and patterns.
GF.Infra.Location contains the types Location and L, factored out from
GF.Grammar.Grammar, and the class HasSourcePath. This allowed the import
of GF.Grammar.Grammar to be removed from GF.Infra.CheckM, making it more
like a pure library module.
PGF exports the public, stable API.
PGF.Internal exports additional things needed in the GF compiler & shell,
including the nonstardard version of Data.Binary.
If the C run-time library is compiled and installed on your system, you can now
do 'cabal configure -fc-runtime' to get the following extras:
+ The haskell binding to the C run-time library will be included in the
PGF library (so you can import it in Haskell applications).
Documentation on the new modules will be included when you run
'cabal haddock'.
+ The new command 'pgf-shell', implemented on top of haskell binding to
the C run-time system.
+ Three new commands in the web API: c-parse, c-linearize and
c-translate. Their interfaces are similar to the corresponding commands
without the "c-" prefix, but they should be considered preliminary.
This means that the -old-comp and -new-comp flags are not recognized anymore.
The only functional difference is that printnames were still normalized with
the old partial evaluator. Now that is done with the new partial evaluator.
This was a fairly simple change thanks to previous work on making the Ident
type abstract and the fact that PGF.CId already uses UTF-8-encoded
ByteStrings.
One potential pitfall is that Data.ByteString.UTF8 uses the same type for
ByteStrings as Data.ByteString. I renamed ident2bs to ident2utf8 and
bsCId to utf8CId, to make it clearer that they work with UTF-8-encoded
ByteStrings.
Since both the compiler input and identifiers are now UTF-8-encoded
ByteStrings, the lexer now creates identifiers without copying any characters.
**END OF DESCRIPTION***
Place the long patch description above the ***END OF DESCRIPTION*** marker.
The first line of this file will be the patch name.
This patch contains the following changes:
M ./src/compiler/GF/Compile/CheckGrammar.hs -3 +3
M ./src/compiler/GF/Compile/GrammarToPGF.hs -2 +2
M ./src/compiler/GF/Grammar/Binary.hs -5 +1
M ./src/compiler/GF/Grammar/Lexer.x -11 +13
M ./src/compiler/GF/Infra/Ident.hs -19 +36
M ./src/runtime/haskell/PGF.hs -1 +1
M ./src/runtime/haskell/PGF/CId.hs -2 +3
