The length limit test previously used the URL-encoded UTF-8 representation
of the source sentense. This was needed because of a fixed size buffer in C.
Now that the server is in Haskell, the only reason the length is limited
is to avoid excessive time and space use in the parser, so it is better to
count source characters. This also avoids being too restrictive with
non-European languages.
For further separation of pretty printing concerns from conversion concerns,
the Haskell AST and pretty printer has been moved to its own module,
GF.Haskell, also allowing it to be reused in other places where Haskell
code is generated.
The parse/translate/c-parse/c-translate commands now recognize the option
jsontree=true to augment the returned JSON structure with a field called
"jsontree" next to the field "tree", or "jsontrees" next to "trees",
containing the the returned syntax tree in JSON format (the same format
returned by the abstrjson command, similar to the format returned in the
"brackets" field).
Too see inflection tables, click on a translated sentense, then click on a
blue function name in the textual representation of the abstract syntax tree.
You can now click on a tree to toggle between the abstract syntax tree and
the parse tree.
Also, the implementation now uses the new C run-time requests, e.g.
App14.pgf?command=c-abstrtree&tree=...
so no need to install AppEng.pgf on the server any more.
The trees are drawn by using the Haskell run-time request
AppEng.pgf?command=abstrtree&tree=...
Thus AppEng.pgf has to be installed on the server and in sync with App14.pgf.
This is to avoid loading the huge App14.pgf in the Haskell run-time system, and
is enough since we only need the abstract syntax.
For improved performance, request only one translation from the server
initially. When the user clicks on a translated sentense, request 9 more
translations.
If the enumaration of table parameter values fails during the static
traversal phase, try again in the dynamic computation phase, when the values
of bound variables are known.
This is necessary to properly deal with generic table construction in opers,
like the ones found in prelude/Coordination.gf, e.g.
consTable : (P : PType) -> ... = \P ... -> {s1 = table P {...} ; ... }
GF.Compile.Optimize.mkLinReference can fail and cause this error because
the helper function inside it applies msum to a list that might be empty
(if there is a record type that does not contain a field of type Str).
This means that it can return mzero::Err, i.e.
Bad "error (no reason given)"
which can slip through the top level test that only catches Bad "no string".
Because the prompt included the name of the abstract syntax, the loading
of the PGF was forced even if -retain was used. Even worse,
if an error occured while loading the PGF, it was repeated and caught
every time the prompt was printed, creating an infite loop. The solution
is to not print the name of the abstract syntax when the grammar is
imported with -retain, which is the way things were before anyway.
The commands available in the shell after import -retain are now a superset
of the commands available after import without -retain.
The PGF is created lazily, so there should be no performance penalty if
the PGF isn't needed. If there are errors, they won't be reported until a
command that uses the PGF is entered.
2 modules: Name clashes caused by Applicative-Monad change in Prelude
2 modules: Ambiguities caused by Foldable/Traversable in Prelude
2 modules: Backwards incompatible changes in time-1.5 for defaultTimeLocale
9 modules: {-# LANGUAGE FlexibleContexts #-} (because GHC checks inferred types
now, in addition to explicitly given type signatures)
Also silenced warnings about tab characters in source files.
+ Some additional simplifying rewrites.
+ Use an intermediate representation for Haskell types, for separation of
concerns and cleaner code.
+ Pretty printer layout tuning
+ Code cleanup.
Introduced an intermediate representation for the generated Haskell expressions.
This allows pretty printing concerns to be separated from conversion concerns,
and makes it easy to apply some simplifying rewrites to the generated
expressions, e.g.
[x] ++ [y] ==> [x,y]
pure f <*> x ==> f <$> x
f <$> pure x ==> pure (f x)
join (pure x) ==> x
By adding the flag -haskell=variants to the command line, GF will now generate
linearization functions in Haskell that support variants. Variants are
represented as lists in Haskell.
Variants inside pre { ... } expressions are still ignored.
TODO: apply some monad laws to generate more compact code (using an
intermediate representation of the generated Haskell code, instead of
pretty printing directly from the GF code).
Move the Haskell representation of the common linearization type {s:T} to the
shared module PGF.Haskell, so that the same overloaded projection function
proj_s can be used for all concrete syntaxes.
Common code has been lifted out from the generated Haskell modules to
an auxiliary module PGF.Haskell, which is currently included in the
regular PGF library, although it is independent of it and probably belongs
in a separate library.
The type Str used by linearization functions is now based on a token
type Tok, which is defined in PGF.Haskell.
PGF.Haskell.Tok is similar to the type GF.Data.Str.Tok, but it has
constructors for the special tokens BIND, SOFT_BIND and CAPIT, and there is
a function
fromStr :: Str -> String
that computes the effects of these special tokens.
+ Instead of including lists of parameter values generated by GF, generate
code to enumerate parameter values (in the same order as GF). This seems
to give a factor of 2-3 code size reduction in the Phrasebook (e.g.
from 84MB to 25MB for Hin, from 338MB to 154MB for Fre).
+ Deduplicate table entries, i.e. convert "table [..,E,..,E,..,E,..]" into
"let x = E in table [..,x,..,x,..,x,..]". This gives even more significant
code size reduction in some cases, e.g. from 569MB to 15MB for
PhrasebookFin.
All phrasebook languages can now be converted to compilable Haskell code,
except PhrasebookPes, which still has the name clash problem.
Many Phrasebook languages can now be converted to compilable Haskell code.
Some languages (Fre, Hin, Snd, Urd) generate too much Haskell code to be
practically useful (e.g. 338MB for Fre). One language (Fin) took too long
to convert to Haskell. One language (Pes) has problems with name clashes in
the generated Haskell code.
STILL TODO:
- variants
- pre { ... }
- reduce code duplication for large tables
- generate qualified names to avoid name clashes
When mkPresent (or another preprocessor) is used, error messages from GF will
show the file name _gf_preproc.tmp instead of the name of the file where
the error occurred. By compiling Alltenses first, the real file name will
appear in error messages (unless the error only happens then mkPresent is
used).
The translation is currently good enough to translate all concrete syntaxes
of the Foods and Letter grammars, and some concrete syntaxes of the Phrasebook
grammar (e.g. PhrasebookEng & PhrasebookSpa works, but there are problems with
e.g. PhrasebookSwe and PhrasebookChi)
This functionality is enabled by running
gf -make -output-format=haskell -haskell=concrete ...
TODO:
- variants
- pre { ... }
- eta expansion of linearization functions
- record subtyping can still cause type errors in the Haskell code
in some cases
- reduce code large tables
It was used only in cases where a lock field needed to be added to a
run-time variable, like e.g. in examples/phrasebook/SentencesTha.gf:
lin
PGreetingMale g = mkText (lin Text g) (lin Text (ss "ครับ")) | g ;
PGreetingFemale g = mkText (lin Text g) (lin Text (ss "ค่ะ")) | g ;
But lock fields are only meaningful during type checking and can safely be
ignored in later passes.
Renamed appIOE to tryIOE (it is analogous to 'try' in the standard libraries).
Removed unused IOE operations & documented the remaining ones.
Removed/simplified superfluous uses of IOE operations.
This replaces the hardwired ANSI escape codes that were accidentally included
in a previous patch.
This adds a dependency on terminfo, but this should be unproblematic, since
haskeline already depends on the same underlying C library.
The color highlighting is omitted on Windows.
Included renamings:
SourceGrammar -> Grammar
SourceModule -> Module
SourceModInfo -> ModuleInfo
emptySourceGrammar -> emptyGrammar
Also introduces a type synonym (which might be good to turn into a newtype):
type ModuleName = Ident
The reason is to make types like the following more self documenting:
type Module = (ModuleName,ModuleInfo)
type QIdent = (ModuleName,Ident)
IOE used to be a monad with extra error handling built on top of the IO monad,
But the IO monad already supports error handling, so this construction was a
superfluous.
The new 'instance ErrorMonad IOE' is defined to preserve the previous error
handling behaviour, i.e. the function 'handle' only catches errors thrown with
'raise' (or 'fail') and not other errors in the IO monad.
* The following modules are no longer used and have been removed completely:
GF.Compile.Compute.ConcreteLazy
GF.Compile.Compute.ConcreteStrict
GF.Compile.Refresh
* The STM monad has been commented out. It was only used in
GF.Compile.SubExpOpt, where could be replaced with a plain State monad,
since no error handling was needed. One of the functions was hardwired to
the Err monad, but did in fact not use error handling, so it was turned
into a pure function.
* The function errVal has been renamed to fromErr (since it is analogous to
fromMaybe).
* Replaced 'fail' with 'raise' and 'return ()' with 'done' in a few places.
* Some additional old code that was already commented out has been removed.
The module src/compiler/GF.hs now serves as a prelimiary compiler API. It just
exports a selection of functions and types from the compiler.
Haddock documentation can be generated with
cabal haddock --hyperlink-source
Also bumbed the version number to 3.6.10.
loadConcr, unloadConcr and addLiteral modify the Concr structure as a side
effect. This means that other functions with a Concr argument (e.g. parse
and linearize) are no longer pure.
Possible solutions:
1. Don't try to hide the imperative nature of the C run-time system: remove
all uses of unsafePerformIO and let all functions operate in the IO monad.
2. Don't export functions with side effects. Perhaps the desired functionality
of loadConcr, unloadConcr and addLiteral can be folded into readPGF.
The Concr structures can then treaded as immutable after after the
readPGF function returns...
(1) introduces the module GF.Infra.Concurreny with lifted concurrency
operators (to reduce uses of liftIO) and some additional concurrency
utilities, e.g. a function for sequential logging that is used in
both GF.CompileInParallel and GFServer.
(2) avoids leaving broken .gfo files behind if compilation is aborted.
* httpd-shed-0.4 does not specify an upper bound on network, but it fails
to build against network>=2.6. This is fixed in httpd-shed-0.4.0.2.
* With network-2.6, the Network.URI modules is moved to a separate package,
so for the time being GF requires network>=2.3 && <2.6. This is compatible
with the four most recent versions of the Haskell Platform.
* Introducing the module CGI, re-exporting a subset of the cgi package. It
might complete replace the cgi package in the future.
* Introducing the module CGIUtils, containing functions from FastCGIUtils that
have nothing to do with fastcgi.
Some low level hackery with unsafePerformIO and global variables was left
in FastCGIUtils, but it is actually not used, neither for gf -server nor
exec/pgf-fcgi.hs.
This makefile just calls GF once and lets GF figure out in which order to
compile things. It uses the -j flag to enable parallel compilation and
specifies an explicit -path, overriding the -path flags in the source files.
This allows all needed modules to be found automatically and ensures that
that alltenses is consistently used everywhere. But for some reason, this
doesn't work...
The script bin/build-binary-dist.sh has been updated to build either a plain
.tar.gz package or OS X Installer package (.pkg).
Note that bin/build-binary-dist.sh is designed to build and include the
C run-time system in the binary package. If the C run-time system fails to
build, no binary package will be created.
On my laptop these changes speed up the full build of the RGL and example
grammars with 'cabal build' from ~95s to ~43s and the zero build from ~18s
to ~5s.
The main change is the introduction of the module GF.CompileInParallel that
replaces GF.Compile and the function GF.Compile.ReadFiles.getAllFiles. At
present, it is activated with the new -j flag, and it is only used when
combined with --make or --batch. In addition, to get parallel computations,
you need to add GHC run-time flags, e.g., +RTS -N -A20M -RTS, to the command
line.
The Setup.hs script has been modified to pass the appropriate flags to GF
for parallel compilation when compiling the RGL and example grammars, but you
need a recent version of Cabal for this to work (probably >=1.20).
Some additonal refactoring were made during this work. A new monad is used to
avoid warnings/error messages from different modules to be intertwined when
compiling in parallel, so some functios that were hardiwred to the IO or IOE
monads have been lifted to work in arbitrary monads that are instances in
the appropriate classes.
hallgren