krasimir
7ee9edd080
fix in the dictionary lookup
2015-05-08 18:12:24 +00:00
krasimir
b87ed432a5
a new nice and elegant algorithm for dictionary lookup in the parser
2015-05-08 14:38:24 +00:00
krasimir
e600eb764a
the parser is now fully Unicode compatible
2015-05-08 09:23:29 +00:00
krasimir
35c4b5f3eb
remove the meta_item variable from the parse state
2015-03-06 15:18:50 +00:00
krasimir
0764d518c7
fix some out-of-date code for the PGF_RESULT_DEBUG mode in the parser
2015-02-26 13:31:26 +00:00
krasimir
42908bd95f
use the same trick for comparison of doubles in cmp_expr_state
2015-02-26 13:10:01 +00:00
krasimir
dfd69d5d4b
bugfix in the parser which caused some items to be poped in the wrong order if their probability differ with less that 0.5
2015-02-26 12:17:59 +00:00
krasimir
79a7ae6553
remove three more uses of meta productions that were left in debugging mode
2015-02-24 10:59:18 +00:00
krasimir
2f35aadc6f
parsing with meta rules is now removed since we don't use them anymore and I would gladly remove some code.
2015-02-19 11:59:51 +00:00
krasimir
27a12ca9fb
fix in the parser for callbacks in the middle of a word
2015-02-18 16:14:29 +00:00
kr.angelov
9b7e18c25e
change in the API for literals
...
The API in the C runtime as well as in the Haskell, Python and Java binding
is changed. Now instead of adding the literal callbacks to the concrete syntax
you need to supply them every time when you need to parse. The main reason is:
- referentially transparent API for Haskell
- when we start using memory mapped files we will not be allowed to change
anything in the grammar data structures. At that point the old API would
be impossible to use.
2014-12-16 10:21:26 +00:00
kr.angelov
995735b272
fix the last patch
2014-12-01 11:03:50 +00:00
kr.angelov
89078c3598
a small optimization in the PGF parser which gives me ~5% speed up
2014-12-01 10:48:15 +00:00
kr.angelov
80cf34bfd1
remove pgf/parser.h
2014-11-27 11:12:14 +00:00
kr.angelov
d830c75da2
fix the handling of the heuristic factor. it was broken when I added the BIND token which means that the App grammar was running about 2-3 times slower.
2014-11-26 15:56:36 +00:00
kr.angelov
b4046e1797
a simple optimization in cmp_item_prob with great effect on the parser's speed
2014-10-23 21:22:22 +00:00
kr.angelov
4efb2d5a22
Prelude.CAPIT is now a built-in primitive. It still generates &| in the Haskell runtime but will be intepreted in the C runtime
2014-10-09 19:34:12 +00:00
kr.angelov
57f7b61c6b
throw away the long obsolete runtime type information in the C runtime
2014-10-09 13:44:26 +00:00
kr.angelov
4ac5902fc9
replace the hash maps in the abstract syntax with binary search tables
2014-10-09 08:38:55 +00:00
kr.angelov
68c8d3cba8
bugfix in the handling of BIND in the C runtime
2014-04-22 13:53:18 +00:00
kr.angelov
d8e5206e7f
now word completion from Java works. It could be made better
2014-04-17 11:00:27 +00:00
kr.angelov
a9ebeaea44
now fully functional Java API for custom literals
2014-04-10 14:14:31 +00:00
kr.angelov
52377b46cf
bugfix in the C runtime that shows up with only some grammars
2014-04-08 12:41:41 +00:00
kr.angelov
709f5dfcd8
small fixes in the C runtime
2014-03-25 13:57:57 +00:00
kr.angelov
bc81d163fb
whitespace cleanup in the parser
2014-03-18 10:19:35 +00:00
kr.angelov
1a1e7cdb2e
compute the right word probability
2014-03-12 15:36:40 +00:00
kr.angelov
a77dc568bb
added pgf_lookup_word_prefix which makes it possible to do simple word prediction
2014-03-07 21:24:20 +00:00
kr.angelov
6028a63b42
loading and unloading of languages in the C runtime and in the Python bindings
2013-12-10 15:00:52 +00:00
kr.angelov
cddc19a5ad
bugfix in the debug mode for the parser
2013-11-22 12:26:49 +00:00
kr.angelov
e77c19c783
bugfix in the C runtime
2013-11-21 14:31:28 +00:00
kr.angelov
33a76f1cd5
bugfix in the robust parser
2013-11-18 09:05:27 +00:00
kr.angelov
63a90f34af
bugfix in the C runtime which had made the parser up to 200 times slower for some sentences
2013-11-13 15:15:28 +00:00
kr.angelov
1d7b6f44b6
added Predef.SOFT_BIND. This special token allows zero or more spaces between ordinary tokens. It is also used in the English RGL to attach the commas to the previous word.
2013-11-12 09:54:57 +00:00
kr.angelov
6729cca568
factor of 3 speedup for the translation without slowing down the grammar loading. The parser still seems to be 8 times slower than before I introduced the BIND. At least the Android UI now feels a lot more comfortable
2013-11-07 11:14:32 +00:00
kr.angelov
6c71830c10
fix in the parsing with pre
2013-11-06 16:25:25 +00:00
kr.angelov
475f213c99
the content of ParseEngAbs3.probs is now merged with ParseEngAbs.probs. The later is now retrained. Once the grammar is compiled with the .probs file now it doesn't need anything more to do robust parsing. The robustness itself is controlled by the flags 'heuristic_search_factor', 'meta_prob' and 'meta_token_prob' in ParseEngAbs.gf
2013-11-06 10:21:46 +00:00
kr.angelov
d7cd5ba195
fix the robust parser for unknown tokens
2013-11-01 13:05:17 +00:00
kr.angelov
8019d3a615
use linref in the C linearizer. The robust linearization is now fully supported
2013-10-30 21:43:52 +00:00
kr.angelov
43fb9b3b7a
added the linref construction in GF. The PGF version number is now bumped
2013-10-30 12:53:36 +00:00
kr.angelov
56893dbf54
another fix for 'pre'
2013-10-28 13:54:06 +00:00
kr.angelov
841880144d
fix the handling of 'pre' in the C runtime
2013-10-28 12:35:37 +00:00
kr.angelov
e3dc98dd0f
bugfix for the custom literals in the C runtime
2013-10-21 12:34:57 +00:00
kr.angelov
82544f74c0
avoid warning in parser.c when debugging is OFF
2013-10-09 13:41:02 +00:00
kr.angelov
b058fdd17d
a major redesign in the C runtime. The parser and the linearizer now fully support BIND. The following things are still broken: parseval, word completion, handling 'pre', the robust mode
2013-10-09 12:08:51 +00:00
kr.angelov
adc952a61d
fix the robust parser after the change of GuString
2013-10-08 15:00:42 +00:00
kr.angelov
2f78333e28
GuString is now an ordinary C string - it makes live easier. In addition PgfSymbolKS, PgfExprFun and PgfLiteralStr now keep their strings as embedded flexible arrays. The latest change gives us the same compactness as the old representation but it is a lot easier to use.
2013-10-04 12:04:39 +00:00
kr.angelov
c4f68adca9
better handling for nonExist in the C runtime. BIND is also supported in the linearizer but not in the parser yet
2013-09-30 10:25:39 +00:00
kr.angelov
efa4bc4d62
a major refactoring in the C and the Haskell runtimes. Note incompatible change in the PGF format!!!
...
The following are the outcomes:
- Predef.nonExist is fully supported by both the Haskell and the C runtimes
- Predef.BIND is now an internal compiler defined token. For now
it behaves just as usual for the Haskell runtime, i.e. it generates &+.
However, the special treatment will let us to handle it properly in
the C runtime.
- This required a major change in the PGF format since both
nonExist and BIND may appear inside 'pre' and this was not supported
before.
2013-09-27 15:09:48 +00:00
kr.angelov
65ea8814aa
fix the debug mode of the parser after the refactoring in libgu
2013-09-25 10:21:35 +00:00
kr.angelov
f984bfeb24
a major refactoring in the C runtime. GuList is now removed and replaced with GuSeq. The GuSeq/GuBuf API is simplified
2013-09-17 12:45:00 +00:00
