gf-core/doc/hackers-guide/DESIDERATA.md

This is an experiment to develop a majestic new GF runtime.

The reason is that there are several features that we want to have and they all require a major rewrite of the existing C runtime. Instead of beating the old code until it starts doing what we want, it is time to start from scratch.

New Features

The features that we want are:

• We want to support even bigger grammars that don't fit in the main memory anymore. Instead, they should reside on the disc and parts will be loaded on demand. The current design is that all memory allocated for the grammars should be from memory-mapped files. In this way the only limit for the grammar size will be the size of the virtual memory, i.e. 2^64 bytes. The swap file is completely circumvented, while all of the available RAM can be used as a cache for loading parts of the grammar.

• We want to be able to update grammars dynamically. This is a highly desired feature since recompiling large grammars takes hours. Instead, dynamic updates should happen instantly.

• We want to be able to store additional information in the PGF. For example that could be application specific semantic data. Another example is to store the source code of the different grammar rules, to allow the compiler to recompile individual rules.

• We want to allow a single file to contain slightly different versions of the grammar. This will be a kind of a version control system, which will allow different users to store their own grammar extensions while still using the same core content.

• We want to avoid the exponential explosion in the size of PMCFG for some grammars. This happens because PMCFG as a formalism is too low-level. By enriching it with light-weight variables, we can make it more powerful and hopefully avoid the exponential explosion.

• We want to finally ditch the old Haskell runtime which has long outlived its time.

There are also two bugs in the old C runtime whose fixes will require a lot of changes, so instead of fixing the old runtime we do it here:

• Integer literals in the C runtime are implemented as 32-bit integers, while the Haskell runtime used unlimited integers. Python supports unlimited integers too, so it would be nice to support them in the new runtime as well.

• The old C runtime assumed that String literals are terminated with the NULL character. None of the modern languages (Haskell, Python, Java, etc) make that assumption, so we should drop it too.

Consequences

The desired features will have the following implementation cosequences.

• The switch from memory-based to disc-based runtime requires one big change. Before it was easy to just keep a pointer from one object to another. Unfortunately this doesn't work with memory-mapped files, since every time when you map a file into memory it may end up at a different virtual address. Instead we must use file offsets. In order to make programming simpler, the new runtime will be implemented in C++ instead of C. This allows us to overload the arrow operator (->) which will dynamically convert file offsets to in-memory pointers.

• The choice of C++ also allows us to ditch the old libgu library and use STL instead.

• The content of the memory mapped files is platform-specific. For that reason there will be two grammar representations:

  • Native Grammar Format (.ngf) - which will be instantly loadable by just mapping it to memory, but will be platform-dependent.
  • Portable Grammar Format (.pgf) - which will take longer to load but will be more compact and platform independent. The runtime will be able to load .pgf files and convert them to .ngf. Conversely .pgf can be exported from the current .ngf.