gf-core/doc/gf-developers.t2t

GF Developers Guide

2018-07-26

%!options(html): --toc

== Before you start ==

This guide is intended for people who want to contribute to
the development of the GF compiler or the Resource Grammar Library. If
you are a GF user who just wants to download and install GF
(e.g to develop your own grammars), the simpler guide on
[the GF download page ../download/index.html] should be sufficient.

== Setting up your system for building GF ==

To build GF from source you need to install some tools on your
system: the Haskell Tool Stack, //Git// and the //Haskeline library//.

**On Linux** the best option is to install the tools via the standard
software distribution channels, i.e. by using the //Software Center//
in Ubuntu or the corresponding tool in other popular Linux distributions.

If the Haskell Tool Stack is already installed, enter the following command in a Terminal:

- On Ubuntu: ``sudo apt-get install git libghc6-haskeline-dev``
- On Fedora: ``sudo dnf install git ghc-haskeline-devel``


**On Mac OS and Windows**, the tools can be downloaded from their respective
web sites, as described below.

=== Git ===

To get the GF source code, you also need //Git//.
//Git// is a distributed version control system, see
https://git-scm.com/downloads for more information.

If you've entered the command above, it incudes git installation.

=== The haskeline library ===

GF uses //haskeline// to enable command line editing in the GF shell.
This should work automatically on Mac OS and Windows, but on Linux one
extra step is needed to make sure the C libraries (terminfo)
required by //haskeline// are installed.  Here is one way to do this:

- On Ubuntu: ``sudo apt-get install libghc-haskeline-dev``
- On Fedora: ``sudo dnf install ghc-haskeline-devel``


== Getting the source ==

Once you have all tools in place you can get the GF source code from
[GitHub https://github.com/GrammaticalFramework/gf-core].

=== Read-only access: clone the main repository ===

If you only want to compile and use GF, you can just clone the repository as follows:

```
  $ git clone https://github.com/GrammaticalFramework/gf-core.git
```

To get new updates, run the following anywhere in your local copy of the repository:

```
  $ git pull
```

=== Contribute your changes: create a fork ===

If you want the possibility to contribute your changes, you should
[create your own fork https://docs.github.com/en/get-started/quickstart/fork-a-repo]
of the repository, and then clone that.

```
  $ git clone https://github.com/<YOUR_USERNAME>/gf-core.git
```

**Updating your copy —**
Once you have cloned your fork, you need to set up the main GrammaticalFramework repository as a remote:

```
  $ git remote add upstream https://github.com/GrammaticalFramework/gf-core.git
```

Then you can get the latest updates by running the following:

```
  $ git pull upstream master
```

**Recording local changes —**
If you are new to Git, we recommend to read a tutorial on how to [record and push your changes https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository] to your fork.


**Pull request —**
TODO


%It is also possible for anyone else to contribute by

%- creating a fork of the GF repository on GitHub,
%- working with local clone of the fork (obtained with ``git clone``),
%- pushing changes to the fork,
%- and finally sending a pull request.


== Compilation from source with Stack ==

Assuming you have the Haskell Tool Stack, Git, and Haskeline installed, entering

```
$ stack install
```

into a Terminal will install GF and all necessary libraries, including Alex and Happy.



== Compilation from source with Cabal ==

The build system of GF is based on //Cabal//, which is part of the
Haskell Platform, so no extra steps are needed to install it. In the simplest
case, all you need to do to compile and install GF, after downloading the
source code as described above, is

```
$ cabal install
```

This will automatically download any additional Haskell libraries needed to
build GF. If this is the first time you use Cabal, you might need to run
``cabal update`` first, to update the list of available libraries.

If you want more control, the process can also be split up into the usual
//configure//, //build// and //install// steps.

=== Configure ===

During the configuration phase Cabal will check that you have all
necessary tools and libraries needed for GF. The configuration is
started by the command:

```
$ cabal configure
```

If you don't see any error message from the above command then you
have everything that is needed for GF. You can also add the option
``-v`` to see more details about the configuration.

You can use ``cabal configure --help`` to get a list of configuration options.

=== Build ===

The build phase does two things. First it builds the GF compiler from
the Haskell source code and after that it builds the GF Resource Grammar
Library using the already build compiler.  The simplest command is:

```
$ cabal build
```

Again you can add the option ``-v`` if you want to see more details.

==== Parallel builds ====

If you have Cabal>=1.20 you can enable parallel compilation by using

```
$ cabal build -j
```

or by putting a line
```
jobs: $ncpus
```
in your ``.cabal/config`` file. Cabal
will pass this option to GHC when building the GF compiler, if you
have GHC>=7.8.

Cabal also passes ``-j`` to GF to enable parallel compilation of the
Resource Grammar Library. This is done unconditionally to avoid
causing problems for developers with Cabal<1.20. You can disable this
by editing the last few lines in ``WebSetup.hs``.


==== Partial builds ====

**NOTE**: The following doesn't work with recent versions of ``cabal``.
%% // TH 2015-06-22

Sometimes you just want to work on the GF compiler and don't want to
recompile the resource library after each change. In this case use
this extended command:

```
$ cabal build rgl-none
```

The resource library could also be compiled in two modes: with present
tense only and with all tenses. By default it is compiled with all
tenses. If you want to use the library with only present tense you can
compile it in this special mode with the command:

```
$ cabal build present
```

You could also control which languages you want to be recompiled by
adding the option ``langs=list``. For example the following command
will compile only the English and the Swedish language:

```
$ cabal build langs=Eng,Swe
```

=== Install ===

After you have compiled GF you need to install the executable and libraries
to make the system usable.

```
$ cabal copy
$ cabal register
```

This command installs the GF compiler for a single user, in the standard
place used by Cabal.
On Linux and Mac this could be ``$HOME/.cabal/bin``.
On Mac it could also be ``$HOME/Library/Haskell/bin``.
On Windows this is ``C:\Program Files\Haskell\bin``.

The compiled GF Resource Grammar Library will be installed
under the same prefix, e.g. in
``$HOME/.cabal/share/gf-3.3.3/lib`` on Linux and
in ``C:\Program Files\Haskell\gf-3.3.3\lib`` on Windows.

If you want to install in some other place then use the ``--prefix``
option during the configuration phase.

=== Clean ===

Sometimes you want to clean up the compilation and start again from clean
sources. Use the clean command for this purpose:

```
$ cabal clean
```


%=== SDist ===
%
%You can use the command:
%
%% This does *NOT* include everything that is needed // TH 2012-08-06
%```
%$ cabal sdist
%```
%
%to prepare archive with all source codes needed to compile GF.

=== Known problems with Cabal ===

Some versions of Cabal (at least version 1.16) seem to have a bug that can
cause the following error:

```
Configuring gf-3.x...
setup: Distribution/Simple/PackageIndex.hs:124:8-13: Assertion failed
```

The exact cause of this problem is unclear, but it seems to happen
during the configure phase if the same version of GF is already installed,
so a workaround is to remove the existing installation with

```
ghc-pkg unregister gf
```

You can check with ``ghc-pkg list gf`` that it is gone.

== Compilation with make ==

If you feel more comfortable with Makefiles then there is a thin Makefile
wrapper arround Cabal for you. If you just type:
```
$ make
```
the configuration phase will be run automatically if needed and after that
the sources will be compiled.

%% cabal build rgl-none does not work with recent versions of Cabal
%If you don't want to compile the resource library
%every time then you can use:
%```
%$ make gf
%```

For installation use:
```
$ make install
```
For cleaning:
```
$ make clean
```
%and to build source distribution archive run:
%```
%$ make sdist
%```

== Compiling GF with C run-time system support ==

The C run-time system is a separate implementation of the PGF run-time services.
It makes it possible to work with very large, ambiguous grammars, using
probabilistic models to obtain probable parses. The C run-time system might
also be easier to use than the Haskell run-time system on certain platforms,
e.g. Android and iOS.

To install the C run-time system, go to the ``src/runtime/c`` directory
%and follow the instructions in the ``INSTALL`` file.
and use the ``install.sh`` script:
```
bash setup.sh configure
bash setup.sh build
bash setup.sh install
```
This will install
the C header files and libraries need to write C programs that use PGF grammars.
% Some example C programs are included in the ``utils`` subdirectory, e.g. ``pgf-translate.c``.

Depending on what you want to do with the C runtime, you can follow one or more of the following steps.

=== 1. Use the C runtime from another programming language ===

% **If you just want to use the C runtime from Python, Java, or Haskell, you don't need to change your GF installation.**

==== What ====

**This is the most common use case for the C runtime:** compile
your GF grammars into PGF with the standard GF executable,
and manipulate the PGFs from another programming language,
using the bindings to the C runtime.

==== How ====

The Python, Java and Haskell bindings are found in the
``src/runtime/{python,java,haskell-bind}`` directories,
respecively. Compile them by following the instructions
in the ``INSTALL`` or ``README`` files in those directories.

The Python library can also be installed from PyPI using ``pip install pgf``.
(If you are on Mac and get an error about ``clang`` version, you can try
some of [these solutions https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target]—but be careful before removing any existing installations.)


=== 2. Use GF shell with C runtime support ===

==== What ====
If you want to use the GF shell with C runtime functionalities, then you need to (re)compile GF with special flags.

The GF shell can be started with ``gf -cshell`` or ``gf -crun`` to use
the C run-time system instead of the Haskell run-time system.
Only limited functionality is available when running the shell in these
modes (use the ``help`` command in the shell for details).

(Re)compiling your GF with these flags will also give you
Haskell bindings to the C runtime, as a library called ``PGF2``,
but if you want Python or Java bindings, you need to do step 1.

% ``PGF2``: a module to import in Haskell programs, providing a binding to the C run-time system.

==== How ====

If you use cabal, run the following command:

```
cabal install -fc-runtime
```

from the top directory.

If you use stack, uncomment the following lines in the ``stack.yaml`` file:

```
flags:
   gf:
     c-runtime: true
extra-lib-dirs:
   - /usr/local/lib
```

First you will need to install the following libraries if not already installed:

```
apt-get install
```

and then run ``stack install``, also from the top directory.


=== 3. Use GF server mode with C runtime ===

==== What ====

With this feature, ``gf -server`` mode is extended with new requests to call the C run-time
system, e.g. ``c-parse``, ``c-linearize`` and ``c-translate``.

==== How ====

If you use cabal, run the following command:

```
cabal install -fc-runtime -fserver
```
from the top directory.

If you use stack, add the following lines in the
If you use stack, uncomment the following lines in the ``stack.yaml`` file:

```
flags:
   gf:
     c-runtime: true
     server: true
extra-lib-dirs:
   - /usr/local/lib
```

and then run ``stack install``, also from the top directory.



== Compilation of RGL ==

As of 2018-07-26, the RGL is distributed separately from the GF compiler and runtimes.

=== Simple ===
To install the RGL, you can use the following commands from within the ``gf-rgl`` repository:
```
$ make install
```
There is also ``make build``, ``make copy`` and ``make clean`` which do what you might expect.

=== Advanced ===
For advanced build options, call the Haskell build script directly:
```
$ runghc Setup.hs ...
```
For more details see the [README https://github.com/GrammaticalFramework/gf-rgl/blob/master/README.md].

=== Haskell-free ===
If you do not have Haskell installed, you can use the simple build script ``Setup.sh``
(or ``Setup.bat`` for Windows).


== Creating binary distribution packages ==

=== Creating .deb packages for Ubuntu ===

This was tested on Ubuntu 14.04 for the release of GF 3.6, and the
resulting ``.deb`` packages appears to work on Ubuntu 12.04, 13.10 and 14.04.
For the release of GF 3.7, we generated ``.deb`` packages on Ubuntu 15.04 and
tested them on Ubuntu 12.04 and 14.04.

Under Ubuntu, Haskell executables are statically linked against other Haskell
libraries, so the .deb packages are fairly self-contained.

==== Preparations ====

```
sudo apt-get install dpkg-dev debhelper
```

==== Creating the package ====

Make sure the ``debian/changelog`` starts with an entry that describes the
version you are building. Then run

```
make deb
```

If get error messages about missing dependencies
(e.g. ``autoconf``, ``automake``, ``libtool-bin``, ``python-dev``,
``java-sdk``, ``txt2tags``)
use ``apt-get intall`` to install them, then try again.


=== Creating OS X Installer packages ===

Run

```
make pkg
```

=== Creating binary tar distributions ===

Run

```
make bintar
```

=== Creating .rpm packages for Fedora ===

This is possible, but the procedure has not been automated.
It involves using the cabal-rpm tool,

```
sudo dnf install cabal-rpm
```

and following the Fedora guide
[How to create an RPM package http://fedoraproject.org/wiki/How_to_create_an_RPM_package].

Under Fedora, Haskell executables are dynamically linked against other Haskell
libraries, so ``.rpm`` packages for all Haskell libraries that GF depends on
are required. Most of them are already available in the Fedora distribution,
but a few of them might have to be built and distributed along with
the GF ``.rpm`` package.
When building ``.rpm`` packages for GF 3.4, we also had to build ``.rpm``s for
``fst`` and ``httpd-shed``.

== Running the test suite ==

The GF test suite is run with one of the following commands from the top directory:

```
  $ cabal test
```

or

```
  $ stack test
```

The testsuite architecture for GF is very simple but still very flexible.
GF by itself is an interpreter and could execute commands in batch mode.
This is everything that we need to organize a testsuite. The root of the
testsuite is the ``testsuite/`` directory. It contains subdirectories
which themselves contain GF batch files (with extension ``.gfs``).
The above command searches the subdirectories of the ``testsuite/`` directory
for files with extension ``.gfs`` and when it finds one, it is executed with
the GF interpreter. The output of the script is stored in file with extension ``.out``
and is compared with the content of the corresponding file with extension ``.gold``, if there is one.

Every time when you make some changes to GF that have to be tested,
instead of writing the commands by hand in the GF shell, add them to one ``.gfs``
file in the testsuite subdirectory where its ``.gf`` file resides and run the test.
In this way you can use the same test later and we will be sure that we will not
accidentally break your code later.

**Test Outcome - Passed:** If the contents of the files with the ``.out`` extension
are identical to their correspondingly-named files with the extension ``.gold``,
the command will report that the tests passed successfully, e.g.

```
 Running 1 test suites...
 Test suite gf-tests: RUNNING...
 Test suite gf-tests: PASS
 1 of 1 test suites (1 of 1 test cases) passed.
```

**Test Outcome - Failed:**  If there is a contents mismatch between the files
with the ``.out`` extension and their corresponding files with the extension ``.gold``,
the test diagnostics will show a fail and the areas that failed. e.g.

```
  testsuite/compiler/compute/Records.gfs: OK
  testsuite/compiler/compute/Variants.gfs: FAIL
  testsuite/compiler/params/params.gfs: OK
  Test suite gf-tests: FAIL
  0 of 1 test suites (0 of 1 test cases) passed.
```

The fail results overview is available in gf-tests.html which shows 4 columns:

+ //Results// - only areas that fail will appear. (Note: There are 3 failures in the gf-tests.html which are labelled as (expected). These failures should be ignored.)
+ //Input// - which is the test written in the .gfs file
+ //Gold// - the expected output from running the test set out in the .gfs file. This column refers to the contents from the .gold extension files.
+ //Output// - This column refers to the contents from the .out extension files which are generated as test output.
After fixing the areas which fail, rerun the test command. Repeat the entire process of fix-and-test until the test suite passes before  submitting a pull request to include your changes.