forked from GitHub/gf-core
240 lines
6.4 KiB
Markdown
240 lines
6.4 KiB
Markdown
# LPGF testsuite & benchmark
|
|
|
|
## Test
|
|
|
|
LPGF must be equivalent to PGF in terms of linearisation output.
|
|
|
|
Possible exceptions:
|
|
- No handling of variants (design choice)
|
|
- Rendering of missing fucntions
|
|
|
|
### Running
|
|
|
|
```
|
|
stack build --test --bench --no-run-tests --no-run-benchmarks
|
|
stack test gf:test:lpgf # all LPGF tests
|
|
stack test gf:test:lpgf --test-arguments="unittests/Params" # specific grammar
|
|
stack test gf:test:lpgf --test-arguments="foods/Foods Fre Ger" # specific grammar and languages
|
|
```
|
|
|
|
```
|
|
stack build --test --bench --no-run-tests --no-run-benchmarks && DEBUG=1 stack test gf:test:lpgf --test-arguments="foods/Foods Fre Ger"
|
|
stack build --test --bench --no-run-tests --no-run-benchmarks && DEBUG=1 stack test gf:test:lpgf --test-arguments="phrasebook/Phrasebook Bul"
|
|
```
|
|
|
|
Set environment variable `DEBUG=1` to enable dumping of intermediate formats.
|
|
|
|
## Benchmark
|
|
|
|
Compare performance metrics between LPGF and PGF[2]. Note: correctness is not checked here.
|
|
|
|
### Compilation
|
|
|
|
Comparing PGF, LPGF along following criteria:
|
|
|
|
- Time
|
|
- Memory
|
|
- Binary file size
|
|
|
|
### Runtime (linearisation)
|
|
|
|
Comparing PGF, PGF2, LPGF along following criteria:
|
|
|
|
- Time
|
|
- Memory
|
|
|
|
### Running
|
|
|
|
Run each command separately so that memory measurements are isolated.
|
|
The `+RTS -T -RTS` is so that GHC can report its own memory usage.
|
|
|
|
```
|
|
stack build --test --bench --no-run-tests --no-run-benchmarks &&
|
|
stack bench --benchmark-arguments "compile pgf testsuite/lpgf/foods/Foods*.gf +RTS -T -RTS" &&
|
|
stack bench --benchmark-arguments "compile lpgf testsuite/lpgf/foods/Foods*.gf +RTS -T -RTS" &&
|
|
stack bench --benchmark-arguments "run pgf Foods.pgf testsuite/lpgf/foods/Foods-all.trees +RTS -T -RTS" &&
|
|
stack bench --benchmark-arguments "run pgf2 Foods.pgf testsuite/lpgf/foods/Foods-all.trees +RTS -T -RTS" &&
|
|
stack bench --benchmark-arguments "run lpgf Foods.lpgf testsuite/lpgf/foods/Foods-all.trees +RTS -T -RTS"
|
|
```
|
|
|
|
```
|
|
stack build --test --bench --no-run-tests --no-run-benchmarks &&
|
|
stack bench --benchmark-arguments "compile pgf testsuite/lpgf/phrasebook/Phrasebook*.gf +RTS -T -RTS" &&
|
|
stack bench --benchmark-arguments "compile lpgf testsuite/lpgf/phrasebook/Phrasebook*.gf +RTS -T -RTS" &&
|
|
stack bench --benchmark-arguments "run pgf Phrasebook.pgf testsuite/lpgf/phrasebook/Phrasebook-10000.trees +RTS -T -RTS" &&
|
|
stack bench --benchmark-arguments "run pgf2 Phrasebook.pgf testsuite/lpgf/phrasebook/Phrasebook-10000.trees +RTS -T -RTS" &&
|
|
stack bench --benchmark-arguments "run lpgf Phrasebook.lpgf testsuite/lpgf/phrasebook/Phrasebook-10000.trees +RTS -T -RTS"
|
|
```
|
|
|
|
## Profiling
|
|
|
|
```
|
|
stack bench --work-dir .stack-work-profile --profile --benchmark-arguments "compile lpgf testsuite/lpgf/phrasebook/PhrasebookFre.gf +RTS -T -p -h -RTS"
|
|
```
|
|
|
|
Produced files:
|
|
- `lpgf-bench.prof` - total time and memory allocation (`-p`)
|
|
- `lpgf-bench.hp` - heap profile (`-h`)
|
|
|
|
```
|
|
stack exec -- hp2ps -c lpgf-bench.hp && open lpgf-bench.ps
|
|
```
|
|
|
|
**Resources**
|
|
|
|
- https://downloads.haskell.org/ghc/8.6.5/docs/html/users_guide/profiling.html
|
|
- http://book.realworldhaskell.org/read/profiling-and-optimization.html
|
|
- https://wiki.haskell.org/Performance
|
|
|
|
# Notes on compilation
|
|
|
|
## 1 (see unittests/Params4)
|
|
|
|
**param defns**
|
|
P = P1 | P2
|
|
Q = Q1 | Q2
|
|
R = RP P | RPQ P Q | R0
|
|
X = XPQ P Q
|
|
|
|
**translation**
|
|
NB: tuples may be nested, but will be concatted at runtime
|
|
|
|
P1 = <1>
|
|
P2 = <2>
|
|
|
|
Q1 = <1>
|
|
Q2 = <2>
|
|
|
|
R P1 = <1,1>
|
|
R P2 = <1,2>
|
|
RPQ P1 Q1 = <2,1,1>
|
|
RPQ P1 Q2 = <2,1,2>
|
|
RPQ P2 Q1 = <2,2,1>
|
|
RPQ P2 Q2 = <2,2,2>
|
|
R0 = <3>
|
|
|
|
XPQ P1 Q1 = <1,1,1>
|
|
XPQ P1 Q2 = <1,1,2>
|
|
XPQ P2 Q1 = <1,2,1>
|
|
XPQ P2 Q2 = <1,2,2>
|
|
|
|
P => Str
|
|
<"P1","P2">
|
|
|
|
{p:P ; q:Q} => Str
|
|
<<"P1;Q1","P1;Q2">,<"P2;Q1","P2;Q2">>
|
|
|
|
{p=P2; q=Q1}
|
|
<<2>,<1>>
|
|
|
|
R => Str
|
|
< <"RP P1","RP P2">,
|
|
< <"RPQ P1 Q1","RPQ P1 Q2">,
|
|
<"RPQ P2 Q1","RPQ P2 Q2"> >,
|
|
"R0"
|
|
>
|
|
|
|
X => Str
|
|
<<<"XPQ P1 Q1","XPQ P1 Q2">,
|
|
<"XPQ P2 Q1","XPQ P2 Q2">>>
|
|
|
|
{p=P2 ; r=R0}
|
|
<<2>,<3>>
|
|
|
|
{p=P2 ; r1=RP P1 ; r2=RPQ P1 Q2 ; r3=R0 }
|
|
< <2> , <1, 1> , <2, 1, 2> , <3>>
|
|
|
|
## 2 (see unittests/Params5)
|
|
|
|
**param defns**
|
|
|
|
P = P1 | PQ Q
|
|
Q = Q1 | QR R
|
|
R = R1 | R2
|
|
|
|
**translation**
|
|
|
|
P1 = <1>
|
|
PQ Q1 = <2,1>
|
|
PQ QR R1 = <2,2,1>
|
|
PQ QR R2 = <2,2,2>
|
|
|
|
Q1 = <1>
|
|
QR R1 = <2,1>
|
|
QR R2 = <2,2>
|
|
|
|
R1 = <1>
|
|
R2 = <2>
|
|
|
|
P => Str
|
|
<"P1",<"PQ Q1",<"PQ (QR R1)","PQ (QR R2)">>>
|
|
|
|
{q:Q ; p:P} => Str
|
|
< <"Q1;P1",<"Q1;PQ Q1",<"Q1;PQ (QR R1)","Q1;PQ (QR R2)">>>,
|
|
<
|
|
<"QR R1;P1",<"QR R1;PQ Q1",<"QR R1;PQ (QR R1)","QR R1;PQ (QR R2)">>>,
|
|
<"QR R2;P1",<"QR R2;PQ Q1",<"QR R2;PQ (QR R1)","QR R2;PQ (QR R2)">>>
|
|
>
|
|
>
|
|
|
|
{q=Q1 ; p=P1} = <<1>,<1>>
|
|
{q=Q1 ; p=PQ Q1} = <<1>,<2,1>>
|
|
{q=Q1 ; p=PQ (QR R1)} = <<1>,<2,2,1>>
|
|
{q=Q1 ; p=PQ (QR R2)} = <<1>,<2,2,2>>
|
|
|
|
{q=QR R1 ; p=P1} = <<2,1>,<1>>
|
|
{q=QR R1 ; p=PQ Q1} = <<2,1>,<2,1>>
|
|
{q=QR R1 ; p=PQ (QR R1)} = <<2,1>,<2,2,1>>
|
|
{q=QR R1 ; p=PQ (QR R2)} = <<2,1>,<2,2,2>>
|
|
|
|
{q=QR R2 ; p=P1} = <<2,2>,<1>>
|
|
{q=QR R2 ; p=PQ Q1} = <<2,2>,<2,1>>
|
|
{q=QR R2 ; p=PQ (QR R1)} = <<2,2>,<2,2,1>>
|
|
{q=QR R2 ; p=PQ (QR R2)} = <<2,2>,<2,2,2>>
|
|
|
|
**NOTE**: GF will swap q and p in record, as part of record field sorting, resulting in the following:
|
|
|
|
{p:P ; q:Q} => Str
|
|
< <"P1;Q1", <"P1;QR R1","P1;QR R2">>,
|
|
< <"PQ Q1;Q1", <"PQ Q1;QR R1","PQ Q1;QR R2">>,
|
|
< <"PQ (QR R1);Q1", <"PQ (QR R1);QR R1","PQ (QR R1);QR R2">>,
|
|
<"PQ (QR R2);Q1", <"PQ (QR R2);QR R1","PQ (QR R2);QR R2">>
|
|
>
|
|
>
|
|
>
|
|
|
|
{p=P1 ; q=Q1} = <<1>,<1>>
|
|
{p=P1 ; q=QR R1} = <<1>,<2,1>>
|
|
{p=P1 ; q=QR R2} = <<1>,<2,2>>
|
|
|
|
{p=PQ Q1 ; q=Q1} = <<2,1>,<1>>
|
|
{p=PQ Q1 ; q=QR R1} = <<2,1>,<2,1>>
|
|
{p=PQ Q1 ; q=QR R2} = <<2,1>,<2,2>>
|
|
|
|
{p=PQ (QR R1) ; q=Q1} = <<2,2,1>,<1>>
|
|
{p=PQ (QR R1) ; q=QR R1} = <<2,2,1>,<2,1>>
|
|
{p=PQ (QR R1) ; q=QR R2} = <<2,2,1>,<2,2>>
|
|
|
|
{p=PQ (QR R2) ; q=Q1} = <<2,2,2>,<1>>
|
|
{p=PQ (QR R2) ; q=QR R1} = <<2,2,2>,<2,1>>
|
|
{p=PQ (QR R2) ; q=QR R2} = <<2,2,2>,<2,2>>
|
|
|
|
|
|
{pp: {p:P} ; q:Q} => Str
|
|
|
|
{pp={p=P1} ; q=Q1} = <<<1>>,<1>>
|
|
{pp={p=P1} ; q=QR R1} = <<<1>>,<2,1>>
|
|
{pp={p=P1} ; q=QR R2} = <<<1>>,<2,2>>
|
|
|
|
{pp={p=PQ Q1} ; q=Q1} = <<<2,1>>, <1>>
|
|
{pp={p=PQ Q1} ; q=QR R1} = <<<2,1>>, <2,1>>
|
|
{pp={p=PQ Q1} ; q=QR R2} = <<<2,1>>, <2,2>>
|
|
|
|
{pp={p=PQ (QR R1)} ; q=Q1} = <<<2,2,1>>,<1>>
|
|
{pp={p=PQ (QR R1)} ; q=QR R1} = <<<2,2,1>>,<2,1>>
|
|
{pp={p=PQ (QR R1)} ; q=QR R2} = <<<2,2,1>>,<2,2>>
|
|
|
|
{pp={p=PQ (QR R2)} ; q=Q1} = <<<2,2,2>>,<1>>
|
|
{pp={p=PQ (QR R2)} ; q=QR R1} = <<<2,2,2>>,<2,1>>
|
|
{pp={p=PQ (QR R2)} ; q=QR R2} = <<<2,2,2>>,<2,2>>
|