10 KiB
LPGF testsuite & benchmark
Testsuite
LPGF must be equivalent to PGF in terms of linearisation output.
Possible exceptions:
- No handling of variants (design choice)
- Rendering of missing functions
N.B. Phrasebook doesn't compile with RGL after 1131058b68c204a8d1312d2e2a610748eb8032cb
Running
Because Stack insists on rebuilding things all the time, I use separate .stack-work folders for testing and benchmarking.
Assumes treebank in same folder with same abstract name as grammar, e.g. unittests/Params.treebank
stack build --work-dir .stack-work-test --test --no-run-tests
stack test --work-dir .stack-work-test gf:test:lpgf # all LPGF tests
stack test --work-dir .stack-work-test gf:test:lpgf --test-arguments="unittests/Params" # specific grammar
stack test --work-dir .stack-work-test gf:test:lpgf --test-arguments="foods/Foods Fre Ger" # specific grammar and languages
stack test --work-dir .stack-work-test gf:test:lpgf --test-arguments="phrasebook/Phrasebook"
Set environment variable DEBUG=1 to enable dumping of intermediate formats into DEBUG/ folder.
Benchmark
Compare performance metrics between LPGF and PGF[2]. Note: correctness is not checked here.
Compilation
Comparing PGF, LPGF along following criteria:
- Time
- Memory
- Binary file size
Runtime (linearisation)
Comparing PGF, PGF2, LPGF along following criteria:
- Time
- Memory
Running
Run each command separately so that memory measurements are isolated.
The +RTS -T -RTS is so that GHC can report its own memory usage.
stack build --work-dir .stack-work-bench --bench --no-run-benchmarks &&
stack bench --work-dir .stack-work-bench --benchmark-arguments "compile pgf testsuite/lpgf/foods/Foods*.gf +RTS -T -RTS" &&
stack bench --work-dir .stack-work-bench --benchmark-arguments "compile lpgf testsuite/lpgf/foods/Foods*.gf +RTS -T -RTS" &&
stack bench --work-dir .stack-work-bench --benchmark-arguments "run pgf Foods.pgf testsuite/lpgf/foods/Foods-all.trees +RTS -T -RTS" &&
stack bench --work-dir .stack-work-bench --benchmark-arguments "run pgf2 Foods.pgf testsuite/lpgf/foods/Foods-all.trees +RTS -T -RTS" &&
stack bench --work-dir .stack-work-bench --benchmark-arguments "run lpgf Foods.lpgf testsuite/lpgf/foods/Foods-all.trees +RTS -T -RTS"
stack build --work-dir .stack-work-bench --bench --no-run-benchmarks &&
stack bench --work-dir .stack-work-bench --benchmark-arguments "compile pgf testsuite/lpgf/phrasebook/Phrasebook*.gf +RTS -T -RTS" &&
stack bench --work-dir .stack-work-bench --benchmark-arguments "compile lpgf testsuite/lpgf/phrasebook/Phrasebook*.gf +RTS -T -RTS" &&
stack bench --work-dir .stack-work-bench --benchmark-arguments "run pgf Phrasebook.pgf testsuite/lpgf/phrasebook/Phrasebook-10000.trees +RTS -T -RTS" &&
stack bench --work-dir .stack-work-bench --benchmark-arguments "run pgf2 Phrasebook.pgf testsuite/lpgf/phrasebook/Phrasebook-10000.trees +RTS -T -RTS" &&
stack bench --work-dir .stack-work-bench --benchmark-arguments "run lpgf Phrasebook.lpgf testsuite/lpgf/phrasebook/Phrasebook-10000.trees +RTS -T -RTS"
Profiling
stack build --work-dir .stack-work-profile --profile --bench --no-run-benchmarks &&
stack bench --work-dir .stack-work-profile --profile --benchmark-arguments "compile lpgf testsuite/lpgf/phrasebook/PhrasebookFre.gf +RTS -T -p -h -RTS"
Produced files:
lpgf-bench.prof- total time and memory allocation (-p)lpgf-bench.hp- heap profile (-h)
Open heap profile graph on-the-fly:
stack exec -- hp2ps -c lpgf-bench.hp && open lpgf-bench.ps
Convert and copy timestamped files into PROF/:
TS="$(date +%Y-%m-%d_%H%M)" &&
stack exec -- hp2ps -c lpgf-bench.hp &&
mv lpgf-bench.prof PROF/$TS.prof &&
mv lpgf-bench.ps PROF/$TS.ps &&
mv lpgf-bench.hs PROF/$TS.hp
Resources
- https://downloads.haskell.org/ghc/8.6.5/docs/html/users_guide/profiling.html
- http://book.realworldhaskell.org/read/profiling-and-optimization.html
- https://wiki.haskell.org/Performance
Honing in
stack build --test --bench --no-run-tests --no-run-benchmarks &&
stack bench --benchmark-arguments "compile lpgf testsuite/lpgf/phrasebook/PhrasebookFre.gf +RTS -T -RTS"
Baseline PGF
- compile: 1.600776s
- size: 2.88 MB Phrasebook.pgf Max memory: 328.20 MB
Baseline LPGF = B
- compile: 12.401099s
- size: 3.01 MB Phrasebook.lpgf Max memory: 1.33 GB
Baseline LPGF String instead of Text
- compile: 12.124689s
- size: 3.01 MB Phrasebook.lpgf Max memory: 1.34 GB
Baseline LPGF with impossible pruning
- compile: 7.406503s
- size: 3.01 MB Phrasebook.lpgf Max memory: 1.13 GB
B -extractStrings
- compile: 13.822735s
- size: 5.78 MB Phrasebook.lpgf Max memory: 1.39 GB
B -cleanupRecordFields
- compile: 13.670776s
- size: 3.01 MB Phrasebook.lpgf Max memory: 1.48 GB
No generation at all = E
- compile: 0.521001s
- size: 3.27 KB Phrasebook.lpgf Max memory: 230.69 MB
+ Concat, Literal, Error, Predef, Tuple, Variant, Commented
- compile: 1.503594s
- size: 3.27 KB Phrasebook.lpgf Max memory: 395.31 MB
+ Var, Pre, Selection
- compile: 1.260184s
- size: 3.28 KB Phrasebook.lpgf Max memory: 392.17 MB
+ Record
- compile: 1.659233s
- size: 7.07 KB Phrasebook.lpgf Max memory: 397.41 MB
+ Projection = X
- compile: 1.446217s
- size: 7.94 KB Phrasebook.lpgf Max memory: 423.62 MB
X + Param
- compile: 2.073838s
- size: 10.82 KB Phrasebook.lpgf Max memory: 619.71 MB
X + Table
- compile: 11.26558s
- size: 2.48 MB Phrasebook.lpgf Max memory: 1.15 GB
RawIdents
- compile: 5.393466s
- size: 3.01 MB Phrasebook.lpgf Max memory: 1.12 GB
Repeated terms in compilation
Param and Table
| Concr | Total | Unique | Perc |
|---|---|---|---|
| PhrasebookEng | 8673 | 1724 | 20% |
| PhrasebookSwe | 14802 | 2257 | 15% |
| PhrasebookFin | 526225 | 4866 | 1% |
Param
| Concr | Total | Unique | Perc |
|---|---|---|---|
| PhrasebookEng | 3211 | 78 | 2% |
| PhrasebookSwe | 7567 | 69 | 1% |
| PhrasebookFin | 316355 | 310 | 0.1% |
Table
| Concr | Total | Unique | Perc |
|---|---|---|---|
| PhrasebookEng | 5470 | 1654 | 30% |
| PhrasebookSwe | 7243 | 2196 | 30% |
| PhrasebookFin | 209878 | 4564 | 2% |
After impelementing state monad for table memoisation
worse!
- compile: 12.55848s
- size: 3.01 MB Phrasebook.lpgf Max memory: 2.25 GB
Params
| Concr | Total | Misses | Perc |
|---|---|---|---|
| PhrasebookEng | 3211 | 72 | 2% |
| PhrasebookSwe | 7526 | 61 | 1% |
| PhrasebookFin | 135268 | 333 | 0.2% |
| PhrasebookFre | 337102 | 76 | 0.02% |
Tables
| Concr | Total | Misses | Perc |
|---|---|---|---|
| PhrasebookEng | 3719 | 3170 | 85% |
| PhrasebookSwe | 4031 | 3019 | 75% |
| PhrasebookFin | 36875 | 21730 | 59% |
| PhrasebookFre | 41397 | 32967 | 80% |
Conclusions:
- map itself requires more memory than actual compilation
- lookup/insert is also as bad as actual compilation
Tried HashMap (deriving Hashable for LinValue), no inprovement. Using show on LinValue for keys is incredibly slow.
Notes on compilation
1 (see unittests/Params4)
param defns P = P1 | P2 Q = Q1 | Q2 R = RP P | RPQ P Q | R0 X = XPQ P Q
translation NB: tuples may be nested, but will be concatted at runtime
P1 = <1> P2 = <2>
Q1 = <1> Q2 = <2>
R P1 = <1,1> R P2 = <1,2> RPQ P1 Q1 = <2,1,1> RPQ P1 Q2 = <2,1,2> RPQ P2 Q1 = <2,2,1> RPQ P2 Q2 = <2,2,2> R0 = <3>
XPQ P1 Q1 = <1,1,1> XPQ P1 Q2 = <1,1,2> XPQ P2 Q1 = <1,2,1> XPQ P2 Q2 = <1,2,2>
P => Str <"P1","P2">
{p:P ; q:Q} => Str <<"P1;Q1","P1;Q2">,<"P2;Q1","P2;Q2">>
{p=P2; q=Q1} <<2>,<1>>
R => Str < <"RP P1","RP P2">, < <"RPQ P1 Q1","RPQ P1 Q2">, <"RPQ P2 Q1","RPQ P2 Q2"> >, "R0"
X => Str <<<"XPQ P1 Q1","XPQ P1 Q2">, <"XPQ P2 Q1","XPQ P2 Q2">>>
{p=P2 ; r=R0} <<2>,<3>>
{p=P2 ; r1=RP P1 ; r2=RPQ P1 Q2 ; r3=R0 } < <2> , <1, 1> , <2, 1, 2> , <3>>
2 (see unittests/Params5)
param defns
P = P1 | PQ Q Q = Q1 | QR R R = R1 | R2
translation
P1 = <1> PQ Q1 = <2,1> PQ QR R1 = <2,2,1> PQ QR R2 = <2,2,2>
Q1 = <1> QR R1 = <2,1> QR R2 = <2,2>
R1 = <1> R2 = <2>
P => Str <"P1",<"PQ Q1",<"PQ (QR R1)","PQ (QR R2)">>>
{q:Q ; p:P} => Str < <"Q1;P1",<"Q1;PQ Q1",<"Q1;PQ (QR R1)","Q1;PQ (QR R2)">>>, < <"QR R1;P1",<"QR R1;PQ Q1",<"QR R1;PQ (QR R1)","QR R1;PQ (QR R2)">>>, <"QR R2;P1",<"QR R2;PQ Q1",<"QR R2;PQ (QR R1)","QR R2;PQ (QR R2)">>>
{q=Q1 ; p=P1} = <<1>,<1>> {q=Q1 ; p=PQ Q1} = <<1>,<2,1>> {q=Q1 ; p=PQ (QR R1)} = <<1>,<2,2,1>> {q=Q1 ; p=PQ (QR R2)} = <<1>,<2,2,2>>
{q=QR R1 ; p=P1} = <<2,1>,<1>> {q=QR R1 ; p=PQ Q1} = <<2,1>,<2,1>> {q=QR R1 ; p=PQ (QR R1)} = <<2,1>,<2,2,1>> {q=QR R1 ; p=PQ (QR R2)} = <<2,1>,<2,2,2>>
{q=QR R2 ; p=P1} = <<2,2>,<1>> {q=QR R2 ; p=PQ Q1} = <<2,2>,<2,1>> {q=QR R2 ; p=PQ (QR R1)} = <<2,2>,<2,2,1>> {q=QR R2 ; p=PQ (QR R2)} = <<2,2>,<2,2,2>>
NOTE: GF will swap q and p in record, as part of record field sorting, resulting in the following:
{p:P ; q:Q} => Str < <"P1;Q1", <"P1;QR R1","P1;QR R2">>, < <"PQ Q1;Q1", <"PQ Q1;QR R1","PQ Q1;QR R2">>, < <"PQ (QR R1);Q1", <"PQ (QR R1);QR R1","PQ (QR R1);QR R2">>, <"PQ (QR R2);Q1", <"PQ (QR R2);QR R1","PQ (QR R2);QR R2">> >
{p=P1 ; q=Q1} = <<1>,<1>> {p=P1 ; q=QR R1} = <<1>,<2,1>> {p=P1 ; q=QR R2} = <<1>,<2,2>>
{p=PQ Q1 ; q=Q1} = <<2,1>,<1>> {p=PQ Q1 ; q=QR R1} = <<2,1>,<2,1>> {p=PQ Q1 ; q=QR R2} = <<2,1>,<2,2>>
{p=PQ (QR R1) ; q=Q1} = <<2,2,1>,<1>> {p=PQ (QR R1) ; q=QR R1} = <<2,2,1>,<2,1>> {p=PQ (QR R1) ; q=QR R2} = <<2,2,1>,<2,2>>
{p=PQ (QR R2) ; q=Q1} = <<2,2,2>,<1>> {p=PQ (QR R2) ; q=QR R1} = <<2,2,2>,<2,1>> {p=PQ (QR R2) ; q=QR R2} = <<2,2,2>,<2,2>>
{pp: {p:P} ; q:Q} => Str
{pp={p=P1} ; q=Q1} = <<<1>>,<1>> {pp={p=P1} ; q=QR R1} = <<<1>>,<2,1>> {pp={p=P1} ; q=QR R2} = <<<1>>,<2,2>>
{pp={p=PQ Q1} ; q=Q1} = <<<2,1>>, <1>> {pp={p=PQ Q1} ; q=QR R1} = <<<2,1>>, <2,1>> {pp={p=PQ Q1} ; q=QR R2} = <<<2,1>>, <2,2>>
{pp={p=PQ (QR R1)} ; q=Q1} = <<<2,2,1>>,<1>> {pp={p=PQ (QR R1)} ; q=QR R1} = <<<2,2,1>>,<2,1>> {pp={p=PQ (QR R1)} ; q=QR R2} = <<<2,2,1>>,<2,2>>
{pp={p=PQ (QR R2)} ; q=Q1} = <<<2,2,2>>,<1>> {pp={p=PQ (QR R2)} ; q=QR R1} = <<<2,2,2>>,<2,1>> {pp={p=PQ (QR R2)} ; q=QR R2} = <<<2,2,2>>,<2,2>>