-Lexing, Parsing, and Layouts
-The C-style languages of my previous experiences have all had quite trivial
-lexical analysis stages, peaking in complexity when I streamed tokens lazily in
-C. The task of tokenising a C-style language is very simple in description: you
-ignore all whitespace and point out what you recognise. If you don’t recognise
-something, check if it’s a literal or an identifier. Should it be neither,
-return an error.
-On paper, both lexing and parsing a Haskell-like language seem to pose a few
-greater challenges. Listed by ascending intimidation factor, some of the
-potential roadblocks on my mind before making an attempt were:
-
-Operators; Haskell has not only user-defined infix operators, but user-defined
-precedence levels and associativities. I recall using an algorithm that looked
-up infix, prefix, postfix, and even mixfix operators up in a global table to
-call their appropriate parser (if their precedence was appropriate, also
-stored in the table). I never modified the table at runtime, however this
-could be a very nice solution for Haskell.
Context-sensitive keywords; Haskell allows for some words to be used as identifiers in
-appropriate contexts, such as family, role, as.
-Reading a note found in GHC’s lexer,
-it appears that keywords are only considered in bodies for which their use is
-relevant, e.g. family and role in type declarations,
-as after case; if, then, and else in
-expressions, etc.
Whitespace sensitivity; While I was comfortable with the idea of a system
-similar to Python’s INDENT/DEDENT tokens, Haskell seemed to use whitespace to
-section code in a way that felt different.
-After a bit of thought and research, whitespace sensitivity in the form of
-layouts as Haskell and I will refer to them as, are easily the scariest thing
-on this list – however they are achievable!
-
-A Lexical Primer: Python
-We will compare and contrast with Python’s lexical analysis. Much to my dismay,
-Python uses newlines and indentation to separate statements and resolve scope
-instead of the traditional semicolons and braces found in C-style languages (we
-may generally refer to these C-style languages as explicitly-sectioned).
-Internally during tokenisation, when the Python lexer begins a new line, they
-compare the indentation of the new line with that of the previous and apply the
-following rules:
-
-If the new line has greater indentation than the previous, insert an INDENT
-token and push the new line’s indentation level onto the indentation stack
-(the stack is initialised with an indentation level of zero).
If the new line has lesser indentation than the previous, pop the stack until
-the top of the stack is greater than the new line’s indentation level. A
-DEDENT token is inserted for each level popped.
If the indentation is equal, insert a NEWLINE token to terminate the previous
-line, and leave it at that!
-Parsing Python with the INDENT, DEDENT, and NEWLINE tokens is identical to
-parsing a language with braces and semicolons. This is a solution pretty in line
-with Python’s philosophy of the “one correct answer” (TODO: this needs a
-source). In developing our layout rules, we will follow in the pattern of
-translating the whitespace-sensitive source language to an explicitly sectioned
-language.
-
-
-But What About Haskell?
-We saw that Python, the most notable example of an implicitly sectioned
-language, is pretty simple to lex. Why then am I so afraid of Haskell’s layouts?
-To be frank, I’m far less scared after asking myself this – however there are
-certainly some new complexities that Python needn’t concern. Haskell has
-implicit line continuation: forms written over multiple lines; indentation
-styles often seen in Haskell are somewhat esoteric compared to Python’s
-“s/[{};]//”.
--- line continuation
-something = this is a
- single expression
-
--- an extremely common style found in haskell
-data Python = Users
- { are :: Crying
- , right :: About
- , now :: Sorry
- }
-
--- another formatting oddity
--- note that this is not a single
--- continued line! `look at`,
--- `this`, and `alignment` are all
--- separate expressions!
-anotherThing = do look at
- this
- alignment
-
-
But enough fear, lets actually think about implementation. Firstly, some
-formality: what do we mean when we say layout? We will define layout as the
-rules we apply to an implicitly-sectioned language in order to yield one that is
-explicitly-sectioned. We will also define indentation of a lexeme as the column
-number of its first character.
-Thankfully for us, our entry point is quite clear; layouts only appear after a
-select few keywords, (with a minor exception; TODO: elaborate) being let
-(followed by supercombinators), where (followed by supercombinators),
-do (followed by expressions), and of (followed by alternatives)
-(TODO: all of these terms need linked glossary entries). In order to manage the
-cascade of layout contexts, our lexer will record a stack for which each element
-is either 
, denoting an explicit layout written with braces
-and semicolons, or a 
, denoting an implicitly laid-out
-layout where the start of each item belonging to the layout is indented
-
columns.
--- layout stack: []
-module M where -- layout stack: [∅]
-
-f x = let -- layout keyword; remember indentation of next token
- y = w * w -- layout stack: [∅, <10>]
- w = x + x
- -- layout ends here
- in do -- layout keyword; next token is a brace!
- { -- layout stack: [∅]
- print y;
- print x;
- }
-
-
Finally, we also need the concept of “virtual” brace tokens, which as far as
-we’re concerned at this moment are exactly like normal brace tokens, except
-implicitly inserted by the compiler. With the presented ideas in mind, we may
-begin to introduce a small set of informal rules describing the lexer’s handling
-of layouts, the first being:
-
-If a layout keyword is followed by the token ‘{’, push
-onto the layout context stack. Otherwise, push onto
-the layout context stack where is the indentation of the token
-following the layout keyword. Additionally, the lexer is to insert a virtual
-opening brace after the token representing the layout keyword.
-Consider the following observations from that previous code sample:
-
-Function definitions should belong to a layout, each of which may start at
-column 1.
A layout can enclose multiple bodies, as seen in the let-bindings and
-the do-expression.
Semicolons should terminate items, rather than separate them.
-Our current focus is the semicolons. In an implicit layout, items are on
-separate lines each aligned with the previous. A naïve implementation would be
-to insert the semicolon token when the EOL is reached, but this proves unideal
-when you consider the alignment requirement. In our implementation, our lexer
-will wait until the first token on a new line is reached, then compare
-indentation and insert a semicolon if appropriate. This comparison – the
-nondescript measurement of “more, less, or equal indentation” rather than a
-numeric value – is referred to as offside by myself internally and the
-Haskell report describing layouts. We informally formalise this rule as follows:
-
-When the first token on a line is preceeded only by whitespace, if the
-token’s first grapheme resides on a column number 
equal to the
-indentation level of the enclosing context – i.e. the 
on top of the layout stack. Should no such context exist on the
-stack, assume 
.
-We have an idea of how to begin layouts, delimit the enclosed items, and last
-we’ll need to end layouts. This is where the distinction between virtual and
-non-virtual brace tokens comes into play. The lexer needs only partial concern
-towards closing layouts; the complete responsibility is shared with the parser.
-This will be elaborated on in the next section. For now, we will be content with
-naïvely inserting a virtual closing brace when a token is indented right of the
-layout.
-
-Under the same conditions as rule 2., when 
the lexer shall
-insert a virtual closing brace and pop the layout stack.
-This rule covers some cases including the top-level, however, consider
-tokenising the in in a let-expression. If our lexical analysis
-framework only allows for lexing a single token at a time, we cannot return both
-a virtual right-brace and a in. Under this model, the lexer may simply
-pop the layout stack and return the in token. As we’ll see in the next
-section, as long as the lexer keeps track of its own context (i.e. the stack),
-the parser will cope just fine without the virtual end-brace.
-
-
-Parsing Lonely Braces
-When viewed in the abstract, parsing and tokenising are near-identical tasks yet
-the two are very often decomposed into discrete systems with very different
-implementations. Lexers operate on streams of text and tokens, while parsers
-are typically far less linear, using a parse stack or recursing top-down. A
-big reason for this separation is state management: the parser aims to be as
-context-free as possible, while the lexer tends to burden the necessary
-statefulness. Still, the nature of a stream-oriented lexer makes backtracking
-difficult and quite inelegant.
-However, simply declaring a parse error to be not an error at all
-counterintuitively proves to be an elegant solution our layout problem which
-minimises backtracking and state in both the lexer and the parser. Consider the
-following definitions found in rlp’s BNF:
-
-VOpen ::= vopen
-VClose ::= vclose | error
-
-A parse error is recovered and treated as a closing brace. Another point of note
-in the BNF is the difference between virtual and non-virtual braces (TODO: i
-don’t like that the BNF is formatted without newlines :/):
-
-LetExpr ::= let VOpen Bindings VClose in Expr | let `{` Bindings `}` in Expr
-
-This ensures that non-virtual braces are closed explicitly.
-This set of rules is adequete enough to satisfy our basic concerns about line
-continations and layout lists. For a more pedantic description of the layout
-system, see chapter 10 of the
-2010 Haskell Report, which I heavily referenced here.
-
-
-![\transrule
-{ a_0 : a_1 : \ldots : a_n : s
-& d
-& h
-\begin{bmatrix}
- a_0 : \mathtt{NSupercomb} \; [x_1,\ldots,x_n] \; e
-\end{bmatrix}
-& g
-}
-{ a_n : s
-& d
-& h'
-& g
-\\
-& \SetCell[c=3]{c}
-\text{where } h' = \mathtt{instantiateU} \; e \; a_n \; h \; g
-}](../_images/math/80185f417c32abbbd850f4429d27d4111dd55ce6.svg)
![\transrule
- {a : s & d & h
- \begin{bmatrix}
- a : \mathtt{NAp} \; a_1 \; a_2 \\
- a_2 : \mathtt{NInd} \; a_3
- \end{bmatrix} & g}
- {a : s & d & h[a : \mathtt{NAp} \; a_1 \; a_3] & g}](../_images/math/e91c8c9e0530c873480544a154b5156c1a3bc5cf.svg)
with internal 
-reduced form![\transrule
-{ a_0 : a_1 : \ldots : a_n : s
-& d
-& h
-\begin{bmatrix}
- a_0 : \mathtt{NSupercomb} \; [x_1,\ldots,x_n] \; e
-\end{bmatrix}
-& g
-}
-{ a_n : s
-& d
-& h'
-& g
-\\
-& \SetCell[c=3]{c}
-\text{where } h' = \mathtt{instantiateU} \; e \; a_n \; h \; g
-}](../_images/math/34e91bc0685dd54adb3e06c06e6fc0cc4d8ad419.svg)
associated with internal 
![\transrule
-{ c : a_1 : \ldots : a_n : \nillist
-& d
-& h
-\begin{bmatrix}
- c : \mathtt{NPrim} \; (\mathtt{ConP} \; t \; n) \\
- a_1 : \mathtt{NAp} \; c \; x_1 \\
- \vdots \\
- a_n : \mathtt{NAp} \; a_{n-1} \; x_n
-\end{bmatrix}
-& g
-}
-{ a_n : \nillist
-& d
-& h
-\begin{bmatrix}
- a_n : \mathtt{NData} \; t \; [x_1, \ldots, x_n]
-\end{bmatrix}
-& g
-}](../_images/math/a540630fbb5173bbf2de2d38326e10e6977b2c18.svg)
![\transrule
-{ c : a_1 : a_2 : s
-& d
-& h
-\begin{bmatrix}
- c : \mathtt{NPrim} \; \mathtt{CasePairP} \\
- p : \mathtt{NData} \; 0 \; [x,y] \\
- a_1 : \mathtt{NAp} \; c \; p \\
- a_2 : \mathtt{NAp} \; a_1 \; f
-\end{bmatrix}
-& g
-}
-{ a_1 : a_2 : s
-& d
-& h
-\begin{bmatrix}
- a_1 : \mathtt{NAp} \; f \; x \\
- a_2 : \mathtt{NAp} \; a_1 \; y
-\end{bmatrix}
-& g
-}](../_images/math/d13e748def2419af7ed6d97286f8f8ba95d695ab.svg)
![\transrule
- {a : s & d & h
- \begin{bmatrix}
- a : \mathtt{NAp} \; a_1 \; a_2 \\
- a_2 : \mathtt{NInd} \; a_3
- \end{bmatrix} & g}
- {a : s & d & h[a : \mathtt{NAp} \; a_1 \; a_3] & g}](../_images/math/538b0fab8c90c1abe8c6abaa3c807886040452db.svg)
with internal 
-reduced form![\transrule
-{ a_0 : a_1 : \ldots : a_n : s
-& d
-& h
-\begin{bmatrix}
- a_0 : \mathtt{NSupercomb} \; [x_1,\ldots,x_n] \; e
-\end{bmatrix}
-& g
-}
-{ a_n : s
-& d
-& h'
-& g
-\\
-& \SetCell[c=3]{c}
-\text{where } h' = \mathtt{instantiateU} \; e \; a_n \; h \; g
-}](../_images/math/5cf94fab747054130daaf436fc83ab590969c35f.svg)
associated with internal 
![\transrule
-{ c : a_1 : \ldots : a_n : \nillist
-& d
-& h
-\begin{bmatrix}
- c : \mathtt{NPrim} \; (\mathtt{ConP} \; t \; n) \\
- a_1 : \mathtt{NAp} \; c \; x_1 \\
- \vdots \\
- a_n : \mathtt{NAp} \; a_{n-1} \; x_n
-\end{bmatrix}
-& g
-}
-{ a_n : \nillist
-& d
-& h
-\begin{bmatrix}
- a_n : \mathtt{NData} \; t \; [x_1, \ldots, x_n]
-\end{bmatrix}
-& g
-}](../_images/math/00420770ada7197f2dc04dc2cb7be7957619046e.svg)
![\transrule
-{ c : a_1 : a_2 : s
-& d
-& h
-\begin{bmatrix}
- c : \mathtt{NPrim} \; \mathtt{CasePairP} \\
- p : \mathtt{NData} \; 0 \; [x,y] \\
- a_1 : \mathtt{NAp} \; c \; p \\
- a_2 : \mathtt{NAp} \; a_1 \; f
-\end{bmatrix}
-& g
-}
-{ a_1 : a_2 : s
-& d
-& h
-\begin{bmatrix}
- a_1 : \mathtt{NAp} \; f \; x \\
- a_2 : \mathtt{NAp} \; a_1 \; y
-\end{bmatrix}
-& g
-}](../_images/math/276f330b47308dae1686a2c0030783ed20a8866d.svg)
-reduction)
![\gmrule
-{ \mathtt{PushInt} \; n : i
-& s
-& h
-& m
-}
-{ i
-& a : s
-& h
-\begin{bmatrix}
- a : \mathtt{NNum} \; n
-\end{bmatrix}
-& m
-\begin{bmatrix}
- n' : a
-\end{bmatrix}
-\\
-\SetCell[c=5]{c}
-\text{where $n'$ is the base-10 string rep. of $n$}
-}](../_images/math/5a00c38ab6f040dfa75be56918953af44dc11637.svg)
![\transrule
-{ \mathtt{PushInt} \; n : i
-& s
-& h
-& m
-\begin{bmatrix}
- n' : a
-\end{bmatrix}
-}
-{ i
-& a : s
-& h
-& m
-\\
-\SetCell[c=5]{c}
-\text{where $n'$ is the base-10 string rep. of $n$}
-}](../_images/math/3fc332dc07916d83f75fcddf6da06a7734032ee9.svg)
![\gmrule
-{ \mathtt{Unwind} : \nillist
-& a_0 : \ldots : a_n : \nillist
-& \langle i, s \rangle : d
-& h
-\begin{bmatrix}
- a_0 : \mathtt{NGlobal} \; k \; c
-\end{bmatrix}
-& m
-}
-{ i
-& a_n : s
-& d
-& h
-& m \\
-\SetCell[c=2]{c}
-\text{when $n < k$}
-}](../_images/math/36a64918095793044bc603da838873e5db145350.svg)
-reduction)
.
.
![\gmrule
-{ \mathtt{Pack} \; t \; n : i
-& a_1 : \ldots : a_n : s
-& d
-& h
-& m
-}
-{ i
-& a : s
-& d
-& h
-\begin{bmatrix}
- a : \mathtt{NConstr} \; t \; [a_1,\ldots,a_n]
-\end{bmatrix}
- & m
-}](../_images/math/ceba2ed1f0be639cb0841ed19d1980c77657c74b.svg)
![\gmrule
-{ \mathtt{PushInt} \; n : i
-& s
-& d
-& h
-& m
-}
-{ i
-& a : s
-& d
-& h
-\begin{bmatrix}
- a : \mathtt{NNum} \; n
-\end{bmatrix}
-& m
-\begin{bmatrix}
- n' : a
-\end{bmatrix}
-\\
-\SetCell[c=5]{c}
-\text{where $n'$ is the base-10 string rep. of $n$}
-}](../_images/math/e8e88c3770a0c3cf4169abbee703c4a6bcffc0f5.svg)
![\gmrule
-{ \mathtt{PushInt} \; n : i
-& s
-& d
-& h
-& m
-\begin{bmatrix}
- n' : a
-\end{bmatrix}
-}
-{ i
-& a : s
-& d
-& h
-& m
-\\
-\SetCell[c=5]{c}
-\text{where $n'$ is the base-10 string rep. of $n$}
-}](../_images/math/321b47cc7986e581fdc26fb6b63195c1cf5409a2.svg)
![\transrule
- {a : s & d & h
- \begin{bmatrix}
- a : \mathtt{NAp} \; a_1 \; a_2 \\
- a_2 : \mathtt{NInd} \; a_3
- \end{bmatrix} & g}
- {a : s & d & h[a : \mathtt{NAp} \; a_1 \; a_3] & g}](../_images/math/ecdbb83ab452d27fcd79c1567929508177d87f1f.svg)
with internal 
associated with internal 
![\transrule
-{ c : a_1 : \ldots : a_n : \nillist
-& d
-& h
-\begin{bmatrix}
- c : \mathtt{NPrim} \; (\mathtt{ConP} \; t \; n) \\
- a_1 : \mathtt{NAp} \; c \; x_1 \\
- \vdots \\
- a_n : \mathtt{NAp} \; a_{n-1} \; x_n
-\end{bmatrix}
-& g
-}
-{ a_n : \nillist
-& d
-& h
-\begin{bmatrix}
- a_n : \mathtt{NData} \; t \; [x_1, \ldots, x_n]
-\end{bmatrix}
-& g
-}](../_images/math/8d27e16980b44cbe4694fb91353c71239780608d.svg)
![\transrule
-{ c : a_1 : a_2 : s
-& d
-& h
-\begin{bmatrix}
- c : \mathtt{NPrim} \; \mathtt{CasePairP} \\
- p : \mathtt{NData} \; 0 \; [x,y] \\
- a_1 : \mathtt{NAp} \; c \; p \\
- a_2 : \mathtt{NAp} \; a_1 \; f
-\end{bmatrix}
-& g
-}
-{ a_1 : a_2 : s
-& d
-& h
-\begin{bmatrix}
- a_1 : \mathtt{NAp} \; f \; x \\
- a_2 : \mathtt{NAp} \; a_1 \; y
-\end{bmatrix}
-& g
-}](../_images/math/34c5ba211557474d30a91f5ea6cf5692b3b88610.svg)
![\transrule{a : s & d & h \begin{bmatrix} a : \mathtt{NAp} \; a_1 \; a_2 \\ a_2 : \mathtt{NInd} \; a_3 \end{bmatrix} & f}
-{a : s & d & h[a : \mathtt{NAp} \; a_1 \; a_3] & f}](_images/math/95f01e99347ef7cc15166f53ec582de9b502f58b.svg)