GitHub/gf-core
1
0
Fork 1
mirror of https://github.com/GrammaticalFramework/gf-core.git synced 2026-04-22 19:22:50 -06:00
Code Issues Packages Projects Releases Wiki Activity

lexer documentation in help and tutorial updated

Browse Source
This commit is contained in:
aarne
2008-11-10 15:53:38 +00:00
parent 7e82e4a710
commit d9ff5aa48c
2 changed files with 12 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
doc/gf-tutorial.txt
View File

@@ -4277,24 +4277,12 @@ In linearization, we use a corresponding **unlexer**:
===Most common lexers and unlexers===
|| lexer | description ||
| ``words`` | (default) tokens are separated by spaces or newlines
| ``literals`` | like words, but integer and string literals recognized
| ``chars`` | each character is a token
| ``code`` | program code conventions (uses Haskell's lex)
| ``text`` | with conventions on punctuation and capital letters
| ``codelit`` | like code, but recognize literals (unknown words as strings)
| ``textlit`` | like text, but recognize literals (unknown words as strings)
|| unlexer | description ||
| ``unwords`` | (default) space-separated token list
| ``text`` | format as text: punctuation, capitals, paragraph <p>
| ``code`` | format as code (spacing, indentation)
| ``textlit`` | like text, but remove string literal quotes
| ``codelit`` | like code, but remove string literal quotes
| ``concat`` | remove all spaces
%TODO: update the names
|| lexer | unlexer | description ||
| ``chars`` | ``unchars`` | each character is a token
| ``lexcode`` | ``unlexcode`` | program code conventions (uses Haskell's lex)
| ``lexmixed`` | ``unlexmixed`` | like text, but between $ signs like code
| ``lextext`` | ``unlextext`` | with conventions on punctuation and capitals
| ``words`` | ``unwords`` | (default) tokens separated by space characters
%TODO: also on alphabet encodings - although somewhere else