Learn from LL(1) to PEG parser the hard way

Curious about CPython's new PEG parser but not a compiler expert? This talk is for you. It covers the fundamentals, from traditional parsing to PEG, based on the speaker's own learning.

Transcript

1. Standing on the shoulders of giants: Learn from LL(1) to PEG parser the hard way Kir Chou @ PyCon TW 2021 1
2. https://ep2021.europython.eu/talks/98ecgon-learn-from-ll1-to-peg-parser-the-hard-way
3. Standing on the shoulders of giants: Learn from LL(1) to PEG parser the hard way Kir Chou @ PyCon TW 2021 3
4. 4 https://www.youtube.com/watch?v=DZTLgVBxET4
5. Standing on the shoulders of giants: Learn from LL(1) to PEG parser the hard way Kir Chou @ PyCon TW 2021 5
6. About me Presented at PyCon since 2017 https://note35.github.io/about/ https://github.com/note35/Parser-Learning 6
7. Agenda • Motivation • What is parser in CPython? • Parser 101 - CFG • Parser 101 - Traditional parser (LL(1) / LR(0)) • Parser 102 - PEG and PEG parser • Parser 102 - Packrat parser • CPython’s PEG parser • Take away 7
8. Motivation 8
9. Motivation What’s New In Python 3.9? PEP 617, CPython now uses a new parser based on PEG; “IIRC, I took a Compiler class in school…” 9
10. Motivation (Cont.) School taught us the brief concept of the Compiler’s frontend and backend. School’s parser assignment used Bison + YACC. And... 10
11. My motivation = Talk objectives What is PEG parser? Why did python use LL(1) parser before? Why did Guido choose PEG parser? What other parsers do we have? What’s the difference between those parsers? How to implement those parsers? 11
12. What is parser in CPython? CPython DevGuide - Design of CPython’s Compiler 12
13. Compilation Steps 13 Source Code Tokens Abstract Syntax Tree (AST) Bytecode Result Lexer Parser Compiler VM Import
14. 14 https://docs.python.org/3/library/tokenize.html#examples Lexer
15. 15 https://docs.python.org/3/library/ast.html Parser
16. 16 https://docs.python.org/3/library/dis.html#dis.disassemble Compiler = print(2*3+4)
17. 17 Source Code Tokens Abstract Syntax Tree (AST) Bytecode Result Lexer Parser Compiler VM Import Talk’s focus!
18. Parser 101 - CFG Uncode - GATE Computer Science - Compiler Design Lecture 18
19. Grammar Context Free Grammar (CFG) 19 Interpretation of this Grammar “Both B and a can be derived from A” Derivation *some paper write <- Non-terminal AND *support ambigious syntax A -> B | a Terminal rule
20. What is “Context Free”? Left-hand side in all the rules only contains 1 non-terminal. Valid CFG Example: Invalid CFG Example: 20 S -> aSb xSy -> axSyb
21. Semantic Analysis: Parse Tree Concret Syntax Tree (CST) An ordered, rooted tree that represents the syntactic structure of a string according to some context-free grammar. Abstract Syntax Tree (AST) A tree representation of the abstract syntactic structure of source code written in a programming language. 21
22. CFG Simplification 1. Ambiguous -> Unambiguous 2. Nondeterministic -> Deterministic 3. Left recursion -> No left recursion 22
23. Ambiguious Definition A grammar contains rules that can generate more than one tree. 23 E -> E + E | E * E | Num N N N E E E + E * E N E E N N E E E * +
24. Ambiguious -> Unambiguous 24 N E E N N T F T * + E -> E + T | T T -> T * F | F F -> Num E -> E + E | E * E | Num Step1 Rewrite Grammar Step2 Make sure the grammar only generate one tree T F F
25. Non-deterministic -> Deterministic A grammar contains rules that have common prefix. 25 A -> ab | ac A -> aA’ A’ -> b | c Rewrite Grammar *A non-deterministic grammar can be rewritten into more than one deterministic grammar.
26. Left recursion -> No left recursion A grammar contains direct or indirect left recursion. 26 E -> E + T | T T -> T * F | F F -> Num E -> TE’ E’ -> +TE’ | None T -> FT’ T’ -> *FT’ | None F -> Num Rewrite Grammar E in first E + T will recursively derives to second E + T, E in second E + T will repeat it to third E + T, and so on recursively.
27. Recap: CFG Simplification 27 Before After Ambiguous Non-deterministic Left Recursion
28. Parser 101 - Traditional parser Uncode - GATE Computer Science - Compiler Design Lecture 28
29. Parser classification 29 N E E N N E E E * + Top-down Type Bottom-up Type N E N E N E + N E E N E E N E E + N E E N N E E E * +
30. LL / LR Parser LL(k) = Left-to-right, Leftmost derivation, k-token lookahead (k>0) LR(k) = Left-to-right, Rightmost derivation, k-token lookahead (k>=0) 30 *Both LL/LR parser scan input string from left to right Input String: 2 + 3 * 4
31. LL / LR Parser LL(k) = Left-to-right, Leftmost derivation, k-token lookahead (k>0) LR(k) = Left-to-right, Rightmost derivation, k-token lookahead (k>=0) 31 *The derivation time of LL/LR parser is different. N E E N N E E E * + N E E N N E E E * + + → * * → +
32. LL / LR Parser LL(k) = Left-to-right, Leftmost derivation, k-token lookahead (k>0) LR(k) = Left-to-right, Rightmost derivation, k-token lookahead (k>=0) 32 Input String: 2 + 3 * 4 I am "a token of number". If I perform 1-token lookahead and meet "a token of +", what to do next?
33. Top Down - Recursive descent parser 33
34. LL(k) - Implementation 34 2 + 3 * 4 parse_E() E -> TE’ E’ -> +TE’ | None T -> FT’ T’ -> *FT’ | None F -> Num parse_Tp(parse_F()) parse_Ep( ) Step3 *recursively parse the input string started from first rule parse_E() Step2 *parse from left to right *perform k-lookahead parse_T() Step1 write function for each non-terminal
35. 35 Grammar E -> TE’ E’ -> +TE’ | None T -> FT’ T’ -> *FT’ | None F -> Num *perform 1-lookahead LL(1) - Example code Derivation x x
36. Top Down - Non recursive descent parser 36
37. LL(1) - Parsing table 37 Step1 Build first/follow table for each non-terminal Note: $ means endmark Step2 Build parsing table based on first/follow table
38. LL(1) - Implementation 38 Step3 Implement with stack (take shift/reduce action based on parsing table) N E E N N E E E * +
39. LL(1) - Example code 39 Grammar E -> TE’ E’ -> +TE’ | None T -> FT’ T’ -> *FT’ | None F -> Num Non-terminal stack Reduce (Derivation) Shift Reduce (Derivation)
40. Bottom Up - LR(0) parser 40
41. LR(0) - Deterministic finite automaton 41 E’ -> .E --- (1) E -> .E + T --- (2) E -> .T --- (3) T -> .T * Num --- (4) T -> .Num --- (5) Step1 Build Deterministic Finite Automaton(DFA) E’ -> E. E -> E. + T E -> T. T -> T. * Num T -> Num. E -> E + .T T -> .T * Num T -> .Num T -> T * .Num E -> E + T. T -> T. * Num T -> T * Num. E T Num * + T Num * Num S1 S2 S3 S4 S5 S6 S7 S8 Left recursion support
42. LR(0) - Parsing table 42 Step2 Build parsing table (For parser like SLR(1), it requires first/follow table) Shift acc Reduce (Derivation) acc
43. LR(0) - Implementation 43 Step3 Implement with stack (take shift/reduce action based on parsing table) N E E N N E E E * +
44. LR(0) - Example code 44 Grammar E -> E + T | T T -> T * F | F F -> Num Shift Reduce (Derivation)
45. Parser 102 - PEG and PEG parser 45
46. Grammar Parsing Expression Grammar (PEG) 46 *Difference from traditional CFG A will try A -> B first. Only after it fails at A -> B, A will only try A -> a. Derivation *some paper write <- Non-Terminal OR (if / elif / ...) *disallow ambigious syntax A -> B | a Terminal *Introduced in 2002 (Packrat Parsing: Simple, Powerful, Lazy, Linear Time) rule *support Regular Expression (EBNF grammar) in another paper
47. Example of difference 47 Grammar1: A -> a b | a Grammar2: A -> a | a b • LL/LR parser will fail to complete when the input grammar is ambiguous. • PEG parser only tries the first PEG rule. The latter rule will never succeed. “A PEG parser generator will resolve unintended ambiguities earliest-match-first, which may be arbitrary and lead to surprising parses.” (source)
48. PEG Parser PEG parser means “parser generated based on PEG”. PEG parser can be a Packrat parser, or other traditional parser with k-lookahead limitation. Mostly, PEG parser means Packrat parser. 48 CFG EBNF grammar PEG Packrat parser Traditional parser PEG Parser
49. Parser 102 - Packrat parser 49
50. Type of Packrat parser 50 Top-down Type N E E N E E N E E + N E E N N E E E * + Packrat parser is top-down type.
51. Packrat Parsing - Implementation 51 2 + 3 * 4 parse_E() E -> E + T | T T -> T * F | F F -> Num parse_T() and parse_F() parse_E() and parse_T() Step2 *parse from left to right *perform infinite lookahead + memoization Step1 *write function for each non-terminal (PEG rule) *Idea of memoization was Introduced in 1970 Step3 *recursively parse the input string started from first rule parse_E() Left recursion support
52. Packrat Parsing - Example code 52 Grammar E -> E + T | T T -> T * F | F F -> Num Derivation Memoization
53. Packrat - what is memoization? 53 509. Fibonacci Number 4 3 2 2 1 fib(0) = 0 fib(1) = 1 fib(2) = fib(1) + fib(0) = 1 fib(3) = fib(2) + fib(1) = fib(1) + fib(0) + fib(1) = 2 ... 1 0 1 0 if n = 4, we calculate fib(2), fib(0) twice, fib(1) thrice, fib(4), fib(3) once TIme Complexity: O(2^n)
54. Packrat - what is memoization? (Cont.) 54 509. Fibonacci Number if n = 4, we… calculate fib(4), fib(3), fib(2), fib(1), fib(0) once Time Complexity: O(2^n) => O(n) Space Complexity: O(1) => O(n)
55. Left recursion in Packrat parser 55 Approach 1 if (count of operator) < (count function call): return False Approach 2 reverse the call stack (adopted in CPython!) Source: Guido's Medium (Left-recursive PEG Grammars)
56. 56 Normal Memoization
57. 57 Left-recursion Memoization *perform infinite-lookahead
58. Traditional parser V.S Packrat parser 58
59. Traditional parser vs Packrat parser 59 Packrat Traditional Scan Left-to-right (*Right-to-left memo) Left-to-right Left Recursion Support (*Not support in first paper) LL needs to rewrite the grammar Ambigious Disallowed (determinism) Allowed Space Complexity O(Code Size) (space consumption) O(Depth of Parse Tree) Worst Time Complexity Super linear time (statelessness) *Because of feature like typedef in C Expotenial time Capability Basically covers all traditional cases (infinite lookahead) No left-recursion/ambigious for LL Has k lookup limitations for both (e.g. dangling else) Red text: 3 highlighted characteristics of Packrat parser.
60. 60 Parenthesized context managers PEP 622/634/635/636 - Structural Pattern Matching New rule in Python 3.10 based on PEG
61. CPython’s PEG parser 61
62. CPython Parser - Before/After CPython3.8 and before use LL(1) parser written by Guido 30 years ago The parser requires steps to generate CST and convert CST to AST. CPython3.9 uses PEG (Packrat) parser (Infinite lookahead) PEG rule supports left-recursion No more CST to AST step - source CPython3.10 drops LL(1) parser support 62 This answers “Why PEG?”
63. CPython Parser - Workflow 63 Meta Grammar Tools/peg_generator/ pegen/metagrammar.gram Grammar Grammar/python.gram Token Grammar/Tokens my_parser.py my_parser.c pegen (PEG Parser) Tools/peg_generator/ *CPython contains a peg parser generator written in python3.8+ (because of warlus operator)
64. Input: Meta Grammar Example Syntax Directed Translation (SDT) 64 rule non-Terminal return type PEG rule divider PEG rule action (python code) Parser header (python code)
65. Output: Generated PEG Parser (Partial code) 65
66. Recap: Benefit / Performance Benefit Grammar is more flexible: from LL(1) to LL(∞) (infinite lookahead) Hardware supports Packrat’s memory consumption now Skip intermediate parse tree (CST) construction Performance Within 10% of LL(1) parser both in speed and memory consumption (PEP 617) 66
67. Take away 67
68. Recap • Parser 101 (Compiler class in school) ◦ CFG ◦ Traditional Parser ▪ Top-down: LL(1) ▪ Bottom-up: LR(0) • Parser 102 ◦ PEG ◦ Packrat Parser • CPython ◦ Parser in CPython ◦ CPython’s PEG parser 68
69. 69 Need Answer? note35/Parser-Learning You can implement traditional parser like LL(1) and LR(0) parser, and Packrat parser from scratch! Leetcode: 227. Basic Calculator II Q. How to verify my understanding? A. Get your hands dirty!
70. Q & A 70
71. Appendix 71
72. Related Articles Guido van Rossum PEG Parsing Series Overview Bryan Ford Packrat Parsing: Simple, Powerful, Lazy, Linear Time Parsing Expression Grammars: A Recognition-Based Syntactic Foundation 72
73. Related Talks Guido van Rossum @ North Bay Python 2019 Writing a PEG parser for fun and profit Pablo Galindo and Lysandros Nikolaou @ Podcast.__init__ The Journey To Replace Python's Parser And What It Means For The Future Emily Morehouse-Valcarcel @ PyCon 2018 The AST and Me Alex Gaynor @ PyCon 2013 So you want to write an interpreter? 73
74. Thanks for your listening! 74