Tweet
Tweet

Slide 1

Slide 1 text

Writing Compilers in Python (with PLY) Dave Beazley http://www.dabeaz.com October 12, 2006

Slide 2

Slide 2 text

Overview • Crash course on compilers • Lex/yacc • An introduction to PLY • Blood and guts (Rated R) • Various PLY features (more gore) • Examples

Slide 3

Slide 3 text

Disclaimer • Compilers is an advanced topic • Please stop me for questions!

Slide 4

Slide 4 text

Motivation • Writing a compiler is hard • Writing Python code seems to be easy • So why not write a compiler in Python?

Slide 5

Slide 5 text

Compilers 101 • What is a compiler? • A program that processes other programs • Typically implements a programming lang. • Examples: • gcc, javac, SWIG, Doxygen, Python # Some program print “Hello World” b = 3 + 4 * 5 for c in range(10): print c ??? Profit! compiler

Slide 6

Slide 6 text

Compiler Design • Compiler broken into stages • Lexing/parsing related to reading input • Type checking is error checking/validation • Code generation does something lexing parsing typecheck codegen in out

Slide 7

Slide 7 text

Example • Parse and generate code for the following: b = 40 + 20*(2+3)/37.5

Slide 8

Slide 8 text

Lexing • Splits input text into tokens • Makes sure the input uses right alphabet b = 40 + 20*(2+3)/37.5 NAME = NUM + NUM * ( NUM + NUM ) / FLOAT • Detects illegal symbols b = 40 * $5 Illegal Character

Slide 9

Slide 9 text

Parsing • Makes sure input is structurally correct b = 40 + 20*(2+3)/37.5 • Builds program structure (e.g., parse tree) NAME = NUM + NUM * ( NUM + NUM ) / FLOAT = NAME + NUM FLOAT / NUM * + NUM NUM

Slide 10

Slide 10 text

Parsing • Detects syntax errors b = 40 + “hello” (Syntax OK) b = 3 * 4 7 / (Syntax error) • If a program parses, it is at least well-formed • Still don’t know if program is correct b = 40 + “hello” (???)

Slide 11

Slide 11 text

Type checking • Enforces underlying semantics b = 40 + 20*(2+3)/37.5 (OK) c = 3 + “hello” (TYPE ERROR) d[4.5] = 4 (BAD INDEX) • Example: + operator + LHS RHS 1. LHS and RHS must be the same type 2. If different types, must be convertible to same type

Slide 12

Slide 12 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 + 40 / * 20 + 2 3 37.5

Slide 13

Slide 13 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 + / * 20 + 2 3 37.5 40 b = 40 + 20*(2+3)/37.5 LOAD R1, 40

Slide 14

Slide 14 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 + / * + 2 3 37.5 40 20 b = 40 + 20*(2+3)/37.5 LOAD R1, 40 LOAD R2, 20

Slide 15

Slide 15 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 + / * + 3 37.5 40 20 2 b = 40 + 20*(2+3)/37.5 LOAD R1, 40 LOAD R2, 20 LOAD R3, 2

Slide 16

Slide 16 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 + / * + 37.5 40 20 2 3 b = 40 + 20*(2+3)/37.5 LOAD R1, 40 LOAD R2, 20 LOAD R3, 2 LOAD R4, 3

Slide 17

Slide 17 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 + / * 37.5 40 20 2 3 + b = 40 + 20*(2+3)/37.5 LOAD R1, 40 LOAD R2, 20 LOAD R3, 2 LOAD R4, 3 ADD R3, R4, R3 ; R3 = (2+3)

Slide 18

Slide 18 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 + / 37.5 40 20 2 3 + * b = 40 + 20*(2+3)/37.5 LOAD R1, 40 LOAD R2, 20 LOAD R3, 2 LOAD R4, 3 ADD R3, R4, R3 ; R3 = (2+3) MUL R2, R3, R2 ; R2 = 20*(2+3)

Slide 19

Slide 19 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 + / 40 20 2 3 + * 37.5 b = 40 + 20*(2+3)/37.5 LOAD R1, 40 LOAD R2, 20 LOAD R3, 2 LOAD R4, 3 ADD R3, R4, R3 ; R3 = (2+3) MUL R2, R3, R2 ; R2 = 20*(2+3) LOAD R3, 37.5

Slide 20

Slide 20 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 + 40 20 2 3 + * 37.5 / b = 40 + 20*(2+3)/37.5 LOAD R1, 40 LOAD R2, 20 LOAD R3, 2 LOAD R4, 3 ADD R3, R4, R3 ; R3 = (2+3) MUL R2, R3, R2 ; R2 = 20*(2+3) LOAD R3, 37.5 DIV R2, R3, R2 ; R2 = 20*(2+3)/37.5

Slide 21

Slide 21 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 40 20 2 3 + * 37.5 / + b = 40 + 20*(2+3)/37.5 LOAD R1, 40 LOAD R2, 20 LOAD R3, 2 LOAD R4, 3 ADD R3, R4, R3 ; R3 = (2+3) MUL R2, R3, R2 ; R2 = 20*(2+3) LOAD R3, 37.5 DIV R2, R3, R2 ; R2 = 20*(2+3)/37.5 ADD R1, R2, R1 ; R1 = 40+20*(2+3)/37.5

Slide 22

Slide 22 text

Code Generation • Processing the parse tree in some way • Usually a traversal of the parse tree b = 40 + 20*(2+3)/37.5 40 20 2 3 + * 37.5 / + b = 40 + 20*(2+3)/37.5 LOAD R1, 40 LOAD R2, 20 LOAD R3, 2 LOAD R4, 3 ADD R3, R4, R3 ; R3 = (2+3) MUL R2, R3, R2 ; R2 = 20*(2+3) LOAD R3, 37.5 DIV R2, R3, R2 ; R2 = 20*(2+3)/37.5 ADD R1, R2, R1 ; R1 = 40+20*(2+3)/37.5 STORE R1, “b”

Slide 23

Slide 23 text

Comments • Concept is mostly straightforward • Omitting many horrible details • More covered in a compilers course.

Slide 24

Slide 24 text

Parsing (revisited) • Parsing is probably most annoying problem • Not a matter of simple text processing • Not obvious • Not fun

Slide 25

Slide 25 text

Lex & Yacc • Programming tools for writing parsers • Lex - Lexical analysis (tokenizing) • Yacc - Yet Another Compiler Compiler (parsing) • History: - Yacc : ~1973. Stephen Johnson (AT&T) - Lex : ~1974. Eric Schmidt and Mike Lesk (AT&T) - Both are standard Unix utilities - GNU equivalents: flex and bison - Part of IEEE POSIX 1003.2 standard - Implementations available for most programming languages

Slide 26

Slide 26 text

Lex/Yacc Big Picture token specification grammar specification scanner.l parser.y

Slide 27

Slide 27 text

Lex/Yacc Big Picture token specification grammar specification scanner.l parser.y /* scanner.l */ %{ #include “header.h” int lineno = 1; %} %% [ \t]* ; /* Ignore whitespace */ \n { lineno++; } [0-9]+ { yylval.val = atoi(yytext); return NUMBER; } [a-zA-Z_][a-zA-Z0-9_]* { yylval.name = strdup(yytext); return ID; } \+ { return PLUS; } - { return MINUS; } \* { return TIMES; } \/ { return DIVIDE; } = { return EQUALS; } %%

Slide 28

Slide 28 text

Lex/Yacc Big Picture token specification grammar specification scanner.l parser.y /* parser.y */ %{ #include “header.h” %} %union { char *name; int val; } %token PLUS MINUS TIMES DIVIDE EQUALS %token ID; %token NUMBER; %% start : ID EQUALS expr; expr : expr PLUS term | expr MINUS term | term ; ...

Slide 29

Slide 29 text

Lex/Yacc Big Picture token specification grammar specification scanner.l parser.y

Slide 30

Slide 30 text

Lex/Yacc Big Picture token specification grammar specification scanner.l parser.y lex scanner.c

Slide 31

Slide 31 text

Lex/Yacc Big Picture token specification grammar specification scanner.l parser.y lex scanner.c yacc parser.c

Slide 32

Slide 32 text

Lex/Yacc Big Picture token specification grammar specification scanner.l parser.y lex scanner.c yacc parser.c typecheck.c codegen.c otherstuff.c

Slide 33

Slide 33 text

Lex/Yacc Big Picture token specification grammar specification scanner.l parser.y lex scanner.c yacc parser.c typecheck.c codegen.c otherstuff.c mycompiler

Slide 34

Slide 34 text

Lex/Yacc Comments • Code generators • Create a parser from a specification • Classic versions create C code. • Variants target other languages

Slide 35

Slide 35 text

PLY • Python Lex-Yacc • 100% Python version of lex/yacc toolset • History: - Late 90’s. “Write don’t you rewrite SWIG in Python?” - 2000 : “No! Now stop bugging me about it!” - 2001 : Dave teaches a compilers course at UofC. An experiment. Students write a compiler in Python. - 2001 : PLY-1.0 developed and released. - 2002 - 2005 : Occasional maintenance and bug fixes. - 2006 : Major update to PLY-2.x (in progress). • This is the first talk about it

Slide 36

Slide 36 text

PLY Overview • Provide same functionality as lex/yacc • Identical parsing algorithm (LALR(1)) • Extensive error checking. • Comparable debugging features (sic) • Keep it simple (ha!) • Make use of Python features

Slide 37

Slide 37 text

PLY Package • PLY consists of two Python modules ply.lex ply.yacc • You simply import the modules to use them • However, PLY is not a code generator • This is where it gets interesting

Slide 38

Slide 38 text

lex.py example import ply.lex as lex tokens = [ ‘NAME’,’NUMBER’,’PLUS’,’MINUS’,’TIMES’, ’DIVIDE’, EQUALS’ ] t_ignore = ‘ \t’ t_PLUS = r’\+’ t_MINUS = r’-’ t_TIMES = r’\*’ t_DIVIDE = r’/’ t_EQUALS = r’=’ t_NAME = r’[a-zA-Z_][a-zA-Z0-9_]*’ def t_NUMBER(t): r’\d+’ t.value = int(t.value) return t lex.lex() # Build the lexer

Slide 39

Slide 39 text

lex.py specification • Tokens denoted by t_TOKEN declarations • Tokens are defined by regular expressions t_NAME = r’[a-zA-Z_][a-zA-Z0-9_]*’ def t_NUMBER(t): r’\d+’ t.value = int(t.value) return t • May be a simple variable or a function • For functions, regex is in docstring.

Slide 40

Slide 40 text

lex.py construction • lex() function is used to build the lexer import ply.lex as lex ... token specifications ... lex.lex() # Build the lexer • Uses introspection to read spec • Token information taken out of calling module • Big difference between Unix lex and PLY

Slide 41

Slide 41 text

lex.py construction • lex() function is used to build the lexer import ply.lex as lex ... token specifications ... lex.lex() # Build the lexer • Uses introspection to read spec • Token information taken out of calling module • Big difference between Unix lex and PLY try: raise RuntimeError except RuntimeError: e,b,t = sys.exc_info() f = t.tb_frame f = f.f_back mdict = f.f_globals Sick introspection hack

Slide 42

Slide 42 text

lex.py Validation • lex.lex() performs extensive error checking • Bad tokens, duplicate tokens, malformed functions, etc. t_NAME = r’[a-zA-Z_][a-zA-Z0-9_]*’ ... t_NAME = r’[a-zA-Z][a-zA-Z0-9]*’ calc.py:20 Rule t_NAME redefined. Previously defined on line 14. • Goal: informative debugging messages

Slide 43

Slide 43 text

lex.py use • Two functions: input(), token() import ply.lex as lex ... lex.lex() # Build the lexer ... data = “x = 3*4+5-6” lex.input(data) # Feed some text while 1: tok = lex.token() # Get next token if not tok: break print tok • Call token() repeatedly to fetch tokens

Slide 44

Slide 44 text

Example

Slide 45

Slide 45 text

yacc.py preliminaries • yacc.py is a module for creating a parser • Assumes you have defined a BNF grammar assign : NAME EQUALS expr expr : expr PLUS term | expr MINUS term | term term : term TIMES factor | term DIVIDE factor | factor factor : NUMBER

Slide 46

Slide 46 text

yacc.py example import ply.yacc as yacc import mylexer # Import lexer information tokens = mylexer.tokens # Need token list def p_assign(p): ‘’’assign : NAME EQUALS expr’’’ def p_expr(p): ‘’’expr : expr PLUS term | expr MINUS term | term’’’ def p_term(p): ‘’’term : term TIMES factor | term DIVIDE factor | factor’’’ def p_factor(p): ‘’’factor : NUMBER’’’ yacc.yacc() # Build the parser

Slide 47

Slide 47 text

yacc.py rules • All rules defined by p_funcname(p) funcs • Grammar specified in docstrings def p_expr(p): ‘’’expr : expr PLUS term | expr MINUS term | term’’’ • Rules may be split apart or combined def p_expr_plus(p): ‘expr : expr PLUS term’ def p_expr_minus(p): ‘expr : expr MINUS term’ def p_expr_term(p): ‘expr : term’

Slide 48

Slide 48 text

yacc.py construction • yacc() function builds the parser import ply.yacc as yacc ... rule specifications ... yacc.yacc() # Build the parser • Uses introspection (as before) • Generates parsing tables and diagnostics % python myparser.py yacc: Warning. no p_error() function is defined yacc: Generating LALR parsing table...

Slide 49

Slide 49 text

yacc.py performance • yacc.yacc() is expensive (several seconds) • Parsing tables written to file parsetab.py • Only regenerated when grammar changes • Avoids performance hit on repeated use

Slide 50

Slide 50 text

yacc.py validation • yacc.yacc() also performs validation • Duplicate rules, malformed grammars, infinite recursion, undefined symbols, bad arguments, etc. • Provides the same error messages provided by Unix yacc.

Slide 51

Slide 51 text

yacc.py parsing • yacc.parse() function yacc.yacc() # Build the parser ... data = “x = 3*4+5*6” yacc.parse(data) # Parse some text • This implicitly feeds data into lexer

Slide 52

Slide 52 text

Example

Slide 53

Slide 53 text

A peek inside • PLY is based on LR-parsing. LALR(1) • AKA: Shift-reduce parsing • Widely used. • Table driven. • Speed is independent of grammar size

Slide 54

Slide 54 text

LR Parsing • Three basic components: • A stack of grammar symbols and values. • Two operators: shift, reduce • An underlying state machine. • Example

Slide 55

Slide 55 text

(1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: LR Example: Step 1 stack input X = 3 + 4 * 5 $end

Slide 56

Slide 56 text

X = 3 + 4 * 5 $end Grammar PLY Rules Action: LR Example: Step 1 stack input (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 57

Slide 57 text

Grammar PLY Rules Action: LR Example: Step 1 stack input NAME ‘X’ shift = 3 + 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 58

Slide 58 text

Grammar PLY Rules Action: LR Example: Step 1 stack input NAME ‘X’ shift = 3 + 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Symbol type Symbol value

Slide 59

Slide 59 text

NAME ‘X’ (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules LR Example: Step 2 stack input Action: = 3 + 4 * 5 $end

Slide 60

Slide 60 text

= 3 + 4 * 5 $end NAME ‘X’ Grammar PLY Rules LR Example: Step 2 stack input Action: (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 61

Slide 61 text

NAME ‘X’ Grammar PLY Rules LR Example: Step 2 stack input Action: shift EQUALS ‘=’ 3 + 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 62

Slide 62 text

(1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules LR Example: Step 3 stack input Action: NAME ‘X’ EQUALS ‘=’ 3 + 4 * 5 $end

Slide 63

Slide 63 text

3 + 4 * 5 $end Grammar PLY Rules LR Example: Step 3 stack input Action: NAME ‘X’ EQUALS ‘=’ (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 64

Slide 64 text

Grammar PLY Rules LR Example: Step 3 stack input Action: shift NAME ‘X’ EQUALS ‘=’ NUMBER 3 + 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 65

Slide 65 text

Grammar PLY Rules LR Example: Step 3 stack input Action: shift NAME ‘X’ EQUALS ‘=’ NUMBER 3 + 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) def t_NUMBER(t): r’\d+’ t.value = int(t.value) return t

Slide 66

Slide 66 text

+ 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules LR Example: Step 4 stack input Action: NAME ‘X’ EQUALS ‘=’ NUMBER 3

Slide 67

Slide 67 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 4 stack input Action: NAME ‘X’ EQUALS ‘=’ NUMBER 3 (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 68

Slide 68 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 4 stack input Action: reduce using rule 8 NAME ‘X’ EQUALS ‘=’ NUMBER 3 (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 69

Slide 69 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 4 stack input Action: reduce using rule 8 NAME ‘X’ EQUALS ‘=’ (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) factor None

Slide 70

Slide 70 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 4 stack input Action: reduce using rule 8 NAME ‘X’ EQUALS ‘=’ (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) factor None This is None because p_factor() didn’t do anything. More later. def p_factor(p): ‘’’factor : NUMBER’’’

Slide 71

Slide 71 text

+ 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules LR Example: Step 5 stack input NAME ‘X’ EQUALS ‘=’ factor None Action:

Slide 72

Slide 72 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 5 stack input NAME ‘X’ EQUALS ‘=’ factor None (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Action:

Slide 73

Slide 73 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 5 stack input NAME ‘X’ EQUALS ‘=’ factor None (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Action: reduce using rule 7

Slide 74

Slide 74 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 5 stack input NAME ‘X’ EQUALS ‘=’ (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) term None Action: reduce using rule 7

Slide 75

Slide 75 text

+ 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules LR Example: Step 6 stack input NAME ‘X’ EQUALS ‘=’ term None Action:

Slide 76

Slide 76 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 6 stack input NAME ‘X’ EQUALS ‘=’ term None (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Action:

Slide 77

Slide 77 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 6 stack input NAME ‘X’ EQUALS ‘=’ term None (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Action: reduce using rule 4

Slide 78

Slide 78 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 6 stack input NAME ‘X’ EQUALS ‘=’ (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Action: reduce using rule 4 expr None

Slide 79

Slide 79 text

+ 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules LR Example: Step 7 stack input NAME ‘X” EQUALS ‘=’ expr None Action:

Slide 80

Slide 80 text

Grammar PLY Rules LR Example: Step 7 stack input Action: ???? NAME ‘X’ EQUALS ‘=’ expr None + 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 81

Slide 81 text

Grammar PLY Rules LR Example: Step 7 stack input Action: + 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) NAME ‘X’ EQUALS ‘=’ expr None shift

Slide 82

Slide 82 text

Grammar PLY Rules LR Example: Step 7 stack input Action: (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) NAME ‘X’ EQUALS ‘=’ expr None shift PLUS ‘+’ 4 * 5 $end

Slide 83

Slide 83 text

4 * 5 $end NAME EQUALS expr PLUS ‘X’ ‘=’ None ‘+’ (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: shift LR Example: Step 8 stack input NUMBER

Slide 84

Slide 84 text

* 5 $end NAME EQUALS expr PLUS NUMBER ‘X’ ‘=’ None ‘+’ 4 (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: reduce using rule 8 LR Example: Step 9 stack input

Slide 85

Slide 85 text

* 5 $end NAME EQUALS expr PLUS factor ‘X’ ‘=’ None ‘+’ None (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: reduce using rule 7 LR Example: Step 9 stack input

Slide 86

Slide 86 text

* 5 $end NAME EQUALS expr PLUS term ‘X’ ‘=’ None ‘+’ None (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: shift LR Example: Step 10 stack input TIMES

Slide 87

Slide 87 text

5 $end NAME EQUALS expr PLUS term TIMES ‘X’ ‘=’ None ‘+’ None ‘*’ (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: shift LR Example: Step 11 stack input NUMBER

Slide 88

Slide 88 text

$end NAME EQUALS expr PLUS term TIMES NUMBER ‘X’ ‘=’ None ‘+’ None ‘*’ 5 (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: reduce using rule 8 LR Example: Step 12 stack input

Slide 89

Slide 89 text

$end NAME EQUALS expr PLUS term TIMES factor ‘X’ ‘=’ None ‘+’ None ‘*’ None (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: reduce using rule 5 LR Example: Step 13 stack input

Slide 90

Slide 90 text

$end NAME EQUALS expr PLUS term ‘X’ ‘=’ None ‘+’ None (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: reduce using rule 2 LR Example: Step 14 stack input

Slide 91

Slide 91 text

$end NAME EQUALS expr ‘X’ ‘=’ None (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: reduce using rule 1 LR Example: Step 15 stack input

Slide 92

Slide 92 text

$end assign None (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules Action: Done. LR Example: Step 16 stack input

Slide 93

Slide 93 text

Yacc Rule Execution • Rules are executed during reduction def p_term_mul(p): ‘term : term TIMES factor’ • Parameter p refers to values on stack stack: NAME EQUALS expr PLUS term TIMES factor stack: NAME EQUALS expr PLUS term reduce term : term TIMES factor p[1] p[2] p[3] p[0]

Slide 94

Slide 94 text

def p_assign(p): ‘’’assign : NAME EQUALS expr’’’ print “Assigning”, p[1],”value”,p[3] def p_expr_plus(p): ‘’’expr : expr PLUS term’’’ p[0] = p[1] + p[3] def p_term_mul(p): ‘’’term : term TIMES factor’’’ p[0] = p[1] * p[3] def p_factor(p): ‘’’factor : NUMBER’’’ p[0] = p[1] Example: Calculator

Slide 95

Slide 95 text

+ 4 * 5 $end (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) Grammar PLY Rules LR Example: Step 4 stack input Action: NAME ‘X’ EQUALS ‘=’ NUMBER 3

Slide 96

Slide 96 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 4 stack input Action: NAME ‘X’ EQUALS ‘=’ NUMBER 3 (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 97

Slide 97 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 4 stack input Action: reduce using rule 8 NAME ‘X’ EQUALS ‘=’ NUMBER 3 (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p)

Slide 98

Slide 98 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 4 stack input Action: reduce using rule 8 NAME ‘X’ EQUALS ‘=’ (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) factor 3

Slide 99

Slide 99 text

+ 4 * 5 $end Grammar PLY Rules LR Example: Step 4 stack input Action: reduce using rule 8 NAME ‘X’ EQUALS ‘=’ (1) assign : NAME EQUALS expr -> p_assign(p) (2) expr : expr PLUS term -> p_expr(p) (3) | expr MINUS term (4) | term (5) term : term TIMES factor -> p_term(p) (6) | term DIVIDE factor (7) | factor (8) factor : NUMBER -> p_factor(p) factor 3 This retains its value because of assignment in p_factor(). def p_factor(p): ‘’’factor : NUMBER’’’ p[0] = p[1]

Slide 100

Slide 100 text

def p_assign(p): ‘’’assign : NAME EQUALS expr’’’ p[0] = (‘ASSIGN’,p[1],p[3]) def p_expr_plus(p): ‘’’expr : expr PLUS term’’’ p[0] = (‘+’,p[1],p[3]) def p_term_mul(p): ‘’’term : term TIMES factor’’’ p[0] = (‘*’,p[1],p[3]) def p_factor(p): ‘’’factor : NUMBER’’’ p[0] = (‘NUM’,p[1]) Example: Parse Tree

Slide 101

Slide 101 text

Ambiguous Grammars def p_assign(p): ‘’’assign : NAME EQUALS expr’’’ def p_expr(p): ‘’’expr : expr PLUS expr | expr MINUS expr | expr TIMES expr | expr DIVIDE expr | NUMBER’’’ 3 + 4 * 5 + 3 * 4 5 * + 5 3 4 ? ?

Slide 102

Slide 102 text

Ambiguous Grammars • Multiple possible parse trees • Is reported as a “shift/reduce conflict” yacc: Generating LALR parsing table... yacc: 16 shift/reduce conflicts • May also get “reduce/reduce conflict” • Probably most mysterious aspect of yacc

Slide 103

Slide 103 text

* 5 $end NAME EQUALS Grammar Possible Actions: Shift/Reduce Conflict Explained stack input expr PLUS expr (1) assign : NAME EQUALS expr (2) expr : expr PLUS expr (3) | expr MINUS expr (4) | expr TIMES expr (5) | expr DIVIDE expr (6) | NUMBER

Slide 104

Slide 104 text

* 5 $end NAME EQUALS (1) assign : NAME EQUALS expr (2) expr : expr PLUS expr (3) | expr MINUS expr (4) | expr TIMES expr (5) | expr DIVIDE expr (6) | NUMBER Grammar Possible Actions: Shift/Reduce Conflict Explained stack input reduce using rule 2 expr PLUS expr

Slide 105

Slide 105 text

* 5 $end NAME EQUALS (1) assign : NAME EQUALS expr (2) expr : expr PLUS expr (3) | expr MINUS expr (4) | expr TIMES expr (5) | expr DIVIDE expr (6) | NUMBER Grammar Possible Actions: Shift/Reduce Conflict Explained stack input reduce using rule 2 NAME EQUALS expr NAME EQUALS expr TIMES NAME EQUALS expr TIMES NUMBER NAME EQUALS expr TIMES expr NAME EQUALS expr expr PLUS expr

Slide 106

Slide 106 text

NAME EQUALS Grammar Possible Actions: Shift/Reduce Conflict Explained stack input reduce using rule 2 shift TIMES NAME EQUALS expr NAME EQUALS expr TIMES NAME EQUALS expr TIMES NUMBER NAME EQUALS expr TIMES expr NAME EQUALS expr (1) assign : NAME EQUALS expr (2) expr : expr PLUS expr (3) | expr MINUS expr (4) | expr TIMES expr (5) | expr DIVIDE expr (6) | NUMBER expr PLUS expr * 5 $end

Slide 107

Slide 107 text

NAME EQUALS Grammar Possible Actions: Shift/Reduce Conflict Explained stack input reduce using rule 2 shift TIMES NAME EQUALS expr NAME EQUALS expr TIMES NAME EQUALS expr TIMES NUMBER NAME EQUALS expr TIMES expr NAME EQUALS expr NAME EQUALS expr PLUS expr TIMES NAME EQUALS expr PLUS expr TIMES NUMBER NAME EQUALS expr PLUS expr TIMES expr NAME EQUALS expr PLUS expr NAME EQUALS expr (1) assign : NAME EQUALS expr (2) expr : expr PLUS expr (3) | expr MINUS expr (4) | expr TIMES expr (5) | expr DIVIDE expr (6) | NUMBER expr PLUS expr * 5 $end

Slide 108

Slide 108 text

Shift/reduce resolution precedence = ( (‘left’,’PLUS’,’MINUS’), (‘left’,’TIMES’,’DIVIDE’), ) def p_assign(p): ‘’’assign : NAME EQUALS expr’’’ def p_expr(p): ‘’’expr : expr PLUS expr | expr MINUS expr | expr TIMES expr | expr DIVIDE expr | NUMBER’’’ • Default action is to always shift • Can sometimes control with precedence

Slide 109

Slide 109 text

Error handling/recovery • Syntax errors first fed through p_error() def p_error(p): print “Syntax error” • Then an ‘error’ symbol is shifted onto stack • Stack is unwound until error is consumed

Slide 110

Slide 110 text

5 $end NAME EQUALS ‘X’ ‘=’ (1) assign : NAME EQUALS expr (2) | NAME EQUALS error (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER Grammar Error recovery stack input expr PLUS expr 3 ‘+’ 4

Slide 111

Slide 111 text

NAME EQUALS ‘X’ ‘=’ (1) assign : NAME EQUALS expr (2) | NAME EQUALS error (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER Grammar Error recovery stack input expr PLUS expr 3 ‘+’ 4 5 $end syntax error

Slide 112

Slide 112 text

NAME EQUALS ‘X’ ‘=’ (1) assign : NAME EQUALS expr (2) | NAME EQUALS error (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER Grammar Error recovery stack input expr PLUS expr 3 ‘+’ 4 5 $end syntax error def p_error(p): print “Syntax error”

Slide 113

Slide 113 text

NAME EQUALS ‘X’ ‘=’ (1) assign : NAME EQUALS expr (2) | NAME EQUALS error (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER Grammar Error recovery stack input expr PLUS expr 3 ‘+’ 4 $end error

Slide 114

Slide 114 text

NAME EQUALS ‘X’ ‘=’ (1) assign : NAME EQUALS expr (2) | NAME EQUALS error (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER Grammar Error recovery stack input expr PLUS error 3 ‘+’ $end

Slide 115

Slide 115 text

NAME EQUALS ‘X’ ‘=’ (1) assign : NAME EQUALS expr (2) | NAME EQUALS error (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER Grammar Error recovery stack input expr error 3 $end

Slide 116

Slide 116 text

NAME EQUALS ‘X’ ‘=’ (1) assign : NAME EQUALS expr (2) | NAME EQUALS error (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER Grammar Error recovery stack input error $end

Slide 117

Slide 117 text

NAME EQUALS ‘X’ ‘=’ Grammar Error recovery stack input error $end (1) assign : NAME EQUALS expr (2) | NAME EQUALS error (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER def p_assign_err(p): ‘assign : NAME EQUALS error’ print “Bad assignment”

Slide 118

Slide 118 text

Grammar Error recovery stack input $end (1) assign : NAME EQUALS expr (2) | NAME EQUALS error (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER def p_assign_err(p): ‘assign : NAME EQUALS error’ print “Bad assignment” assign

Slide 119

Slide 119 text

Debugging Output • PLY creates a file parser.out • Contains detailed debugging information • Reading it involves voodoo and magic • Useful if trying to track down conflicts

Slide 120

Slide 120 text

Debugging Output Grammar Rule 1 statement -> NAME = expression Rule 2 statement -> expression Rule 3 expression -> expression + expression Rule 4 expression -> expression - expression Rule 5 expression -> expression * expression Rule 6 expression -> expression / expression Rule 7 expression -> NUMBER Terminals, with rules where they appear * : 5 + : 3 - : 4 / : 6 = : 1 NAME : 1 NUMBER : 7 error : Nonterminals, with rules where they appear expression : 1 2 3 3 4 4 5 5 6 6 statement : 0 Parsing method: LALR state 0 (0) S' -> . statement (1) statement -> . NAME = expression (2) statement -> . expression (3) expression -> . expression + expression (4) expression -> . expression - expression (5) expression -> . expression * expression (6) expression -> . expression / expression (7) expression -> . NUMBER NAME shift and go to state 1 NUMBER shift and go to state 2 expression shift and go to state 4 statement shift and go to state 3 state 1 (1) statement -> NAME . = expression = shift and go to state 5 state 10 (1) statement -> NAME = expression . (3) expression -> expression . + expression (4) expression -> expression . - expression (5) expression -> expression . * expression (6) expression -> expression . / expression $end reduce using rule 1 (statement -> NAME = expression .) + shift and go to state 7 - shift and go to state 6 * shift and go to state 8 / shift and go to state 9 state 11 (4) expression -> expression - expression . (3) expression -> expression . + expression (4) expression -> expression . - expression (5) expression -> expression . * expression (6) expression -> expression . / expression ! shift/reduce conflict for + resolved as shift. ! shift/reduce conflict for - resolved as shift. ! shift/reduce conflict for * resolved as shift. ! shift/reduce conflict for / resolved as shift. $end reduce using rule 4 (expression -> expression - expression .) + shift and go to state 7 - shift and go to state 6 * shift and go to state 8 / shift and go to state 9 ! + [ reduce using rule 4 (expression -> expression - expression .) ] ! - [ reduce using rule 4 (expression -> expression - expression .) ] ! * [ reduce using rule 4 (expression -> expression - expression .) ] ! / [ reduce using rule 4 (expression -> expression - expression .) ]

Slide 121

Slide 121 text

Debugging Output Grammar Rule 1 statement -> NAME = expression Rule 2 statement -> expression Rule 3 expression -> expression + expression Rule 4 expression -> expression - expression Rule 5 expression -> expression * expression Rule 6 expression -> expression / expression Rule 7 expression -> NUMBER Terminals, with rules where they appear * : 5 + : 3 - : 4 / : 6 = : 1 NAME : 1 NUMBER : 7 error : Nonterminals, with rules where they appear expression : 1 2 3 3 4 4 5 5 6 6 statement : 0 Parsing method: LALR state 0 (0) S' -> . statement (1) statement -> . NAME = expression (2) statement -> . expression (3) expression -> . expression + expression (4) expression -> . expression - expression (5) expression -> . expression * expression (6) expression -> . expression / expression (7) expression -> . NUMBER NAME shift and go to state 1 NUMBER shift and go to state 2 expression shift and go to state 4 statement shift and go to state 3 state 1 (1) statement -> NAME . = expression = shift and go to state 5 state 10 (1) statement -> NAME = expression . (3) expression -> expression . + expression (4) expression -> expression . - expression (5) expression -> expression . * expression (6) expression -> expression . / expression $end reduce using rule 1 (statement -> NAME = expression .) + shift and go to state 7 - shift and go to state 6 * shift and go to state 8 / shift and go to state 9 state 11 (4) expression -> expression - expression . (3) expression -> expression . + expression (4) expression -> expression . - expression (5) expression -> expression . * expression (6) expression -> expression . / expression ! shift/reduce conflict for + resolved as shift. ! shift/reduce conflict for - resolved as shift. ! shift/reduce conflict for * resolved as shift. ! shift/reduce conflict for / resolved as shift. $end reduce using rule 4 (expression -> expression - expression .) + shift and go to state 7 - shift and go to state 6 * shift and go to state 8 / shift and go to state 9 ! + [ reduce using rule 4 (expression -> expression - expression .) ] ! - [ reduce using rule 4 (expression -> expression - expression .) ] ! * [ reduce using rule 4 (expression -> expression - expression .) ] ! / [ reduce using rule 4 (expression -> expression - expression .) ] ... state 11 (4) expression -> expression - expression . (3) expression -> expression . + expression (4) expression -> expression . - expression (5) expression -> expression . * expression (6) expression -> expression . / expression ! shift/reduce conflict for + resolved as shift. ! shift/reduce conflict for - resolved as shift. ! shift/reduce conflict for * resolved as shift. ! shift/reduce conflict for / resolved as shift. $end reduce using rule 4 (expression -> expression - expression .) + shift and go to state 7 - shift and go to state 6 * shift and go to state 8 / shift and go to state 9 ! + [ reduce using rule 4 (expression -> expression - expression .) ] ! - [ reduce using rule 4 (expression -> expression - expression .) ] ! * [ reduce using rule 4 (expression -> expression - expression .) ] ! / [ reduce using rule 4 (expression -> expression - expression .) ] ...

Slide 122

Slide 122 text

Advanced PLY • PLY supports more advanced yacc features • Empty productions • Error handling/recovery • Inherited attributes • Embedded actions

Slide 123

Slide 123 text

Advanced PLY (cont) • Some Python specific features • Lexers/parsers can be defined as classes • Support for multiple lexers and parsers • Support for optimized mode (-O)

Slide 124

Slide 124 text

Class Example import ply.yacc as yacc class MyParser: def p_assign(self,p): ‘’’assign : NAME EQUALS expr’’’ def p_expr(self,p): ‘’’expr : expr PLUS term | expr MINUS term | term’’’ def p_term(self,p): ‘’’term : term TIMES factor | term DIVIDE factor | factor’’’ def p_factor(self,p): ‘’’factor : NUMBER’’’ def build(self): self.parser = yacc.yacc(object=self)

Slide 125

Slide 125 text

Summary • This has been a quick tour of PLY/yacc • Have skipped a lot of subtle details.

Slide 126

Slide 126 text

Why use PLY? • Standard lex/yacc well known and used • Suitable for large grammars • Decent performance • Very extensive error checking/validation

Slide 127

Slide 127 text

PLY Usage • Thousands of downloads over five years • Some applications (that I know of) • Teaching compilers • numbler.com (Carl Shimer) • Parsing Ada source code. • Parsing molecule descriptions • Reading configuration files

Slide 128

Slide 128 text

Resources • PLY homepage http://www.dabeaz.com/ply • Mailing list/group http://groups.google.com/group/ply-hack

Slide 129

Slide 129 text

$end (1) assign : NAME EQUALS expr (2) | NAME EQUALS NUMBER (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER Grammar Possible Actions: Reduce/Reduce Conflict Explained stack input NAME EQUALS NUMBER

Slide 130

Slide 130 text

$end Grammar Possible Actions: Reduce/Reduce Conflict Explained stack input reduce using rule 2 NAME EQUALS NUMBER (1) assign : NAME EQUALS expr (2) | NAME EQUALS NUMBER (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER

Slide 131

Slide 131 text

$end Grammar Possible Actions: Reduce/Reduce Conflict Explained stack input reduce using rule 2 (1) assign : NAME EQUALS expr (2) | NAME EQUALS NUMBER (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER assign

Slide 132

Slide 132 text

$end Grammar Possible Actions: Reduce/Reduce Conflict Explained stack input reduce using rule 2 reduce using rule 7 assign NAME EQUALS NUMBER (1) assign : NAME EQUALS expr (2) | NAME EQUALS NUMBER (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER

Slide 133

Slide 133 text

$end Grammar Possible Actions: Reduce/Reduce Conflict Explained stack input reduce using rule 2 reduce using rule 7 assign NAME EQUALS (1) assign : NAME EQUALS expr (2) | NAME EQUALS NUMBER (3) expr : expr PLUS expr (4) | expr MINUS expr (5) | expr TIMES expr (6) | expr DIVIDE expr (7) | NUMBER expr