Finding meaning: grammars, lexers, and parsers

Have you ever wondered how your computer figures out the meaning of your code? Humans can quickly get lost, but the computer always keeps up, no matter how deeply you nest those conditionals. In this session, we'll learn how computers use lexers and parsers to determine the meaning of code (or any other structured syntax) in accordance to a particular grammar. We'll discuss some of the theory behind grammars, and the role of context in grammars (and human languages). And we'll walk through the eqn gem (https://github.com/schneidmaster/eqn) which applies these principles and uses a parser generator to safely evaluate mathematical equation strings in ruby.

Transcript

1. FINDING MEANING: GRAMMARS, LEXERS, AND PARSERS ZACH SCHNEIDER
2. OVERVIEW • First, we’ll take a high-level look at how languages function in general (both human and computer) • Second, we’ll discuss how computers interpret code in accordance to the rules of a particular grammar • Third, we’ll explore how parsers can be a powerful tool for application-level programming, and walk through the eqn gem as an example of applying a parser to solve a real-world problem • A caveat: it would take years to study all of the branches of linguistics and compiler theory in detail; this is a 30-minute presentation and I will gloss over (or entirely omit) a number of topics :)
3. A BRIEF INTRODUCTION TO LANGUAGE • Natural vs. formal languages • Natural languages, like English, evolved naturally and are predominantly used for communication • Formal languages, like ruby, are designed and specified by people for a specific purpose (in ruby’s case, to express computations and computer instructions)
4. A BRIEF INTRODUCTION TO LANGUAGE • Lexical vs. syntactical grammar • Lexical grammar describes every valid word (or token) that can be used in a language. In English, “developer” is a valid word but “zxkwefl” is not. In ruby, “def” and “end” are valid words, but “>.<” is not. • Syntactical grammar describes how words can be arranged to form statements or sentences. In English, valid sentences require a noun and a verb; “banana apple pear” is not a sentence, but “Zach is a programmer” is a sentence. In ruby, there are many restrictions on valid token arrangements; for example, every “def” must have a matching “end”
5. A BRIEF INTRODUCTION TO LANGUAGE • Syntax vs. semantics • The syntax of a language describes the form of valid statements/sentences but does not guarantee that the program conveys meaning (just that it is structured correctly) • “Colorless green ideas sleep furiously” is a grammatically correct English sentence but does not mean anything • “John is a married bachelor” is also grammatically correct but has a meaning that is self-contradictory • Semantic analysis is performed after syntactical analysis, and may provide further guarantees that the meaning of a program is valid or matches the programmer’s intent — for example, type checking in strongly typed languages is a form of semantic analysis
6. A BRIEF INTRODUCTION TO LANGUAGE • Context and ambiguity • Natural languages like English are poorly suited for computer programs • Context: information that aids in interpretation of language but is not strictly contained in the grammar • What does “what’s up?” mean? “What is currently above my head?” or “What have you been doing recently?” • Ambiguity: statements that can only be interpreted with context, often fluid and hard to deterministically describe • Words with multiple meanings, like “bank”
7. A BRIEF INTRODUCTION TO LANGUAGE • Noam Chomsky: the “father of modern linguistics” (and a respected analytic philosopher and sociologist) • Developed the theory of universal grammar (which might be wrong, but was a foundational work for decades of linguistics scholarship) • Wrote a number of political and philosophical works, such as “Manufacturing Consent” • And developed the “Chomsky hierarchy” of formal grammars
8. A BRIEF INTRODUCTION TO LANGUAGE • Chomsky hierarchy • Type-0: all formal grammars (all languages that can be recognized by a Turing machine) • Type-1: context-sensitive languages (nearly all natural languages can be expressed in a context-sensitive grammar) • Type-2: context-free languages (nearly all programming languages are context-free, sort of — more on that in a minute) • Type-3: regular languages (also represented as regular expressions) • Hierarchical relationship — all regular expressions could be represented by a context-free grammar, but not all context-free languages can be represented by regular expressions
9. LANGUAGE & PROGRAMMING • First, a few terms: • A lexer is a program that takes text input and converts it into a series of tokens. “( 2 - 3 ) * 5” would be represented as something like [LEFT_PAREN, NUM, MINUS_OP, NUM, RIGHT_PAREN, MUL_OP, NUM] • Invalid lexical grammar is caught by the lexer (if it encounters some text that it does not recognize as a valid token) • A parser is another program that reads in the series of tokens and creates a syntax tree, or a logical representation of the relationship between tokens • Invalid syntactical grammar is caught by the parser (if it encounters a series of tokens that are in an order that does not make sense)
10. LANGUAGE & PROGRAMMING • How parsers work • There are many different types of parser implementations — optimized for different language structures, ease of expressing grammar, etc. • Fundamentally, most parsers use a “stack” of tokens. They consume input tokens from the lexer, and based on the rules of the grammar they either “shift” (push the new token onto the stack) or “reduce” (combine the top token on the stack with the new token) • There are a number of other implementation details beyond the scope of this talk — for example, some “lookahead” to future tokens which makes it easier to express grammatical rules • There are tradeoffs between performance and ease of expression
11. LANGUAGE & PROGRAMMING • Programming languages are context free (ish) • All programming languages are something close to context free: their syntax can be understood by looking at the current token and the token at the top of the stack • But many of them get some help. For example, context-free grammars cannot express whitespace-sensitive languages like Python, so the parser gets some help from the lexer which does some extra processing. (But the actual syntax of the Python code itself is context-free.)
12. LANGUAGE & PROGRAMMING • Putting it all together • For most programming languages, the computer interprets code via a few steps: • First, the lexer converts the code into a series of tokens • Second, the parser consumes the tokens and converts them into a syntax tree representing the logic of the program • Third, the syntax tree is evaluated — at runtime, this translates to a series of commands to the computer • Programming languages are generally context-free, which makes it relatively easy for computers to understand them
13. COOL. WHY DO I CARE? • There’s a few reasons why you should care about how computers understand code: • First, it’s nice to have a sense of what goes on under the hood • Second, it helps you code better: now you know what a “syntax error” really means! • Third, in addition to executing code itself, parsers are an extremely powerful tool at the application layer for evaluating input
14. AHA! SCORECARDS • Aha! has scorecard functionality — you can define a custom equation for scoring your features, and score each feature by filling in metric values
15. AHA! SCORECARDS • If you sub in the values, you end up with a string like “2 - (2 * 3) + (3 * 5)” (but it could be any arbitrary math equation) — how do we evaluate this? • Can’t plug it into eval — the user could inject all sorts of nasty things into their equation. (We could try to scrub the input, but that’s a complicated problem that’s very scary to get wrong.) • Can’t easily solve with regex — we could maybe write a recursive regular expression, but this would be super painful and complicated • Can’t give up — my boss will be unhappy :(
16. PARSERS TO THE RESCUE! • Turns out that basic algebraic arithmetic can be expressed as a context-free grammar! • Very safe — not eval-ing anything, and nasty user input will simply fail to parse • Relatively easy to avoid bugs — the parser does the heavy lifting, and the application code is to solve relatively simple problems like “combine these two numbers using a multiplication operation”
17. SPECIFICALLY, PARSER GENERATORS TO THE RESCUE • You should always, always, always use a parser generator — don’t try to roll your own • Every language has them: lex and yacc in C/C++, rex and racc in ruby (and Treetop — more on that shortly) • Rock solid — built on decades of computer science scholarship, very hard to find bugs • Conversely, very hard to roll your own — tons of bugs and edge cases, a lot of theory to understand, etc • And you end up writing an order of magnitude more code than you need to
18. EQN • eqn: my gem that uses the Treetop parser generator to safely evaluate mathematical equation strings • https://github.com/schneidmaster/eqn • dentaku: a counterexample of a homebuilt lexer/parser • Solve the same problem, but eqn has 818 LOC and dentaku has 3311 LOC (excluding tests) • Bugs: unary minus, recursion in general • https://github.com/rubysolo/dentaku
19. QUESTIONS? • That’s all! Questions?