Compiler in JavaScript using ANTLR

Slides from my talk at BerlinJS meetup in August 2018. The talk was about language compilers and basic principles behind it.

Transcript

1. COMPILER ALENA KHINEIKA IN JAVASCRIPT USING ANTLR
2. ALENA KHINEIKA
4. C# JAVA SHELL JAVASCRIPT PYTHON
5. SOURCE-TO-SOURCE TRANSFORMATION
6. • Find existing parsers for specific programming languages SOURCE-TO-SOURCE TRANSFORMATION
7. • Find existing parsers for specific programming languages • Create your own parser SOURCE-TO-SOURCE TRANSFORMATION
8. • Find existing parsers for specific programming languages • Create your own parser • Use some tools and libraries to generate a parser SOURCE-TO-SOURCE TRANSFORMATION
9. JAVASCRIPT PARSERS ESPRIMA FALAFEL UGLIFYJS JISON
10. + JAVASCRIPT PARSERS ESPRIMA FALAFEL UGLIFYJS JISON
11. ANY LANGUAGE TO ANY LANGUAGE + BSON DOCUMENTS
12. HOW TO WRITE COMPILER FROM SCRATCH?
13. Lexical Analysis Syntax Analysis Code Generation JS 3 STAGES OF COMPILER DESIGN
14. 1. LEXICAL ANALYSIS { X : 1 } OpenBrace { Identifier x Colon : Identifier 1 CloseBrace } Tokens Raw Code Lexical Analysis JS Tokens Input
15. 2. SYNTAX ANALYSIS Lexical Analysis Syntax Analysis Code Generation JS Tokens Tree Input Output
16. FROM STRING TO TREE Tree Structure { X : 1 } OpenBrace { Identifier x Colon : Identifier 1 CloseBrace } Tokens Raw Code
17. 3. CODE GENERATION Lexical Analysis Syntax Analysis Code Generation JS Tokens Tree Input Output
18. TO DO LIST 1. So 2. Many 3. Things
19. HOW TO SIMPLIFY Don’t write from scratch!
20. USE SOME TOOLS AND LIBRARIES
24. NOT SIMPLE ENOUGH...
25. +
26. ANTLR • Another Tool for Language Recognition • Generates parser basing on grammars • Written in Java, but has JavaScript runtime
27. PARSE TREE VS. AST * AST - Abstract Syntax Tree
28. Parse Tree • Represents concrete syntax • Contain unusable information Abstract Syntax Tree • Represents abstract syntax • Contain only meaningful information
29. JAVASCRIPT INPUT + ANTLR 4 = = PYTHON OUTPUT + BREAKPOINT
32. ANTLR AUXILIARY FILES *Lexer.js Lexical Analysis *Parser.js Syntax Analysis JS Characters Tree Input Tokens *Visitor.js Tree Traversal New Tree
35. Input JS Characters
36. Characters Tokens
37. Tree Tokens
38. http://bit.ly/compiler-in-javascript This article can be viewed as both a practical guide for writing a JavaScript compiler and a theoretical resource that describes the basic concepts and principles of compiler design.
40. VISITOR VS. LISTENER
41. → enterPropertyAssignment(ctx) → enterPropertyName(ctx) → enterIdentifierName(ctx) → enterTerminal(ctx) ← exitTerminal(ctx) ← exitIdentifierName(ctx) ← exitPropertyName(ctx) ← exitPropertyAssignment(ctx) ↓ visitPropertyNameAndValue(ctx) ↓ visitPropertyAssignment(ctx) ↓ visitPropertyName(ctx) ↓ visitIdentifierName(ctx) ↓ visitTerminal(ctx) Visitor • You should explicitly call child methods • Methods can return any custom type • You can modify existing nodes Listener • Methods are called automatically • Methods can’t return a value • You should store values outside the tree
42. JAVASCRIPT INPUT { x: 1 } VISITOR 1
43. JAVASCRIPT INPUT { x: 1 } VISITOR 1 ROOT NODE LEAF
44. JAVASCRIPT INPUT { x: 1 } VISITOR 1 ROOT NODE LEAF NO CHANGES HERE 2
45. JAVASCRIPT INPUT { x: 1 } VISITOR 1 ROOT NODE LEAF NO CHANGES HERE 2 FORMATTING x: 1 ‘x’: 1 3
46. JAVASCRIPT INPUT { x: 1 } VISITOR 1 ROOT NODE LEAF NO CHANGES HERE 2 FORMATTING x: 1 ‘x’: 1 3 PYTHON OUTPUT { ‘x’: 1 }
47. Not here! Here!
48. Visiting the root node 1
49. JAVASCRIPT { x: 1 } PYTHON { ‘x’: 1 } TRANSFORMATION
50. Visiting the left and the right side of the tree separately
51. Standard methods: visit() visitTerminal() LEFT HAND SIDE VISITING * The same logic is applicable for the right side NO CHANGES HERE 2
52. Formatting x: 1 ‘x’: 1 3
53. FLOATING POINT NUMBERS DIFFERENT NUMERAL SYSTEMS SINGLE AND DOUBLE QUOTES What do we support? Don't damage Pi Answer should be accurate USER INPUT FORMAT Spaces, empty lines etc. ESCAPE SEQUENCES Have to be escaped COMMENTS Yes / No
55. JavaScript input: { x: Number(1) } Python output: { ‘x’: int(1) }
56. COMPILERS ARE AWESOME!
58. ALENA KHINEIKA @alena_khineika @alena_khineika @alenakhineika