Reimplement RegExp in JavaScript

Regular expressions are used in many places to filter and validate input. While they are cute and powerful, they are usually hard to write and debug. So why not implement the entire RegExp object in JavaScript, such that each execution step can be visualized and debugging becomes easy? And what happens if you build a RegExp JIT in JavaScript that gets JITed by the JavaScript JIT … ;)

Transcript

1. RegExp. exec(‘js’) @jviereck JSConf.EU 2013
2. /a/ /.?/ /(foo|bar)+/ /(^.*a)+/ /([a-zA-Z]\s\d)+/ /jsconf\.eu/ /^abc$/ /(?=Hello)\sWorld/ /.*?/ /[^]/
3. /a/ /.?/ /(foo|bar)+/ /(^.*a)+/ /([a-zA-Z]\s\d)+/ /jsconf\.eu/ /^abc$/ /(?=Hello)\sWorld/ /.*?/ /[^]/ They look cute
4. /a/ /.?/ /(foo|bar)+/ /(^.*a)+/ /([a-zA-Z]\s\d)+/ /jsconf\.eu/ /^abc$/ /(?=Hello)\sWorld/ /.*?/ /[^]/ They look cute But they are hard to get right :(
5. “Why are you doing this?”
6. Motivation • Debug-ability • Writing JIT compiler • Wanted to present at JSConf.EU :)
9. Demo http://jviereck.github.io/regexp.js/
10. Overview • ParseTree • NodeList • Matcher • RegExpJS object itself • Writing a JIT in a JIT • Conclusion
11. ParseTree 4
12. ParseTree • ParseTree: • structural representation of a string • described by grammar rules • Highlevel: • RegExp =[Parser]=> ParseTree
13. Example: /(ab|c)/ ParseTree
14. Example: /(ab|c)/ ParseTree Group
15. Disjunction Example: /(ab|c)/ ParseTree Group
16. Alternative Disjunction Example: /(ab|c)/ ParseTree Group
17. Char: “a” Term Alternative Disjunction Example: /(ab|c)/ ParseTree Group
18. Char: “b” Term Char: “a” Term Alternative Disjunction Example: /(ab|c)/ ParseTree Group
19. Char: “b” Term Char: “a” Term Alternative Alternative Disjunction Example: /(ab|c)/ ParseTree Group
20. Char: “c” Term Char: “b” Term Char: “a” Term Alternative Alternative Disjunction Example: /(ab|c)/ ParseTree Group
21. Parser • Handwritten - LL(?) • Only basic error handling/messages • Uses RegExp iteself << replace them ;) • Emits “useful” information
22. Useful information { "type": "ref", "ref": 1, "from": 5, "to": 7, "raw": "\\1" } (a|b)\1
27. /[a-]/.exec(‘-’): ✓
28. /[a-]/.exec(‘-’): ✓ /[a-b]/.exec(‘-’): ✗
29. Notes on parsing • Should have gone with parser generator? • Mostly follows the spec • Addition: added handling for /]/ • Parsing numbers needs an extra slide...
30. Number parsing
31. Number parsing • parseDecimalEscape(...)
32. Number parsing • parseDecimalEscape(...) • /(a)(b)\2/ ‏ reference
33. Number parsing • parseDecimalEscape(...) • /(a)(b)\2/ ‏ reference • /(a)(b)\71/ ‏ NOT reference
34. Number parsing • parseDecimalEscape(...) • /(a)(b)\2/ ‏ reference • /(a)(b)\71/ ‏ NOT reference • IF: octal number, parseInt(71,8)
35. Number parsing • parseDecimalEscape(...) • /(a)(b)\2/ ‏ reference • /(a)(b)\71/ ‏ NOT reference • IF: octal number, parseInt(71,8) • /(a)(b)\91/ ‏ NOT reference
36. Number parsing • parseDecimalEscape(...) • /(a)(b)\2/ ‏ reference • /(a)(b)\71/ ‏ NOT reference • IF: octal number, parseInt(71,8) • /(a)(b)\91/ ‏ NOT reference • ELSE: Ignore slash, match /91/
37. Number parsing • parseDecimalEscape(...) • /(a)(b)\2/ ‏ reference • /(a)(b)\71/ ‏ NOT reference • IF: octal number, parseInt(71,8) • /(a)(b)\91/ ‏ NOT reference • ELSE: Ignore slash, match /91/ V8 at least...
38. NodeList 8
39. • ParseTree ➡ NodeList • Data structure Matcher uses NodeList
40. • ParseTree ➡ NodeList • Data structure Matcher uses NodeList Example: /ab/
41. • ParseTree ➡ NodeList • Data structure Matcher uses NodeList Data: “a” CHAR Example: /ab/
42. CHAR Data: “b” • ParseTree ➡ NodeList • Data structure Matcher uses NodeList Data: “a” CHAR Example: /ab/
43. DONE CHAR Data: “b” • ParseTree ➡ NodeList • Data structure Matcher uses NodeList Data: “a” CHAR Example: /ab/
44. Example: /(ab|c)*c/
45. Example: /(ab|c)*c/ Id: 1 Min: 0 Max: ? REPEAT
46. Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT
47. ALTR Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT
48. Data: “a” CHAR ALTR Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT
49. Data: “b” CHAR Data: “a” CHAR ALTR Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT
50. Data: “b” CHAR Data: “a” CHAR CHAR Data: “c” ALTR Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT
51. JOIN Data: “b” CHAR Data: “a” CHAR CHAR Data: “c” ALTR Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT
52. Index: 1 END_GROUP JOIN Data: “b” CHAR Data: “a” CHAR CHAR Data: “c” ALTR Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT
53. Index: 1 END_GROUP JOIN Data: “b” CHAR Data: “a” CHAR CHAR Data: “c” ALTR Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT
54. Index: 1 END_GROUP JOIN Data: “b” CHAR Data: “a” CHAR CHAR Data: “c” ALTR Data: “c” CHAR Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT
55. Index: 1 END_GROUP JOIN Data: “b” CHAR Data: “a” CHAR CHAR Data: “c” ALTR DONE Data: “c” CHAR Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT
56. Index: 1 END_GROUP JOIN Data: “b” CHAR Data: “a” CHAR CHAR Data: “c” ALTR DONE Data: “c” CHAR Example: /(ab|c)*c/ Index: 1 BEGIN_GROUP Id: 1 Min: 0 Max: ? REPEAT Char: “c” Term Char: “b” Term Char: “a” Term Alternative Alternative Disjunction Group
57. NodeList • Look complicated to build • Easy by walking over ParseTree • Combine one node after the other
58. NodeList: \s
59. NodeList: \s • \s: matches whitespace OR line terminator
60. NodeList: \s • \s: matches whitespace OR line terminator • stolen functions from Esprima :)
61. Matcher 12
62. Matcher • Does the actual RegExp matching • Matcher(state, nodeList) • state = { string, index, traces, counters, ...} • Tries all possibilities till match OR no progress • Calls Matcher(state’, nodeList’) recursively • If trace fails ➠ return function = backtracking
63. Matcher
64. Matcher • s = new State(‘abc’)
65. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/)
66. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) a b c d a b d ^
67. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ a b c d a b d ^
68. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ a b c d a b d ^
69. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ • try /b/: match(s’, /(b|c)d/) a b c d a b d ^
70. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ • try /b/: match(s’, /(b|c)d/) a b c d a b d ^ “ s’=s.clone()
71. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ • try /b/: match(s’, /(b|c)d/) a b c d a b d ^ “ s’=s.clone()
72. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ • try /b/: match(s’, /(b|c)d/) • test(‘b’, /b/): ✓ a b c d a b d ^ “ s’=s.clone()
73. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ • try /b/: match(s’, /(b|c)d/) • test(‘b’, /b/): ✓ a b c d a b d ^ “ s’=s.clone()
74. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ • try /b/: match(s’, /(b|c)d/) • test(‘b’, /b/): ✓ a b c d a b d ^ “ s’=s.clone()
75. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ • try /b/: match(s’, /(b|c)d/) • test(‘b’, /b/): ✓ • test(‘d’, /d/): ✓ a b c d a b d ^ “ s’=s.clone()
76. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ • try /b/: match(s’, /(b|c)d/) • test(‘b’, /b/): ✓ • test(‘d’, /d/): ✓ a b c d a b d ^ “ s’=s.clone()
77. Matcher • s = new State(‘abc’) • match(s, /a(b|c)d/) • test(s, /a/): ✓ • try /b/: match(s’, /(b|c)d/) • test(‘b’, /b/): ✓ • test(‘d’, /d/): ✓ • DONE: ✓ a b c d a b d ^ “ s’=s.clone()
79. • s = new State(‘abc’);
80. • s = new State(‘abc’); • match(s, /a(b|c)d/) a c d ^ a b c d
81. • s = new State(‘abc’); • match(s, /a(b|c)d/) a c d ^ a b c d
82. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ a c d ^ a b c d
83. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ a c d ^ a b c d
84. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) a c d ^ a b c d
85. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) a c d ^ a b c d “ s’=s.clone()
86. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ a c d ^ a b c d “ s’=s.clone()
87. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack a c d ^ a b c d “ s’=s.clone()
88. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack a c d ^ a b c d “ s’=s.clone()
89. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack • try /c/: match(s’’, /(b|c)d/) a c d ^ a b c d “ s’=s.clone()
90. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack • try /c/: match(s’’, /(b|c)d/) a c d ^ a b c d “ s’=s.clone()
91. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack • try /c/: match(s’’, /(b|c)d/) • test(‘c’, /c/): ✓ a c d ^ a b c d “ s’=s.clone()
92. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack • try /c/: match(s’’, /(b|c)d/) • test(‘c’, /c/): ✓ a c d ^ a b c d “ s’=s.clone()
93. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack • try /c/: match(s’’, /(b|c)d/) • test(‘c’, /c/): ✓ a c d ^ a b c d “ s’=s.clone()
94. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack • try /c/: match(s’’, /(b|c)d/) • test(‘c’, /c/): ✓ • test(‘d’, /d/): ✓ a c d ^ a b c d “ s’=s.clone()
95. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack • try /c/: match(s’’, /(b|c)d/) • test(‘c’, /c/): ✓ • test(‘d’, /d/): ✓ a c d ^ a b c d “ s’=s.clone()
96. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack • try /c/: match(s’’, /(b|c)d/) • test(‘c’, /c/): ✓ • test(‘d’, /d/): ✓ • DONE a c d ^ a b c d “ s’=s.clone()
97. • s = new State(‘abc’); • match(s, /a(b|c)d/) • test(‘a’, /a/): ✓ • try /b/: match(s’, /bd/) • test(s’, /b/): ✗ ‏ Backtrack • try /c/: match(s’’, /(b|c)d/) • test(‘c’, /c/): ✓ • test(‘d’, /d/): ✓ • DONE record the trace along the way a c d ^ a b c d “ s’=s.clone()
98. Matcher pitfalls • /(a(b)*c)*/ • need to reset the matches of (b)* to null • also need to reset repeat counter • /()*a/ • Infinity matches ➡ Loop detection!
99. RegExpJS Object 16
100. re = new RegExpJS(‘abc’, ‘ig’); re.exec(‘abc’) re.test(‘abc’) Sorry, nothing on the String.prototype...
101. RegExpJS Object • So far: • created ParseTree • created NodeList • have Matcher(string, parseNodes) • Needed: • something to play with :) • actual JS object similar to RegExp object
102. RegExpJS Object • first attempt: • RegExpJS(regExp).exec = (str) => { regExp = this.regExp; nodes = buildNodeList(parse(regExp)); return match(str, nodes); } • RegExpJS(/abc/).exec(‘abc’): ✓ • RegExpJS(/abc/).exec(‘JSConf abc’): ✗
103. Match arbitrary start position
104. Match arbitrary start position • easy fix:
105. Match arbitrary start position • easy fix: • wrap the nodeList by “fake loop”
106. Match arbitrary start position • easy fix: • wrap the nodeList by “fake loop” • e.g: /abc/
107. Match arbitrary start position • easy fix: • wrap the nodeList by “fake loop” • e.g: /abc/ • /abc/ ➠ /[^]*?(abc)/
108. Match arbitrary start position • easy fix: • wrap the nodeList by “fake loop” • e.g: /abc/ • /abc/ ➠ /[^]*?(abc)/ • assign the first group index 0
109. Match arbitrary start position • easy fix: • wrap the nodeList by “fake loop” • e.g: /abc/ • /abc/ ➠ /[^]*?(abc)/ • assign the first group index 0 • /abc/.exec(‘JSConf abc’): ✓
110. Match arbitrary start position • easy fix: • wrap the nodeList by “fake loop” • e.g: /abc/ • /abc/ ➠ /[^]*?(abc)/ • assign the first group index 0 • /abc/.exec(‘JSConf abc’): ✓ • problem: V8 stack size not large enough
111. How about flags?
112. How about flags? •multiline flag: /foo/m
113. How about flags? •multiline flag: /foo/m • affects only /^/ and /$/ • easy to implement
114. How about flags? •multiline flag: /foo/m • affects only /^/ and /$/ • easy to implement •global flag: /foo/g
115. How about flags? •multiline flag: /foo/m • affects only /^/ and /$/ • easy to implement •global flag: /foo/g • repeat from last position • okay’isch to implement
116. How about flags? •ignore-case flag: /foo/i • not just str.toUpperCase() • e.g. if turns one character into two, then use the input character • fun with ranges: • /[a-}]/ ➠ /[a-}A-Z]/
118. More fun
119. More fun • Spec requirement: • Call `ToString(...)`, `ToNumber(...)` on input • E.g. • str = { toString: function() { return false; } } • /LS/i.exec(str) ➡ returns?
120. More fun • Spec requirement: • Call `ToString(...)`, `ToNumber(...)` on input • E.g. • str = { toString: function() { return false; } } • /LS/i.exec(str) ➡ returns? • /LS/i.exec(str) === “ls”
121. so, RegExpJS object • There is more “fun” stuff required to get the ECMA test suite to pass • index.js (implements RegExpJS) ~240 LOC
122. JIT (just in time compiler) 20
123. JIT (just in time compiler) :) :) :) 20
124. JIT Overview • Convert RegExp ➡ JS-Code • Have a RegExp-JIT running in JS-JIT • Idea: match input faster • use state machine / finite automata • only support a subset of RegExp • ... but these very fast & efficient
125. Supported features • All features, except: • groups matches, lookahead - only (?: ) • no backreferences • only greedy repetitions • Not supported yet • repetitions, ... (under construction) • (written in three evenings hacks ;) )
126. this.name = id + ': NFA-States={' + nfaStates.map(function(state) {return state.name}).join(', ') + '} '; I really mean three hack evenings!
127. RegExp:
128. RegExp: /ab/
129. RegExp: /ab/ NFA:
130. RegExp: /ab/ NFA: 0 1 4 a b 2 ε
131. RegExp: NFA: 0 1 4 a b 2 ε
132. RegExp: NFA: 0 1 4 a b 2 ε /ab|a/
133. RegExp: NFA: 0 1 4 a b 2 ε 3 a ε /ab|a/
134. RegExp: NFA: 0 1 4 a b 2 ε 3 a ε DFA: /ab|a/
135. RegExp: NFA: 0 1 4 a b 2 ε 3 a ε DFA: 0 1 a b 2 /ab|a/
136. JIT Implementation • Convert RegExp to state machine • RegExp • Non-deterministic Finite Automata (NFA) • Deterministic Finite Automata (DFA) • CodeGen ➠ JavaScript code
137. /ab/ 0 1 2 a b 23
138. alphabet: Σ={a, b} /ab/ 0 1 2 a b 23
139. What alphabet to use?
140. What alphabet to use? • e.g.: ranges like /[a-\u0777]/
141. What alphabet to use? • e.g.: ranges like /[a-\u0777]/ • use one transition per character
142. What alphabet to use? • e.g.: ranges like /[a-\u0777]/ • use one transition per character 0 1 a, b, c, ...
143. What alphabet to use? • e.g.: ranges like /[a-\u0777]/ • use one transition per character • better solution? 0 1 a, b, c, ...
144. Alphabet Classes
145. Alphabet Classes • Example: /0[1-9]/
146. Alphabet Classes • Example: /0[1-9]/ 0 1 2 0 1...9
147. Alphabet Classes • Example: /0[1-9]/ 0 1 2 0 1...9 alphabet: Σ={0, 1, 2, ..., 9}
148. Alphabet Classes • Example: /0[1-9]/ • before /ab/ 0 1 2 0 1...9 alphabet: Σ={0, 1, 2, ..., 9}
149. Alphabet Classes • Example: /0[1-9]/ • before /ab/ 0 1 2 a b 0 1 2 0 1...9 alphabet: Σ={0, 1, 2, ..., 9}
150. Alphabet Classes • Example: /0[1-9]/ • before /ab/ • let 0 1 2 a b 0 1 2 0 1...9 alphabet: Σ={0, 1, 2, ..., 9}
151. Alphabet Classes • Example: /0[1-9]/ • before /ab/ • let • a ≐ 0 0 1 2 a b 0 1 2 0 1...9 alphabet: Σ={0, 1, 2, ..., 9}
152. Alphabet Classes • Example: /0[1-9]/ • before /ab/ • let • a ≐ 0 • b ≐ 1...9 0 1 2 a b 0 1 2 0 1...9 alphabet: Σ={0, 1, 2, ..., 9}
153. Alphabet Classes • Example: /0[1-9]/ • before /ab/ • let • a ≐ 0 • b ≐ 1...9 0 1 2 a b 0 1 2 0 1...9 alphabet: Σ={0, 1, 2, ..., 9} alphabet: Σ={a, b}
154. Alphabet Classes Calculation 0 numberline
155. Alphabet Classes Calculation A ≐ 65 0 numberline
156. Alphabet Classes Calculation A ≐ 65 Z ≐ 90 0 numberline
157. Alphabet Classes Calculation A-E /[A-E]/ A ≐ 65 Z ≐ 90 0 numberline
158. Alphabet Classes Calculation A-E /[A-E]/ C-Z /[C-Z]/ A ≐ 65 Z ≐ 90 0 numberline
159. Alphabet Classes Calculation A-E /[A-E]/ C-Z /[C-Z]/ A ≐ 65 Z ≐ 90 0 numberline
160. Alphabet Classes Calculation A-E /[A-E]/ C-Z /[C-Z]/ A-C D-E F-Z A ≐ 65 Z ≐ 90 0 numberline
161. Alphabet Classes Calculation A-E /[A-E]/ C-Z /[C-Z]/ A-C D-E F-Z ≐ a ≐ b ≐ c A ≐ 65 Z ≐ 90 0 numberline
162. RegExp: /[A-E]|[C-Z]/
163. RegExp: /[A-E]|[C-Z]/ 0 1 2 3 A...E C...Z ε ε
164. RegExp: /[A-E]|[C-Z]/ 0 1 2 3 A...E C...Z ε ε 0 1 2 3 a, b b, c ε ε
165. How JITy is the JIT code?
166. “Well, branchy. Not good”
167. Closing Up 26
168. TODO • Code cleanup • JIT still under construction • Align Parser/Traversal API with Esprima’s
169. Status • Matcher works agains all test cases • Help needed: • project page • bugfixes • tester!
170. More important • Crazy ideas what to use this for :) • Esprima enabled many analysis tools • Hope RegExp.JS becomes used as well • Lint RegExp • Monitor RegExp execution
171. Thank you! Looking for feedback
172. Contact julian.viereck@gmail.com @jviereck + Julian Viereck