Pattern Matching

MagRails 2011

Transcript

1. Pattern Matching
2. omg
3. Steak Holder
4. Aaron Patterson
5. @tenderlove
6. Ruby core team
7. Rails core team
8. ! WARNING !
9. Stage II: Advanced Beginner
10. 0. Excited to be in England 1. English country, I have accent. 2. Fear and guilt 3. Fear of plugs and outlets 4. Can't stop thinking of strange 5. Romance Ebi
11. Manchester Agile Rails
16. ! WARNING !
17. Graphviz http://graphviz.org
18. world hello greeting
19. digraph nfa { rankdir=LR; world [shape = doublecircle]; hello [shape = circle]; hello -> world [label="greeting"]; } graph.dot
20. digraph nfa { rankdir=LR; world [shape = doublecircle]; hello [shape = circle]; goodbye [shape = circle]; hello -> world [label="greeting"]; goodbye -> world } graph.dot
21. world hello greeting goodbye
22. 2 3 . 4 / /articles(.:format) 5 6 /articles/new(.:format) 0 1 / articles (?-mix:[^./?]+) new
23. Pattern Matching Move between theory and practice
24. Journey Yes, it's named after the '70s rock sensation
25. What is a router? Provides 1: URL generation 2: Path Recognition 3: Route parsing
26. Why a new router?
27. Maintenance
28. 0 700 1400 2100 2800 Journey Journey-2 Rack-Mount LOC
29. Known algorithms
30. References • Compilers: Principles, Techniques, & Tools (Aho, Lam, Sethi, Ullman) • Intro to Formal Languages and Automata (Linz)
31. Predictability
32. CPU Time
33. Memory Usage
34. Current Performance url generation path recognition route parsing rack-mount Journey
35. Patterns
36. Why patterns matter
37. Regular Expressions /om(g)!/
38. Rails Routes
39. resource :articles
40. /articles(.:format) /articles/new(.:format) /articles/:id/edit(.:format) /articles/:id(.:format)
41. Parens don't capture
42. :whatever => /([^./?]+)/
43. /articles(.:format) /\/articles(?:\.([^.\/?]+))?$/
44. /\/articles(?:.([^.\/?]+))?$/ /\/articles\/new(?:.([^.\/?]+))?$/ /\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/ /\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/ GET /articles/new 200 OK!
45. /\/articles(?:.([^.\/?]+))?$/ /\/articles\/new(?:.([^.\/?]+))?$/ /\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/ /\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/ GET /foos/new 404 Not Found
46. How long does this take?
47. r = routes.length
48. x = regexp compare
49. O(r ⨉ x)
50. Can we do better?
51. Finite State Machines Parse Trees Automata
52. Parse Trees
53. Parsing regexp
54. (a|b)*abb
55. ◦ ◦ b ◦ b a * a b () | Describe node types
56. Parsing rails routes
57. Journey::Parser parser = Journey::Parser.new ast = parser.parse '/articles(.:format)' puts ast.to_dot
58. ◦ ◦ () / articles ◦ . :format
59. To String! parser = Journey::Parser.new ast = parser.parse '/articles(.:format)' puts ast.to_s puts ast.to_regexp
60. OR AST parser = Journey::Parser.new trees = [ '/articles(.:format)', '/articles/new(.:format)', ].map { |s| parser.parse s } ast = Journey::Nodes::Or.new trees puts ast.to_dot
61. | ◦ ◦ ◦ () / articles ◦ . :format ◦ () ◦ new ◦ / / articles ◦ . :format
62. SECRET FEATURE parser = Journey::Parser.new ast = parser.parse '/articles|books(.:format)' SUPER SECRET!
63. Automata
64. (a|b)*abb
65. 3 0 b 2 a b 1 a a b b a Double circle is acceptance state
66. baabb
67. 3 0 b 2 a b 1 a a b b a
68. 3 0 b 2 a b 1 a a b b a b
69. 3 0 b 2 a b 1 a a b b a ba
70. 3 0 b 2 a b 1 a a b b a baa
71. 3 0 b 2 a b 1 a a b b a baabb
72. aab 3 0 b 2 a b 1 a a b b a
73. Deterministic Finite Automaton
74. Only one path
75. Storage class DFA attr_reader :table def initialize @table = Hash.new { |h, from| h[from] = {} } @table[0]['a'] = 1 @table[0]['b'] = 0 @table[1]['a'] = 1 @table[1]['b'] = 2 @table[2]['a'] = 1 @table[2]['b'] = 3 @table[3]['b'] = 0 @table[3]['a'] = 2 end end
76. FSM Simulation
77. class Simulator def initialize(dfa) @dfa = dfa end def simulate(symbols) state = 0 until symbols.empty? state = @dfa.move(state, symbols.shift) end state end end
78. class DFA ... def move(from, symbol) @table[from][symbol] end end move function
79. irb> sim = Simulator.new(DFA.new) => #<Simulator:0x007f95929a82e0 ...> irb> sim.simulate %w{ b a a b b } => 3 irb> sim.simulate %w{ a a b } => 2
80. Time: O(n) n = string.length
81. Space: S + T states.length + transitions.length
82. Nondeterministic Finite Automaton
83. Has nil edges
84. Can't tell direction
85. Simulation of NFA
86. 6 0 2 ε 4 ε 3 a 5 b ε ε a|b
87. 6 0 2 ε 4 ε 3 a 5 b ε ε nil-closure
88. 6 0 2 ε 4 ε 3 a ε 5 b ε a
89. Storage class NFA def initialize @table = Hash.new { |h, from| h[from] = Hash.new { |i,sym| i[sym] = [] } } @table[0][nil] << 2 @table[0][nil] << 4 @table[2]['a'] << 3 @table[4]['b'] << 5 @table[3][nil] << 6 @table[5][nil] << 6 end def nil_closure(states) states.map { |s| @table[s][nil] }.flatten end end
90. FSM Simulation class Simulator def initialize(nfa) @nfa = nfa end def simulate(symbols) states = @nfa.nil_closure([0]) until symbols.empty? next_s = @nfa.move(states, symbols.shift) states = @nfa.nil_closure(next_s) end states end end
91. Move function class NFA ... def move(states, symbol) states.map { |s| @table[s][symbol] }.flatten end end
92. irb> sim = Simulator.new(NFA.new) => #<Simulator:0x007faa188a5f88 ...> irb> sim.simulate %w{ a } => [6] irb> sim.simulate %w{ b } => [6] irb> sim.simulate %w{ b b } => [] irb> sim.simulate %w{ c } => []
93. Time: O(r ⨉ x) r = operators.length, x = string.length
94. NFA Construction
95. ◦ ◦ () / articles ◦ . :format /articles(.:format)
96. cat nodes 1 0 /
97. /articles 2 0 1 / articles
98. Optional 3 0 ε 1 ε 2 ????? ε
99. 6 0 1 / 2 articles ε 3 ε 4 . 5 :format ε
100. parser = Journey::Parser.new ast = parser.parse '/articles(.:format)' nfa = Journey::NFA::Builder.new ast tt = nfa.transition_table puts tt.to_dot Journey::NFA
101. Converting NFA to DFA
102. Eliminate nil transitions
103. Collapse duplicate edges
104. 6 0 1 / 2 articles ε 3 ε 4 . 5 :format ε /articles(.:format)
105. 2 3 . 4 0 1 / articles :format /articles(.:format)
106. CODES! parser = Journey::Parser.new ast = parser.parse '/articles(.:format)' nfa = Journey::NFA::Builder.new ast tt = nfa.transition_table.generalized_table puts tt.to_dot
107. SHORTER CODES! parser = Journey::Parser.new ast = parser.parse '/articles(.:format)' dfa = Journey::GTG::Builder.new ast tt = dfa.transition_table puts tt.to_dot
108. ANY NFA converts to DFA
109. O(r ⨉ x) => O(x) r = operations.length, x = string.length
110. Converting Automata to Regexp
111. 2 3 . 4 0 1 / articles :format /articles(.:format)
112. 2 3 . 4 0 /articles :format /articles(.:format)
113. /articles(.:format) 2 4 .:format 0 /articles
114. /articles(.:format) 4 0 /articles(.:format)?
115. Generalized Transition Graph
116. /users(.:format) /users/new(.:format) /users/:id/edit(.:format) /users/:id(.:format) resource :users
117. 2 3 . 4 / 5 6 8 . 7 9 / 10 . 11 12 14 . 13 15 0 1 / users (?-mix:[^./?]+) new (?-mix:[^./?]+) (?-mix:[^./?]+) edit (?-mix:[^./?]+) (?-mix:[^./?]+) resource :users
118. The Plan?
119. Combine All Routes
120. Produce DFA
121. Simulate in O(n) Time
122. The Future
123. JS Simulation
125. Rails Console irb> File.open('out.html', 'wb') { |f| irb* f.write( irb* Wot::Application.routes.router.visualizer irb> )} => 69074
126. Table => JSON parser = Journey::Parser.new ast = parser.parse '/articles(.:format)' dfa = Journey::GTG::Builder.new ast tt = dfa.transition_table puts tt.to_json
127. Table => SVG parser = Journey::Parser.new ast = parser.parse '/articles(.:format)' dfa = Journey::GTG::Builder.new ast tt = dfa.transition_table puts tt.to_svg
128. JS Tokenizer function tokenize(input, callback) { while(input.length > 0) { callback(input.match(/^[\/\.\?]|[^\/\.\?]+/)[0]); input = input.replace(/^[\/\.\?]|[^\/\.\?]+/, ''); } }
129. JS Simulator tokenize(input, function(token) { var new_states = []; for(var key in states) { var state = states[key]; if(string_states[state] && string_states[state][token]) { var new_state = string_states[state][token]; highlight_edge(state, new_state); highlight_state(new_state); new_states.push(new_state); } } if(new_states.length == 0) { return; } states = new_states; });
130. routes.rb marshalling
131. Test suggestions
132. Test coverage
133. Usage Heat Maps
134. ????
135. Open Questions
136. Is our GTG deterministic?
137. /users/new|/users/:id 4 5 0 1 / 2 users 3 / new (?-mix:[^./?]+)
138. "new" =~ /new|[^.\/?]+/
139. Can we make it deterministic?
140. L1 = {new} L2 = {[^./?]+}
141. /users/new|/users/:id 4 5 0 1 / 2 users 3 / L1 L2 - L1
142. Is it worth our effort?
143. REGEX AST NFA DFA
144. DFA => YACC
145. DFA => RACC
146. DFA => Ragel
147. Is it worth our effort?
148. Thanks Tatsuya Ono!
149. Thank You!!
150. <3<3<3<3<3