Modelling State Machines with Ragel

Slide 1

Slide 1 text

MODELLING STATE MACHINES WITH RAGEL Eurucamp, 2013 Drew Neil @nelstrom “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

Slide 2

Slide 2 text

Vimprint Turns keystrokes into plain English

Slide 3

Slide 3 text

What is a state machine anyway? “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

Slide 4

Slide 4 text

Chapter 3 The Simplest Computers A Finite State Machine, also known as a Finite Automaton, is a drastically simplified model of a computer that is easy to understand, easy to reason about, and easy to implement in hardware or software.

Slide 5

Slide 5 text

1 2 a a

Slide 6

Slide 6 text

1 2 a a

Slide 7

Slide 7 text

Chapter 3 The Simplest Computers It’s possible to convert any regular expression into an equivalent Nondeterministic Finite Automaton -- every string matched by the regular expression is accepted by the NFA, and vice versa.

Slide 8

Slide 8 text

What about Vim? “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Normal h, j, k, l

Slide 11

Slide 11 text

Normal h, j, k, l Insert i, I, a, A any

Slide 12

Slide 12 text

Normal Insert Visual CmdLine

Slide 13

Slide 13 text

Meet Ragel “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

Slide 14

Slide 14 text

Ragel state machines can not only recognize byte sequences as regular expression machines do, but can also execute code at arbitrary points in the recognition of a regular language.

Slide 15

Slide 15 text

host %% inline ragel %%{ multi-line ragel action { host }; }%% host Birds-eye view of a ragel file

Slide 16

Slide 16 text

$ ragel -h | grep "host language" host language: -C The host language is C (default) -D The host language is D -Z The host language is Go -J The host language is Java -R The host language is Ruby -A The host language is C# -O The host language is OCaml Supported host languages

Slide 17

Slide 17 text

$ ragel -R source.rl Compile source.rl using Ruby as host language Producing source.rb as output $ ragel -V source.rl > graph.dot Generate a visualisation in .dot format Can be viewed with Graphviz Compiling Ragel

Slide 18

Slide 18 text

%%{ machine accepter; motion = [hjklbwe0]; switch = [iIaAsSoO]; escape = 27; input = (any - escape); insert_mode := ( input* escape @{ fret; } ); normal_mode := ( motion | switch @{ fcall insert_mode; } )*; }%% class Accepter attr_accessor :data def initialize() %% write data; end def process(input) @data = input.unpack("c*") stack = [] %% write init; %% write exec; return cs end end accepter.rl

Slide 19

Slide 19 text

def process(input) @data = input.unpack("c*") stack = [] begin " p ||= 0 " pe ||= data.length " cs = accepter_start " top = 0 end begin " _klen, _trans, _keys, _acts = nil " _goto_level = 0 " _resume = 10 " _eof_trans = 15 " _again = 20 # 150 more LOC like this... end return cs end " def process(input) @data = input.unpack("c*") stack = [] %% write init; %% write exec; return cs end

Slide 20

Slide 20 text

hellohello “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

Slide 21

Slide 21 text

h e l l o h e l l o ␛ h move left e move to end of word l move right l move right o hello enter “hello” on a new line ␛ return to Normal mode

Slide 22

Slide 22 text

Machine definition: = ; Ragel syntax Machine instantiation: := ; TOKENS PARSERS

Slide 23

Slide 23 text

Machine definition: = ; Ragel syntax Machine instantiation: := ; # TOKENS motion = [hjklbwe0]; switch = [iIaAsSoO]; escape = 27; input = (any - escape); # PARSER normal_mode := (motion)*;

Slide 24

Slide 24 text

Ragel actions expr >{action} entering action expr @{action} finishing action

Slide 25

Slide 25 text

Ragel control flow fcall ; Push the target state and jump to the entry point defined by fret; Return to the target state of the transition on which the last fcall was made. PUSH POP

Slide 26

Slide 26 text

%%{ machine accepter; # ... normal_mode := ( motion | switch @{ fcall insert_mode; } )*; }%% https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl

Slide 27

Slide 27 text

%%{ machine accepter; # ... insert_mode := ( input* escape @{ fret; } ); }%% https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl

Slide 28

Slide 28 text

%%{ machine accepter; motion = [hjklbwe0]; switch = [iIaAsSoO]; escape = 27; input = (any - escape); insert_mode := ( input* escape @{ fret; } ); normal_mode := ( motion | switch @{ fcall insert_mode; } )*; }%% https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl

Slide 29

Slide 29 text

2 IN 1 insert_mode normal_mode '0', 'b', 'e', 'h', 'j'..'l', 'w' 'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode 3 27 / return DEF

Slide 30

Slide 30 text

class Accepter def process(input) @data = input.unpack("c*") %% write init; %% write exec; return cs end def accept?(input) process(input) > 0 end end https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl

Slide 31

Slide 31 text

describe Accepter do it 'accepts hellohello' do assert Accepter.new.accept?("hellohello\e") end it 'rejects visual mode (for example)' do refute Accepter.new.accept?("viw") end end https://github.com/nelstrom/ragel-vim-demo/blob/master/test/accepter_test.rb

Slide 32

Slide 32 text

Producing side-effects “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

Slide 33

Slide 33 text

index character 0 1 2 3 4 5 6 7 8 9 10 h e l l o h e l l o Ragel’s cursor 2 IN 1 insert_mode normal_mode '0', 'b', 'e', 'h', 'j'..'l', 'w' 'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode 3 27 / return DEF p = 0 cs = 2

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

%%{ machine emitter; action H { @head = p } action T { @tail = p } # TOKENS motion = [hjklbwe0] >H@T; switch = [iIaAsSoO] >H@T; escape = 27 >H@T; input = (any - escape) >H@T; # ... }%% https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/emitter.rl

Slide 45

Slide 45 text

class Emitter attr_accessor :data # initialize # process def strokes @data[@head..@tail].pack('c*') end end https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/emitter.rl

Slide 46

Slide 46 text

%%{ machine emitter; motion = [hjklbwe0] >H@T @{ @events << {motion: strokes} }; switch = [iIaAsSoO] >H@T @{ @events << {switch: strokes} }; escape = 27 >H@T @{ @events << {escape: ''} }; input = (any - escape) >H@T @{ @events << {input: strokes} }; # ... }%% https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/emitter.rl

Slide 47

Slide 47 text

describe Emitter do it 'accepts motions, switches, and insertions' do Emitter.new(events = []).process("hellohello\e") assert_equal [ {:motion=>"h"}, {:motion=>"e"}, {:motion=>"l"}, {:motion=>"l"}, {:switch=>"o"}, {:input=>"h"}, {:input=>"e"}, {:input=>"l"}, {:input=>"l"}, {:input=>"o"}, {:escape=>""} ], events end end https://github.com/nelstrom/ragel-vim-demo/blob/master/test/emitter_test.rb

Slide 48

Slide 48 text

Handling Visual mode “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

Slide 49

Slide 49 text

h e l l Normal o hello Normal > Insert ␛ Normal Insertion

Slide 50

Slide 50 text

h e l l Normal Normal > Visual ␛ Normal Selection b iw v

Slide 51

Slide 51 text

h e l l Normal Normal > Visual ␛ Normal Change selection b iw v c hello Normal > Insert

Slide 52

Slide 52 text

fcall ; Push the target state and jump to the entry point defined by fret; Return to the target state of the transition on which the last fcall was made. fnext ; Set the next state to be the entry point defined by label. PUSH POP SWITCH Ragel control flow

Slide 53

Slide 53 text

%%{ machine visual; action H { @head = p } action T { @tail = p } # ... start_visual = 'v' >H@T @{ @events << {start_visual: strokes} }; normal_mode := ( motion | start_visual @{ fcall visual_mode; } )*; }%% https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/visual.rl

Slide 54

Slide 54 text

%%{ machine visual; action H { @head = p } action T { @tail = p } # ... text_object = ([ai][bBpstwW]) >H@T @{ @events << {text_object: strokes} }; v_switch = [sScC] >H@T @{ @events << {switch: strokes} }; # ... }%% https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/visual.rl

Slide 55

Slide 55 text

%%{ machine visual; # ... visual_mode := ( ( motion | text_object )* ( v_switch @{ fnext insert_mode; } | escape @{ fret; } ) ); }%% https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/visual.rl

Slide 56

Slide 56 text

it 'accepts vhjiwhelloel' do Visual.new(events = []).process("vhjiwchello\eel") assert_equal [ {:start_visual=>"v"}, {:motion=>"h"}, {:motion=>"j"}, {:text_object=>"iw"}, {:switch=>"c"}, {:input=>"h"}, {:input=>"e"}, {:input=>"l"}, {:input=>"l"}, {:input=>"o"}, {:escape=>""}, {:motion=>"e"}, {:motion=>"l"} ], events end https://github.com/nelstrom/ragel-vim-demo/blob/master/test/visual_test.rb

Slide 57

Slide 57 text

4 IN 1 insert_mode 2 visual_mode normal_mode '0', 'b', 'e', 'h', 'j'..'l', 'w' / H, T, 6:33 'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / H, T, 7:33, 30:13 'v' / H, T, 11:40, 31:19 5 6 27 / H, T, 8:33, 17:13 DEF / H, T, 9:33 27 / H, T, 8:33, 24:15 '0', 'b', 'e', 'h', 'j'..'l', 'w' / H, T, 6:33 'C', 'S', 'c', 's' / H, T, 13:40, 23:17 3 'a', 'i' / H 'B', 'W', 'b', 'p', 's'..'t', 'w' / T, 12:40

Slide 58

Slide 58 text

State charts for debugging “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

Slide 59

Slide 59 text

["x]x Delete [count] characters under and after the cursor [into register x]. Does the same as "dl". :help x x cut 1 character, save to default register 2x cut 2 characters, save to default register “ax cut 1 character, save to register a

Slide 60

Slide 60 text

2”ax cut 2 characters, save to register a “a2x cut 2 characters, save to register a 3“a2x cut 6 characters, save to register a ["x]x Delete [count] characters under and after the cursor [into register x]. Does the same as "dl". :help x

Slide 61

Slide 61 text

count = [1-9]; register = '"' [a-z]; cut_command = count? register? count? 'x'; normal_mode := (cut_command)*; https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/faulty_cut.rl

Slide 62

Slide 62 text

5 IN normal_mode 1 '"' 4 '1'..'9' 'x' 2 'a'..'z' 3 '1'..'9' 'x' 'x' '"' '1'..'9' 'x'

Slide 63

Slide 63 text

count = [1-9]; register = '"' [a-z]; cut_command = count? register? count? 'x'; normal_mode := (cut_command)*; https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/faulty_cut.rl

Slide 64

Slide 64 text

count = [1-9]; register = '"' [a-z]; cut_command = ( count? register )? count? 'x'; normal_mode := (cut_command)*; https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/cut.rl

Slide 65

Slide 65 text

5 IN normal_mode 1 '"' 4 '1'..'9' 'x' 2 'a'..'z' 3 '1'..'9' 'x' 'x' '"' 'x'

Slide 66

Slide 66 text

5 IN normal_mode 1 '"' 4 '1'..'9' 'x' 2 'a'..'z' 3 '1'..'9' 'x' 'x' '"' 'x' x

Slide 67

Slide 67 text

5 IN normal_mode 1 '"' 4 '1'..'9' 'x' 2 'a'..'z' 3 '1'..'9' 'x' 'x' '"' 'x' 2x

Slide 68

Slide 68 text

5 IN normal_mode 1 '"' 4 '1'..'9' 'x' 2 'a'..'z' 3 '1'..'9' 'x' 'x' '"' 'x' “a2x

Slide 69

Slide 69 text

5 IN normal_mode 1 '"' 4 '1'..'9' 'x' 2 'a'..'z' 3 '1'..'9' 'x' 'x' '"' 'x' 3“a2x

Slide 70

Slide 70 text

Wrapping up all code from these slides github.com/nelstrom/ragel-vim-demo github.com/nelstrom/vimprint a parser for Vim commands #pairwithme