Modelling State Machines with Ragel

Ragel is a State Machine Compiler, which can generate Ruby code (as well as C, Java, Go, and more). It can be used for writing robust protocol implementations, parsing data formats, and performing lexical analysis of programming languages. Ragel is used in many open source projects including Gherkin, Thin, Min, Mongrel, Redcloth, Radiant, and Hpricot.

To demonstrate Ragel's capabilities we'll model Vim, which is a Finite State Machine. We'll do so by parsing a stream of Vim keystrokes, using Ragel state machines to manage the transitions between Vim's modes. We'll see how to generate state chart visualisations with Ragel, and how these can be used to debug our state machines. The resulting program will enable us to analyse and visualise the interactions of a Vim user.

Transcript

1. MODELLING
   STATE
   MACHINES
   WITH RAGEL
   Eurucamp, 2013
   Drew Neil
   @nelstrom
   “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira
   

   View Slide

2. Vimprint
   Turns keystrokes into plain English
   

   View Slide

3. What is a
   state machine
   anyway?
   “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira
   

   View Slide

4. Chapter 3
   The Simplest
   Computers
   A Finite State Machine,
   also known as a Finite
   Automaton, is a drastically
   simplified model of a
   computer that is easy to
   understand, easy to
   reason about, and easy to
   implement in hardware
   or software.
   

   View Slide

5. 1 2
   a
   a
   

   View Slide

6. 1 2
   a
   a
   

   View Slide

7. Chapter 3
   The Simplest
   Computers
   It’s possible to convert
   any regular expression
   into an equivalent
   Nondeterministic Finite
   Automaton -- every string
   matched by the regular
   expression is accepted
   by the NFA, and vice
   versa.
   

   View Slide

8. What about
   Vim?
   “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira
   

   View Slide

9. View Slide

10. Normal
    h, j, k, l
    

    View Slide

11. Normal
    h, j, k, l
    Insert
    i, I, a, A
    
    any
    

    View Slide

12. Normal
    Insert
    Visual
    CmdLine
    

    View Slide

13. Meet Ragel
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira
    

    View Slide

14. Ragel state machines can not only
    recognize byte sequences as regular
    expression machines do, but can also
    execute code at arbitrary points in the
    recognition of a regular language.
    

    View Slide

15. host
    %% inline ragel
    %%{
    multi-line ragel
    action { host };
    }%%
    host
    Birds-eye view of a ragel file
    

    View Slide

16. $ ragel -h | grep "host language"
    host language:
    -C The host language is C (default)
    -D The host language is D
    -Z The host language is Go
    -J The host language is Java
    -R The host language is Ruby
    -A The host language is C#
    -O The host language is OCaml
    Supported host languages
    

    View Slide

17. $ ragel -R source.rl
    Compile source.rl using Ruby as host language
    Producing source.rb as output
    $ ragel -V source.rl > graph.dot
    Generate a visualisation in .dot format
    Can be viewed with Graphviz
    Compiling Ragel
    

    View Slide

18. %%{
    machine accepter;
    motion = [hjklbwe0];
    switch = [iIaAsSoO];
    escape = 27;
    input = (any - escape);
    insert_mode := (
    input*
    escape @{ fret; }
    );
    normal_mode := (
    motion |
    switch @{ fcall insert_mode; }
    )*;
    }%%
    class Accepter
    attr_accessor :data
    def initialize()
    %% write data;
    end
    def process(input)
    @data = input.unpack("c*")
    stack = []
    %% write init;
    %% write exec;
    return cs
    end
    end
    accepter.rl
    

    View Slide

19. def process(input)
    @data = input.unpack("c*")
    stack = []
    begin
    " p ||= 0
    " pe ||= data.length
    " cs = accepter_start
    " top = 0
    end
    begin
    " _klen, _trans, _keys, _acts = nil
    " _goto_level = 0
    " _resume = 10
    " _eof_trans = 15
    " _again = 20
    # 150 more LOC like this...
    end
    return cs
    end
    "
    def process(input)
    @data = input.unpack("c*")
    stack = []
    %% write init;
    %% write exec;
    return cs
    end
    

    View Slide

20. hellohello
    
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira
    

    View Slide

21. h e l l o h e l l o ␛
    h
    move left
    e
    move to end of word
    l
    move right
    l
    move right
    o hello
    enter “hello” on a new line
    ␛
    return to Normal mode
    

    View Slide

22. Machine definition:
    = ;
    Ragel syntax
    Machine instantiation:
    := ;
    TOKENS
    PARSERS
    

    View Slide

23. Machine definition:
    = ;
    Ragel syntax
    Machine instantiation:
    := ;
    # TOKENS
    motion = [hjklbwe0];
    switch = [iIaAsSoO];
    escape = 27;
    input = (any - escape);
    # PARSER
    normal_mode := (motion)*;
    

    View Slide

24. Ragel actions
    expr >{action}
    entering action
    expr @{action}
    finishing action
    

    View Slide

25. Ragel control flow
    fcall ;
    Push the target state and jump to the
    entry point defined by
    fret;
    Return to the target state of the
    transition on which the last fcall
    was made.
    PUSH
    POP
    

    View Slide

26. %%{
    machine accepter;
    # ...
    normal_mode := (
    motion |
    switch @{ fcall insert_mode; }
    )*;
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl
    

    View Slide

27. %%{
    machine accepter;
    # ...
    insert_mode := (
    input*
    escape @{ fret; }
    );
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl
    

    View Slide

28. %%{
    machine accepter;
    motion = [hjklbwe0];
    switch = [iIaAsSoO];
    escape = 27;
    input = (any - escape);
    insert_mode := (
    input*
    escape @{ fret; }
    );
    normal_mode := (
    motion |
    switch @{ fcall insert_mode; }
    )*;
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl
    

    View Slide

29. 2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    

    View Slide

30. class Accepter
    def process(input)
    @data = input.unpack("c*")
    %% write init;
    %% write exec;
    return cs
    end
    def accept?(input)
    process(input) > 0
    end
    end
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl
    

    View Slide

31. describe Accepter do
    it 'accepts hellohello' do
    assert Accepter.new.accept?("hellohello\e")
    end
    it 'rejects visual mode (for example)' do
    refute Accepter.new.accept?("viw")
    end
    end
    https://github.com/nelstrom/ragel-vim-demo/blob/master/test/accepter_test.rb
    

    View Slide

32. Producing
    side-effects
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira
    

    View Slide

33. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 0
    cs = 2
    

    View Slide

34. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 1
    cs = 2
    

    View Slide

35. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 2
    cs = 2
    

    View Slide

36. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 3
    cs = 2
    

    View Slide

37. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 4
    cs = 2
    

    View Slide

38. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 5
    cs = 1
    

    View Slide

39. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 6
    cs = 1
    

    View Slide

40. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 7
    cs = 1
    

    View Slide

41. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 8
    cs = 1
    

    View Slide

42. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 9
    cs = 1
    

    View Slide

43. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 10
    cs = 3
    

    View Slide

44. %%{
    machine emitter;
    action H { @head = p }
    action T { @tail = p }
    # TOKENS
    motion = [hjklbwe0] >H@T;
    switch = [iIaAsSoO] >H@T;
    escape = 27 >H@T;
    input = (any - escape) >H@T;
    # ...
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/emitter.rl
    

    View Slide

45. class Emitter
    attr_accessor :data
    # initialize
    # process
    def strokes
    @data[@head..@tail].pack('c*')
    end
    end
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/emitter.rl
    

    View Slide

46. %%{
    machine emitter;
    motion = [hjklbwe0]
    >H@T @{ @events << {motion: strokes} };
    switch = [iIaAsSoO]
    >H@T @{ @events << {switch: strokes} };
    escape = 27
    >H@T @{ @events << {escape: ''} };
    input = (any - escape)
    >H@T @{ @events << {input: strokes} };
    # ...
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/emitter.rl
    

    View Slide

47. describe Emitter do
    it 'accepts motions, switches, and insertions' do
    Emitter.new(events = []).process("hellohello\e")
    assert_equal [
    {:motion=>"h"},
    {:motion=>"e"},
    {:motion=>"l"},
    {:motion=>"l"},
    {:switch=>"o"},
    {:input=>"h"},
    {:input=>"e"},
    {:input=>"l"},
    {:input=>"l"},
    {:input=>"o"},
    {:escape=>""}
    ], events
    end
    end
    https://github.com/nelstrom/ragel-vim-demo/blob/master/test/emitter_test.rb
    

    View Slide

48. Handling
    Visual mode
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira
    

    View Slide

49. h e l l
    Normal
    o hello
    Normal > Insert
    ␛
    Normal
    Insertion
    

    View Slide

50. h e l l
    Normal
    Normal > Visual
    ␛
    Normal
    Selection
    b iw
    v
    

    View Slide

51. h e l l
    Normal
    Normal > Visual
    ␛
    Normal
    Change selection
    b iw
    v
    c hello
    Normal > Insert
    

    View Slide

52. fcall ;
    Push the target state and jump to the
    entry point defined by
    fret;
    Return to the target state of the
    transition on which the last fcall
    was made.
    fnext ;
    Set the next state to be the entry point
    defined by label.
    PUSH
    POP
    SWITCH
    Ragel control flow
    

    View Slide

53. %%{
    machine visual;
    action H { @head = p }
    action T { @tail = p }
    # ...
    start_visual = 'v'
    >H@T @{ @events << {start_visual: strokes} };
    normal_mode := (
    motion |
    start_visual @{ fcall visual_mode; }
    )*;
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/visual.rl
    

    View Slide

54. %%{
    machine visual;
    action H { @head = p }
    action T { @tail = p }
    # ...
    text_object = ([ai][bBpstwW])
    >H@T @{ @events << {text_object: strokes} };
    v_switch = [sScC]
    >H@T @{ @events << {switch: strokes} };
    # ...
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/visual.rl
    

    View Slide

55. %%{
    machine visual;
    # ...
    visual_mode := (
    ( motion | text_object )*
    (
    v_switch @{ fnext insert_mode; } |
    escape @{ fret; }
    )
    );
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/visual.rl
    

    View Slide

56. it 'accepts vhjiwhelloel' do
    Visual.new(events = []).process("vhjiwchello\eel")
    assert_equal [
    {:start_visual=>"v"},
    {:motion=>"h"},
    {:motion=>"j"},
    {:text_object=>"iw"},
    {:switch=>"c"},
    {:input=>"h"},
    {:input=>"e"},
    {:input=>"l"},
    {:input=>"l"},
    {:input=>"o"},
    {:escape=>""},
    {:motion=>"e"},
    {:motion=>"l"}
    ], events
    end
    https://github.com/nelstrom/ragel-vim-demo/blob/master/test/visual_test.rb
    

    View Slide

57. 4
    IN
    1
    insert_mode
    2
    visual_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w' / H, T, 6:33
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / H, T, 7:33, 30:13
    'v' / H, T, 11:40, 31:19
    5
    6
    27 / H, T, 8:33, 17:13
    DEF / H, T, 9:33
    27 / H, T, 8:33, 24:15
    '0', 'b', 'e', 'h', 'j'..'l', 'w' / H, T, 6:33
    'C', 'S', 'c', 's' / H, T, 13:40, 23:17
    3
    'a', 'i' / H
    'B', 'W', 'b', 'p', 's'..'t', 'w' / T, 12:40
    

    View Slide

58. State charts
    for
    debugging
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira
    

    View Slide

59. ["x]x Delete [count] characters under and after the cursor
    [into register x]. Does the same as "dl".
    :help x
    x cut 1 character, save to default register
    2x cut 2 characters, save to default register
    “ax cut 1 character, save to register a
    

    View Slide

60. 2”ax cut 2 characters, save to register a
    “a2x cut 2 characters, save to register a
    3“a2x cut 6 characters, save to register a
    ["x]x Delete [count] characters under and after the cursor
    [into register x]. Does the same as "dl".
    :help x
    

    View Slide

61. count = [1-9];
    register = '"' [a-z];
    cut_command =
    count?
    register?
    count?
    'x';
    normal_mode := (cut_command)*;
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/faulty_cut.rl
    

    View Slide

62. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"' '1'..'9'
    'x'
    

    View Slide

63. count = [1-9];
    register = '"' [a-z];
    cut_command =
    count?
    register?
    count?
    'x';
    normal_mode := (cut_command)*;
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/faulty_cut.rl
    

    View Slide

64. count = [1-9];
    register = '"' [a-z];
    cut_command =
    (
    count?
    register
    )?
    count?
    'x';
    normal_mode := (cut_command)*;
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/cut.rl
    

    View Slide

65. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"'
    'x'
    

    View Slide

66. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"'
    'x'
    x
    

    View Slide

67. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"'
    'x'
    2x
    

    View Slide

68. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"'
    'x'
    “a2x
    

    View Slide

69. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"'
    'x'
    3“a2x
    

    View Slide

70. Wrapping up
    all code from these slides
    github.com/nelstrom/ragel-vim-demo
    github.com/nelstrom/vimprint
    a parser for Vim commands
    #pairwithme
    

    View Slide