Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Modelling State Machines with Ragel

nelstrom
August 18, 2013

Modelling State Machines with Ragel

Ragel is a State Machine Compiler, which can generate Ruby code (as well as C, Java, Go, and more). It can be used for writing robust protocol implementations, parsing data formats, and performing lexical analysis of programming languages. Ragel is used in many open source projects including Gherkin, Thin, Min, Mongrel, Redcloth, Radiant, and Hpricot.

To demonstrate Ragel's capabilities we'll model Vim, which is a Finite State Machine. We'll do so by parsing a stream of Vim keystrokes, using Ragel state machines to manage the transitions between Vim's modes. We'll see how to generate state chart visualisations with Ragel, and how these can be used to debug our state machines. The resulting program will enable us to analyse and visualise the interactions of a Vim user.

nelstrom

August 18, 2013
Tweet

More Decks by nelstrom

Other Decks in Technology

Transcript

  1. MODELLING
    STATE
    MACHINES
    WITH RAGEL
    Eurucamp, 2013
    Drew Neil
    @nelstrom
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

    View Slide

  2. Vimprint
    Turns keystrokes into plain English

    View Slide

  3. What is a
    state machine
    anyway?
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

    View Slide

  4. Chapter 3
    The Simplest
    Computers
    A Finite State Machine,
    also known as a Finite
    Automaton, is a drastically
    simplified model of a
    computer that is easy to
    understand, easy to
    reason about, and easy to
    implement in hardware
    or software.

    View Slide

  5. 1 2
    a
    a

    View Slide

  6. 1 2
    a
    a

    View Slide

  7. Chapter 3
    The Simplest
    Computers
    It’s possible to convert
    any regular expression
    into an equivalent
    Nondeterministic Finite
    Automaton -- every string
    matched by the regular
    expression is accepted
    by the NFA, and vice
    versa.

    View Slide

  8. What about
    Vim?
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

    View Slide

  9. View Slide

  10. Normal
    h, j, k, l

    View Slide

  11. Normal
    h, j, k, l
    Insert
    i, I, a, A

    any

    View Slide

  12. Normal
    Insert
    Visual
    CmdLine

    View Slide

  13. Meet Ragel
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

    View Slide

  14. Ragel state machines can not only
    recognize byte sequences as regular
    expression machines do, but can also
    execute code at arbitrary points in the
    recognition of a regular language.

    View Slide

  15. host
    %% inline ragel
    %%{
    multi-line ragel
    action { host };
    }%%
    host
    Birds-eye view of a ragel file

    View Slide

  16. $ ragel -h | grep "host language"
    host language:
    -C The host language is C (default)
    -D The host language is D
    -Z The host language is Go
    -J The host language is Java
    -R The host language is Ruby
    -A The host language is C#
    -O The host language is OCaml
    Supported host languages

    View Slide

  17. $ ragel -R source.rl
    Compile source.rl using Ruby as host language
    Producing source.rb as output
    $ ragel -V source.rl > graph.dot
    Generate a visualisation in .dot format
    Can be viewed with Graphviz
    Compiling Ragel

    View Slide

  18. %%{
    machine accepter;
    motion = [hjklbwe0];
    switch = [iIaAsSoO];
    escape = 27;
    input = (any - escape);
    insert_mode := (
    input*
    escape @{ fret; }
    );
    normal_mode := (
    motion |
    switch @{ fcall insert_mode; }
    )*;
    }%%
    class Accepter
    attr_accessor :data
    def initialize()
    %% write data;
    end
    def process(input)
    @data = input.unpack("c*")
    stack = []
    %% write init;
    %% write exec;
    return cs
    end
    end
    accepter.rl

    View Slide

  19. def process(input)
    @data = input.unpack("c*")
    stack = []
    begin
    " p ||= 0
    " pe ||= data.length
    " cs = accepter_start
    " top = 0
    end
    begin
    " _klen, _trans, _keys, _acts = nil
    " _goto_level = 0
    " _resume = 10
    " _eof_trans = 15
    " _again = 20
    # 150 more LOC like this...
    end
    return cs
    end
    "
    def process(input)
    @data = input.unpack("c*")
    stack = []
    %% write init;
    %% write exec;
    return cs
    end

    View Slide

  20. hellohello

    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

    View Slide

  21. h e l l o h e l l o ␛
    h
    move left
    e
    move to end of word
    l
    move right
    l
    move right
    o hello
    enter “hello” on a new line

    return to Normal mode

    View Slide

  22. Machine definition:
    = ;
    Ragel syntax
    Machine instantiation:
    := ;
    TOKENS
    PARSERS

    View Slide

  23. Machine definition:
    = ;
    Ragel syntax
    Machine instantiation:
    := ;
    # TOKENS
    motion = [hjklbwe0];
    switch = [iIaAsSoO];
    escape = 27;
    input = (any - escape);
    # PARSER
    normal_mode := (motion)*;

    View Slide

  24. Ragel actions
    expr >{action}
    entering action
    expr @{action}
    finishing action

    View Slide

  25. Ragel control flow
    fcall ;
    Push the target state and jump to the
    entry point defined by
    fret;
    Return to the target state of the
    transition on which the last fcall
    was made.
    PUSH
    POP

    View Slide

  26. %%{
    machine accepter;
    # ...
    normal_mode := (
    motion |
    switch @{ fcall insert_mode; }
    )*;
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl

    View Slide

  27. %%{
    machine accepter;
    # ...
    insert_mode := (
    input*
    escape @{ fret; }
    );
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl

    View Slide

  28. %%{
    machine accepter;
    motion = [hjklbwe0];
    switch = [iIaAsSoO];
    escape = 27;
    input = (any - escape);
    insert_mode := (
    input*
    escape @{ fret; }
    );
    normal_mode := (
    motion |
    switch @{ fcall insert_mode; }
    )*;
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl

    View Slide

  29. 2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF

    View Slide

  30. class Accepter
    def process(input)
    @data = input.unpack("c*")
    %% write init;
    %% write exec;
    return cs
    end
    def accept?(input)
    process(input) > 0
    end
    end
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/accepter.rl

    View Slide

  31. describe Accepter do
    it 'accepts hellohello' do
    assert Accepter.new.accept?("hellohello\e")
    end
    it 'rejects visual mode (for example)' do
    refute Accepter.new.accept?("viw")
    end
    end
    https://github.com/nelstrom/ragel-vim-demo/blob/master/test/accepter_test.rb

    View Slide

  32. Producing
    side-effects
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

    View Slide

  33. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 0
    cs = 2

    View Slide

  34. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 1
    cs = 2

    View Slide

  35. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 2
    cs = 2

    View Slide

  36. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 3
    cs = 2

    View Slide

  37. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 4
    cs = 2

    View Slide

  38. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 5
    cs = 1

    View Slide

  39. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 6
    cs = 1

    View Slide

  40. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 7
    cs = 1

    View Slide

  41. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 8
    cs = 1

    View Slide

  42. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 9
    cs = 1

    View Slide

  43. index
    character
    0 1 2 3 4 5 6 7 8 9 10
    h e l l o h e l l o
    Ragel’s cursor
    2
    IN
    1
    insert_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w'
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / push_insert_mode
    3
    27 / return
    DEF
    p = 10
    cs = 3

    View Slide

  44. %%{
    machine emitter;
    action H { @head = p }
    action T { @tail = p }
    # TOKENS
    motion = [hjklbwe0] >[email protected];
    switch = [iIaAsSoO] >[email protected];
    escape = 27 >[email protected];
    input = (any - escape) >[email protected];
    # ...
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/emitter.rl

    View Slide

  45. class Emitter
    attr_accessor :data
    # initialize
    # process
    def strokes
    @data[@[email protected]].pack('c*')
    end
    end
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/emitter.rl

    View Slide

  46. %%{
    machine emitter;
    motion = [hjklbwe0]
    >[email protected] @{ @events << {motion: strokes} };
    switch = [iIaAsSoO]
    >[email protected] @{ @events << {switch: strokes} };
    escape = 27
    >[email protected] @{ @events << {escape: ''} };
    input = (any - escape)
    >[email protected] @{ @events << {input: strokes} };
    # ...
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/emitter.rl

    View Slide

  47. describe Emitter do
    it 'accepts motions, switches, and insertions' do
    Emitter.new(events = []).process("hellohello\e")
    assert_equal [
    {:motion=>"h"},
    {:motion=>"e"},
    {:motion=>"l"},
    {:motion=>"l"},
    {:switch=>"o"},
    {:input=>"h"},
    {:input=>"e"},
    {:input=>"l"},
    {:input=>"l"},
    {:input=>"o"},
    {:escape=>""}
    ], events
    end
    end
    https://github.com/nelstrom/ragel-vim-demo/blob/master/test/emitter_test.rb

    View Slide

  48. Handling
    Visual mode
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

    View Slide

  49. h e l l
    Normal
    o hello
    Normal > Insert

    Normal
    Insertion

    View Slide

  50. h e l l
    Normal
    Normal > Visual

    Normal
    Selection
    b iw
    v

    View Slide

  51. h e l l
    Normal
    Normal > Visual

    Normal
    Change selection
    b iw
    v
    c hello
    Normal > Insert

    View Slide

  52. fcall ;
    Push the target state and jump to the
    entry point defined by
    fret;
    Return to the target state of the
    transition on which the last fcall
    was made.
    fnext ;
    Set the next state to be the entry point
    defined by label.
    PUSH
    POP
    SWITCH
    Ragel control flow

    View Slide

  53. %%{
    machine visual;
    action H { @head = p }
    action T { @tail = p }
    # ...
    start_visual = 'v'
    >[email protected] @{ @events << {start_visual: strokes} };
    normal_mode := (
    motion |
    start_visual @{ fcall visual_mode; }
    )*;
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/visual.rl

    View Slide

  54. %%{
    machine visual;
    action H { @head = p }
    action T { @tail = p }
    # ...
    text_object = ([ai][bBpstwW])
    >[email protected] @{ @events << {text_object: strokes} };
    v_switch = [sScC]
    >[email protected] @{ @events << {switch: strokes} };
    # ...
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/visual.rl

    View Slide

  55. %%{
    machine visual;
    # ...
    visual_mode := (
    ( motion | text_object )*
    (
    v_switch @{ fnext insert_mode; } |
    escape @{ fret; }
    )
    );
    }%%
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/visual.rl

    View Slide

  56. it 'accepts vhjiwhelloel' do
    Visual.new(events = []).process("vhjiwchello\eel")
    assert_equal [
    {:start_visual=>"v"},
    {:motion=>"h"},
    {:motion=>"j"},
    {:text_object=>"iw"},
    {:switch=>"c"},
    {:input=>"h"},
    {:input=>"e"},
    {:input=>"l"},
    {:input=>"l"},
    {:input=>"o"},
    {:escape=>""},
    {:motion=>"e"},
    {:motion=>"l"}
    ], events
    end
    https://github.com/nelstrom/ragel-vim-demo/blob/master/test/visual_test.rb

    View Slide

  57. 4
    IN
    1
    insert_mode
    2
    visual_mode
    normal_mode
    '0', 'b', 'e', 'h', 'j'..'l', 'w' / H, T, 6:33
    'A', 'I', 'O', 'S', 'a', 'i', 'o', 's' / H, T, 7:33, 30:13
    'v' / H, T, 11:40, 31:19
    5
    6
    27 / H, T, 8:33, 17:13
    DEF / H, T, 9:33
    27 / H, T, 8:33, 24:15
    '0', 'b', 'e', 'h', 'j'..'l', 'w' / H, T, 6:33
    'C', 'S', 'c', 's' / H, T, 13:40, 23:17
    3
    'a', 'i' / H
    'B', 'W', 'b', 'p', 's'..'t', 'w' / T, 12:40

    View Slide

  58. State charts
    for
    debugging
    “Charles Babbage’s Difference Engine No. 2” by Ricardo Ferreira

    View Slide

  59. ["x]x Delete [count] characters under and after the cursor
    [into register x]. Does the same as "dl".
    :help x
    x cut 1 character, save to default register
    2x cut 2 characters, save to default register
    “ax cut 1 character, save to register a

    View Slide

  60. 2”ax cut 2 characters, save to register a
    “a2x cut 2 characters, save to register a
    3“a2x cut 6 characters, save to register a
    ["x]x Delete [count] characters under and after the cursor
    [into register x]. Does the same as "dl".
    :help x

    View Slide

  61. count = [1-9];
    register = '"' [a-z];
    cut_command =
    count?
    register?
    count?
    'x';
    normal_mode := (cut_command)*;
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/faulty_cut.rl

    View Slide

  62. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"' '1'..'9'
    'x'

    View Slide

  63. count = [1-9];
    register = '"' [a-z];
    cut_command =
    count?
    register?
    count?
    'x';
    normal_mode := (cut_command)*;
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/faulty_cut.rl

    View Slide

  64. count = [1-9];
    register = '"' [a-z];
    cut_command =
    (
    count?
    register
    )?
    count?
    'x';
    normal_mode := (cut_command)*;
    https://github.com/nelstrom/ragel-vim-demo/blob/master/lib/cut.rl

    View Slide

  65. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"'
    'x'

    View Slide

  66. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"'
    'x'
    x

    View Slide

  67. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"'
    'x'
    2x

    View Slide

  68. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"'
    'x'
    “a2x

    View Slide

  69. 5
    IN
    normal_mode
    1
    '"'
    4
    '1'..'9'
    'x'
    2
    'a'..'z'
    3
    '1'..'9'
    'x'
    'x'
    '"'
    'x'
    3“a2x

    View Slide

  70. Wrapping up
    all code from these slides
    github.com/nelstrom/ragel-vim-demo
    github.com/nelstrom/vimprint
    a parser for Vim commands
    #pairwithme

    View Slide