$30 off During Our Annual Pro Sale. View Details »

Modeling State Transitions with Specification-based Random Testing

Modeling State Transitions with Specification-based Random Testing

What if you thought about tests only in terms of properties and counterexamples? Properties that may assert failures and/or successes. Counterexamples to a set of properties that can “shrink” to smaller failures and be better reasoned about. Properties and counterexamples are the foundation of QuickCheck, a tool to generate tests over concurrent and non-deterministic code.

The difficult component of most real-world approaches to generative testing is understanding the bounds and requirements surrounding a problem/feature/application. Using Erlang’s QuickCheck implementation, we’ll walk through an example which models a continuous, side-effecting, hashtree-based synchronization mechanism, called Active Anti-Entropy (AAE), as an abstract state machine. By being able to query the (Erlang) process state and compare it against our model state, we can assure that our system matches its intended specification – which is a whole lot more important than tests being green.

Zeeshan Lakhani

July 15, 2015
Tweet

More Decks by Zeeshan Lakhani

Other Decks in Programming

Transcript

  1. Modeling State Transitions with Specification-based
    Random Testing
    Zeeshan Lakhani
    Software Engineer at Basho Technologies,Inc | Founder/Organizer Papers We Love
    @zeeshanlakhani
    07-15-2015 (LambdaJam)
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 1 / 39

    View Slide

  2. a beginning
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 2 / 39

    View Slide

  3. a starter example1
    membership in a list
    1Testing Java with QuickCheck [Benac, Fredlund] http://bit.ly/1UYejBE
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 3 / 39

    View Slide

  4. an initial state
    initial_state() -> [].

    View Slide

  5. a group of function calls
    -spec add(list(), non_neg_integer()) -> list().
    add(AList, N) ->
    [N|AList].
    -spec is_member(list(), non_neg_integer()) -> boolean().
    is_member(S, N) ->
    lists:member(N, S).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 5 / 39

    View Slide

  6. a few terms2, 3, 4
    commands - symbolic calls to run during test sequences
    symbolic variables - generated during test generation
    dynamic values - generated during test execution
    next_state - operates during test generation and execution
    preconditions - only depend on the model, not execution time values
    postcondition - predicate called during test execution (that must hold)
    with the dynamic state before the call
    aggregate - Collects a list of values in each test, and shows the
    distribution of list elements
    2eqc_statem notes [Stone] http://bit.ly/1dZ6qu2
    3Testing Erlang Concurrency with QuickCheck [Cao] http://bit.ly/1UZQVnb
    4Ulf Norell’s notes
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 6 / 39

    View Slide

  7. a group of commands
    -spec add_command(S :: eqc_statem:symbolic_state()) ->
    eqc_gen:gen(eqc_statem:call()).
    add_command(S) ->
    {call, ?MODULE, add, [S, nat()]}.
    -spec is_member_command(
    S :: eqc_statem:symbolic_state()) ->
    eqc_gen:gen(eqc_statem:call()).
    is_member_command(S) ->
    {call, ?MODULE, is_member, [S, nat()]}.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 7 / 39

    View Slide

  8. a group of preconditions
    -spec add_pre(S :: eqc_statem:symbolic_state(),
    Args :: [term()]) -> boolean().
    add_pre(S, _Args) ->
    S /= undefined.
    -spec is_member_pre(S :: eqc_statem:symbolic_state(),
    Args :: [term()]) -> boolean().
    is_member_pre(S, _Args) ->
    S /= undefined.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 8 / 39

    View Slide

  9. a group of next states
    -spec add_next(S :: eqc_statem:symbolic_state(),
    V :: eqc_statem:var(),
    Args :: [term()])
    -> eqc_statem:symbolic_state().
    add_next(S, _Value, [_, N]) ->
    [N|S].
    -spec is_member_next(S :: eqc_statem:symbolic_state(),
    V :: eqc_statem:var(),
    Args :: [term()])
    -> eqc_statem:symbolic_state().
    is_member_next(S, _Value, _Args) ->
    S.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 9 / 39

    View Slide

  10. a group of postconditions
    -spec add_post(S :: eqc_statem:dynamic_state(),
    Args :: [term()],
    R :: term()) -> true | term().
    add_post(S, [_, N], Res) ->
    [N|S] =:= Res.
    -spec is_member_post(S :: eqc_statem:dynamic_state(),
    Args :: [term()],
    R :: term()) -> true | term().
    is_member_post(S, [_, N], Res) ->
    Res =:= lists:member(N, S).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 10 / 39

    View Slide

  11. a set of invariants
    -spec invariant(eqc_statem:dynamic_state()) -> boolean().
    invariant(S) when length(S) >= 0 ->
    true;
    invariant(S) when length(S) > 0 ->
    FirstNum = hd(S),
    is_number(FirstNum) andalso FirstNum >= 0;
    invariant(_) ->
    false.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 11 / 39

    View Slide

  12. a ?FORALL
    prop_test() ->
    ?FORALL(Cmds,commands(?MODULE),
    aggregate(
    command_names(Cmds),
    begin
    {H, S, Res} = run_commands(?MODULE, Cmds),
    pretty_commands(?MODULE,
    Cmds,
    {H, S, Res},
    Res =:= ok)
    end)).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 12 / 39

    View Slide

  13. a fake failure
    .Failed! After 2 tests.
    [{set,{var,1},{call,stateful_sm,add,[[],0]}},
    {set,{var,2},{call,stateful_sm,add,[[0],0]}},
    {set,{var,3},{call,stateful_sm,is_member,[[0,0],0]}},
    {set,{var,4},{call,stateful_sm,add,[[0,0],0]}},
    {set,{var,5},{call,stateful_sm,is_member,[[0,0,0],0]}}]
    stateful_sm:add([], 0) -> [0]
    stateful_sm:add([0], 0) -> [0, 0]
    stateful_sm:is_member([0, 0], 0) -> true
    Reason: false
    Shrinking xxx...(3 times)
    [{set,{var,1},{call,stateful_sm,add,[[0,0],0]}}]
    stateful_sm:add([0, 0], 0) -> [0, 0, 0]
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 13 / 39

    View Slide

  14. a search system
    curl "$RIAK_HOST/search/query/lambdajam?wt=json&q=name_s:lj_*"
    distributed across a cluster of nodes5
    5Exploiting Parallelism in Query Processing for Web Document Search Using
    Shared-Memory and Cluster-Based Architectures [Aboutabl] http://bit.ly/1f2JElM
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 14 / 39

    View Slide

  15. a way to index key-value data
    A liveness property guarantees that “something good eventually
    happens;” for example, all requests eventually receive a response.6
    how it does it7
    6Eventual Consistency Today [Bailis, Ghodsi] http://bit.ly/1K3Lgcw
    7Riak Search 2.0 [Redmond] http://bit.ly/1HvOrE7
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 15 / 39

    View Slide

  16. a mechanism - active anti-entropy
    Merkle Treesa
    aA space- and time-efficient Implementation of the Merkle Tree Traversal
    Algorithm [Knecht, Nicola] http://bit.ly/1GjNLj7
    a complete binary tree with an n-bit value associated with each node.
    Each internal node value is the result of a hash of the node values of
    its children (n is the number of bits returned by the hash function).
    Goals a
    aActive Anti-Antropy [Blomstedt]
    disk-based -> issues with in-memory trees containing billions of keys
    persistent -> build a tree once
    real-time updates - liveness
    non-blocking - can’t affect incoming write-rate
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 16 / 39

    View Slide

  17. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 17 / 39

    View Slide

  18. a difference
    riak search / yokozuna uses lucene’s termsenum8 to iterate over
    entropy data containing bucket/key & object hash
    riak search / yokozuna trees always repair entries remote_missing. . .
    key-value data is canonical
    8http://bit.ly/1RyHewY
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 18 / 39

    View Slide

  19. a problem
    riak_kv_vnode:actual_put:1460 vnode-kv:
    {r_object,<<"fruit_aae">>,<<"testfor spaces 342">>...
    and
    [info] <0.5785.0>@yz_solr:get_pairs:421 yz:
    [{struct,[{<<"vsn">>,<<"2">>},
    {<<"riak_bucket_type">>,<<"default">>},
    {<<"riak_bucket_name">>,<<"fruit_aae">>},
    {<<"riak_key">>,<<"testfor spaces 342">>}...
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 19 / 39

    View Slide

  20. a collection of generators
    vclock() ->
    ?LET(VclockSym, vclock_sym(), eval(VclockSym)).
    vclock_sym() ->
    ?LAZY(
    oneof([
    {call, vclock, fresh, []},
    ?LETSHRINK([Clock], [vclock_sym()],
    {call, ?MODULE, increment,
    [noshrink(binary(4)), nat(), Clock]})
    ])).
    not_empty(G) ->
    ?SUCHTHAT(X, G, X /= [] andalso X /= <<>>).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 20 / 39

    View Slide

  21. increment(Actor, Count, Vclock) ->
    lists:foldl(
    fun vclock:increment/2,
    Vclock,
    lists:duplicate(Count, Actor)).
    riak_object() ->
    ?LET({{Bucket, Key}, Vclock, Value},
    {bkey(), vclock(), binary()},
    riak_object:set_vclock(
    riak_object:new(Bucket, Key, Value),
    Vclock)).
    bkey() ->
    {non_blank_string(), %% bucket
    non_blank_string()}. %% key
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 21 / 39

    View Slide

  22. a generated riak object
    {r_object,<<"|plVWx&F">>,<<"?#sjiGS|">>,
    [{r_content,{dict,0,16,16,8,80,48,
    {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
    [],[]},
    {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
    [],[]}}},
    <<"t+">>}],
    [],
    {dict,1,16,16,8,80,48,
    {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
    {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
    [[clean|true]],
    []}}},
    undefined}
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 22 / 39

    View Slide

  23. a state record9
    -record(state, {yz_idx_tree,
    kv_idx_tree,
    yz_idx_objects = dict:new(),
    kv_idx_objects = dict:new(),
    trees_updated = false,
    both = []}).
    %% Initialize State
    initial_state() ->
    #state{}.
    9eqc_util.erl (http://bit.ly/1K68zmd) and yz_index_hashtree_eqc.erl
    (http://bit.ly/1INKmwQ) [Daily, Lakhani]
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 23 / 39

    View Slide

  24. a couple of tree processes begin
    we start a 1 riak key-value idx-tree process and a 1 yokozuna idx-tree
    process
    this happens per vnode (each vnode is responsible for a partition)
    each process contains multiple hashtrees due to preflist overlap
    some nodes contain data not part of preflist, otherwise always
    divergent10
    10Active Anti-Antropy [Blomstedt]
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 24 / 39

    View Slide

  25. an insert into a kv-tree
    insert_kv_tree_command(S) ->
    {call, ?MODULE, insert_kv_tree,
    [insert_method(), eqc_util:riak_object(),
    S#state.kv_idx_tree]}.
    insert_kv_tree_next(S, _V, [_, RObj, _]) ->
    {ok, TreeData} = dict:find(?TEST_INDEX_N,
    S#state.kv_idx_objects),
    S#state{kv_idx_objects=dict:store(
    ?TEST_INDEX_N,
    set_treedata(RObj, TreeData),
    S#state.kv_idx_objects),
    trees_updated=false}.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 25 / 39

    View Slide

  26. insert_kv_tree_pre(S, _Args) ->
    S#state.kv_idx_tree /= undefined.
    insert_kv_tree_post(_S, _Args, _Res) ->
    true.
    insert_kv_tree(Method, RObj, {ok, TreePid}) ->
    {Bucket, Key} = eqc_util:get_bkey_from_object(RObj),
    Items = [{object, {Bucket, Key}, RObj}],
    case Method of
    sync ->
    riak_kv_index_hashtree:insert(
    Items, [], TreePid);
    async ->
    riak_kv_index_hashtree:async_insert(
    Items, [], TreePid)
    end.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 26 / 39

    View Slide

  27. a number of inserts and dirty segments
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 27 / 39

    View Slide

  28. an insert into a yz-tree
    insert_yz_tree_command(S) ->
    {call, ?MODULE, insert_yz_tree,
    [insert_method(), eqc_util:riak_object(),
    S#state.yz_idx_tree]}.
    insert_yz_tree_next(S, _V, [_, RObj, _]) ->
    {ok, TreeData} = dict:find(?TEST_INDEX_N,
    S#state.yz_idx_objects),
    S#state{yz_idx_objects=dict:store(
    ?TEST_INDEX_N,
    set_treedata(RObj, TreeData),
    S#state.yz_idx_objects),
    trees_updated=false}.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 28 / 39

    View Slide

  29. insert_yz_tree(Method, RObj, {ok, TreePid}) ->
    BKey = eqc_util:get_bkey_from_object(RObj),
    yz_index_hashtree:insert(
    Method, ?TEST_INDEX_N, BKey,
    yz_kv:hash_object(RObj), TreePid, []).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 29 / 39

    View Slide

  30. an insert into both trees
    insert_both(Method, RObj, YZOkTree, KVOkTree) ->
    {insert_yz_tree(Method, RObj, YZOkTree),
    insert_kv_tree(Method, RObj, KVOkTree)}.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 30 / 39

    View Slide

  31. an update
    update_command(S) ->
    {call, ?MODULE, update,
    [S#state.yz_idx_tree, S#state.kv_idx_tree]}.
    update_pre(S, _Args) ->
    S#state.yz_idx_tree /= undefined andalso
    S#state.kv_idx_tree /= undefined.
    update_next(S, _Value, _Args) ->
    S#state{trees_updated=true}.
    update_post(S, _Args, _Res) ->
    ...
    eq(ModelKVKeyCount, RealKVKeyCount) and
    eq(ModelYZKeyCount, RealYZKeyCount).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 31 / 39

    View Slide

  32. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 32 / 39

    View Slide

  33. a comparison of trees
    compare_command(S) ->
    {call, ?MODULE, compare, [S#state.yz_idx_tree,
    S#state.kv_idx_tree]}.
    compare({ok, YZTreePid}, {ok, KVTreePid}) ->
    Remote =
    fun(get_bucket, {L, B}) ->
    riak_kv_index_hashtree:exchange_bucket(?TEST_INDEX_N,
    L, B,
    KVTreePid);
    (key_hashes, Segment) ->
    riak_kv_index_hashtree:exchange_segment(?TEST_INDEX_N,
    Segment,
    KVTreePid);
    (_, _) -> ok
    end,
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 33 / 39

    View Slide

  34. AccFun =
    fun(KeyDiff, Count) ->
    lists:foldl(fun(Diff, InnerCount) ->
    case repair(0, Diff) of
    full_repair -> InnerCount + 1;
    _ -> InnerCount
    end
    end, Count, KeyDiff)
    end,
    yz_index_hashtree:compare(?TEST_INDEX_N, Remote,
    AccFun, 0, YZTreePid).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 34 / 39

    View Slide

  35. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 35 / 39

    View Slide

  36. a key precondition
    compare_pre(S, _Args) ->
    S#state.yz_idx_tree /= undefined andalso
    S#state.kv_idx_tree /= undefined andalso
    S#state.trees_updated.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 36 / 39

    View Slide

  37. a key postcondition
    compare_post(S, _Args, Res) ->
    YZTreeData = dict:fetch(?TEST_INDEX_N,
    S#state.yz_idx_objects),
    KVTreeData = dict:fetch(?TEST_INDEX_N,
    S#state.kv_idx_objects),
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 37 / 39

    View Slide

  38. LeftDiff = dict:fold(fun(BKey, Hash, Count) ->
    case dict:find(BKey, KVTreeData) of
    {ok, Hash} -> Count;
    {ok, _OtherHash} -> Count;
    error -> Count+1
    end
    end, 0, YZTreeData),
    RightDiff = dict:fold(fun(BKey, Hash, Count) ->
    case dict:find(BKey, YZTreeData) of
    {ok, Hash} -> Count;
    {ok, _OtherHash} -> Count;
    error -> Count+1
    end
    end, LeftDiff, KVTreeData),
    eq(RightDiff, Res).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 38 / 39

    View Slide

  39. a distribution of commands
    .................
    OK, passed 100 tests
    22.3% {yz_index_hashtree_eqc,insert_yz_tree,3}
    20.7% {yz_index_hashtree_eqc,insert_kv_tree,3}
    18.1% {yz_index_hashtree_eqc,insert_both,4}
    17.7% {yz_index_hashtree_eqc,update,2}
    9.2% {yz_index_hashtree_eqc,start_kv_tree,0}
    8.8% {yz_index_hashtree_eqc,start_yz_tree,0}
    3.3% {yz_index_hashtree_eqc,compare,2}
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 39 / 39

    View Slide