Modeling State Transitions with Specification-based Random Testing

What if you thought about tests only in terms of properties and counterexamples? Properties that may assert failures and/or successes. Counterexamples to a set of properties that can “shrink” to smaller failures and be better reasoned about. Properties and counterexamples are the foundation of QuickCheck, a tool to generate tests over concurrent and non-deterministic code.

The difficult component of most real-world approaches to generative testing is understanding the bounds and requirements surrounding a problem/feature/application. Using Erlang’s QuickCheck implementation, we’ll walk through an example which models a continuous, side-effecting, hashtree-based synchronization mechanism, called Active Anti-Entropy (AAE), as an abstract state machine. By being able to query the (Erlang) process state and compare it against our model state, we can assure that our system matches its intended specification – which is a whole lot more important than tests being green.

Transcript

1. Modeling State Transitions with Specification-based
   Random Testing
   Zeeshan Lakhani
   Software Engineer at Basho Technologies,Inc | Founder/Organizer Papers We Love
   @zeeshanlakhani
   07-15-2015 (LambdaJam)
   Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
   07-15-2015 (LambdaJam) 1 / 39
   

   View Slide

2. a beginning
   Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
   07-15-2015 (LambdaJam) 2 / 39
   

   View Slide

3. a starter example1
   membership in a list
   1Testing Java with QuickCheck [Benac, Fredlund] http://bit.ly/1UYejBE
   Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
   07-15-2015 (LambdaJam) 3 / 39
   

   View Slide

4. an initial state
   initial_state() -> [].
   

   View Slide

5. a group of function calls
   -spec add(list(), non_neg_integer()) -> list().
   add(AList, N) ->
   [N|AList].
   -spec is_member(list(), non_neg_integer()) -> boolean().
   is_member(S, N) ->
   lists:member(N, S).
   Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
   07-15-2015 (LambdaJam) 5 / 39
   

   View Slide

6. a few terms2, 3, 4
   commands - symbolic calls to run during test sequences
   symbolic variables - generated during test generation
   dynamic values - generated during test execution
   next_state - operates during test generation and execution
   preconditions - only depend on the model, not execution time values
   postcondition - predicate called during test execution (that must hold)
   with the dynamic state before the call
   aggregate - Collects a list of values in each test, and shows the
   distribution of list elements
   2eqc_statem notes [Stone] http://bit.ly/1dZ6qu2
   3Testing Erlang Concurrency with QuickCheck [Cao] http://bit.ly/1UZQVnb
   4Ulf Norell’s notes
   Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
   07-15-2015 (LambdaJam) 6 / 39
   

   View Slide

7. a group of commands
   -spec add_command(S :: eqc_statem:symbolic_state()) ->
   eqc_gen:gen(eqc_statem:call()).
   add_command(S) ->
   {call, ?MODULE, add, [S, nat()]}.
   -spec is_member_command(
   S :: eqc_statem:symbolic_state()) ->
   eqc_gen:gen(eqc_statem:call()).
   is_member_command(S) ->
   {call, ?MODULE, is_member, [S, nat()]}.
   Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
   07-15-2015 (LambdaJam) 7 / 39
   

   View Slide

8. a group of preconditions
   -spec add_pre(S :: eqc_statem:symbolic_state(),
   Args :: [term()]) -> boolean().
   add_pre(S, _Args) ->
   S /= undefined.
   -spec is_member_pre(S :: eqc_statem:symbolic_state(),
   Args :: [term()]) -> boolean().
   is_member_pre(S, _Args) ->
   S /= undefined.
   Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
   07-15-2015 (LambdaJam) 8 / 39
   

   View Slide

9. a group of next states
   -spec add_next(S :: eqc_statem:symbolic_state(),
   V :: eqc_statem:var(),
   Args :: [term()])
   -> eqc_statem:symbolic_state().
   add_next(S, _Value, [_, N]) ->
   [N|S].
   -spec is_member_next(S :: eqc_statem:symbolic_state(),
   V :: eqc_statem:var(),
   Args :: [term()])
   -> eqc_statem:symbolic_state().
   is_member_next(S, _Value, _Args) ->
   S.
   Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
   07-15-2015 (LambdaJam) 9 / 39
   

   View Slide

10. a group of postconditions
    -spec add_post(S :: eqc_statem:dynamic_state(),
    Args :: [term()],
    R :: term()) -> true | term().
    add_post(S, [_, N], Res) ->
    [N|S] =:= Res.
    -spec is_member_post(S :: eqc_statem:dynamic_state(),
    Args :: [term()],
    R :: term()) -> true | term().
    is_member_post(S, [_, N], Res) ->
    Res =:= lists:member(N, S).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 10 / 39
    

    View Slide

11. a set of invariants
    -spec invariant(eqc_statem:dynamic_state()) -> boolean().
    invariant(S) when length(S) >= 0 ->
    true;
    invariant(S) when length(S) > 0 ->
    FirstNum = hd(S),
    is_number(FirstNum) andalso FirstNum >= 0;
    invariant(_) ->
    false.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 11 / 39
    

    View Slide

12. a ?FORALL
    prop_test() ->
    ?FORALL(Cmds,commands(?MODULE),
    aggregate(
    command_names(Cmds),
    begin
    {H, S, Res} = run_commands(?MODULE, Cmds),
    pretty_commands(?MODULE,
    Cmds,
    {H, S, Res},
    Res =:= ok)
    end)).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 12 / 39
    

    View Slide

13. a fake failure
    .Failed! After 2 tests.
    [{set,{var,1},{call,stateful_sm,add,[[],0]}},
    {set,{var,2},{call,stateful_sm,add,[[0],0]}},
    {set,{var,3},{call,stateful_sm,is_member,[[0,0],0]}},
    {set,{var,4},{call,stateful_sm,add,[[0,0],0]}},
    {set,{var,5},{call,stateful_sm,is_member,[[0,0,0],0]}}]
    stateful_sm:add([], 0) -> [0]
    stateful_sm:add([0], 0) -> [0, 0]
    stateful_sm:is_member([0, 0], 0) -> true
    Reason: false
    Shrinking xxx...(3 times)
    [{set,{var,1},{call,stateful_sm,add,[[0,0],0]}}]
    stateful_sm:add([0, 0], 0) -> [0, 0, 0]
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 13 / 39
    

    View Slide

14. a search system
    curl "$RIAK_HOST/search/query/lambdajam?wt=json&q=name_s:lj_*"
    distributed across a cluster of nodes5
    5Exploiting Parallelism in Query Processing for Web Document Search Using
    Shared-Memory and Cluster-Based Architectures [Aboutabl] http://bit.ly/1f2JElM
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 14 / 39
    

    View Slide

15. a way to index key-value data
    A liveness property guarantees that “something good eventually
    happens;” for example, all requests eventually receive a response.6
    how it does it7
    6Eventual Consistency Today [Bailis, Ghodsi] http://bit.ly/1K3Lgcw
    7Riak Search 2.0 [Redmond] http://bit.ly/1HvOrE7
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 15 / 39
    

    View Slide

16. a mechanism - active anti-entropy
    Merkle Treesa
    aA space- and time-efficient Implementation of the Merkle Tree Traversal
    Algorithm [Knecht, Nicola] http://bit.ly/1GjNLj7
    a complete binary tree with an n-bit value associated with each node.
    Each internal node value is the result of a hash of the node values of
    its children (n is the number of bits returned by the hash function).
    Goals a
    aActive Anti-Antropy [Blomstedt]
    disk-based -> issues with in-memory trees containing billions of keys
    persistent -> build a tree once
    real-time updates - liveness
    non-blocking - can’t affect incoming write-rate
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 16 / 39
    

    View Slide

17. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 17 / 39
    

    View Slide

18. a difference
    riak search / yokozuna uses lucene’s termsenum8 to iterate over
    entropy data containing bucket/key & object hash
    riak search / yokozuna trees always repair entries remote_missing. . .
    key-value data is canonical
    8http://bit.ly/1RyHewY
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 18 / 39
    

    View Slide

19. a problem
    riak_kv_vnode:actual_put:1460 vnode-kv:
    {r_object,<<"fruit_aae">>,<<"testfor spaces 342">>...
    and
    [info] <0.5785.0>@yz_solr:get_pairs:421 yz:
    [{struct,[{<<"vsn">>,<<"2">>},
    {<<"riak_bucket_type">>,<<"default">>},
    {<<"riak_bucket_name">>,<<"fruit_aae">>},
    {<<"riak_key">>,<<"testfor spaces 342">>}...
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 19 / 39
    

    View Slide

20. a collection of generators
    vclock() ->
    ?LET(VclockSym, vclock_sym(), eval(VclockSym)).
    vclock_sym() ->
    ?LAZY(
    oneof([
    {call, vclock, fresh, []},
    ?LETSHRINK([Clock], [vclock_sym()],
    {call, ?MODULE, increment,
    [noshrink(binary(4)), nat(), Clock]})
    ])).
    not_empty(G) ->
    ?SUCHTHAT(X, G, X /= [] andalso X /= <<>>).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 20 / 39
    

    View Slide

21. increment(Actor, Count, Vclock) ->
    lists:foldl(
    fun vclock:increment/2,
    Vclock,
    lists:duplicate(Count, Actor)).
    riak_object() ->
    ?LET({{Bucket, Key}, Vclock, Value},
    {bkey(), vclock(), binary()},
    riak_object:set_vclock(
    riak_object:new(Bucket, Key, Value),
    Vclock)).
    bkey() ->
    {non_blank_string(), %% bucket
    non_blank_string()}. %% key
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 21 / 39
    

    View Slide

22. a generated riak object
    {r_object,<<"|plVWx&F">>,<<"?#sjiGS|">>,
    [{r_content,{dict,0,16,16,8,80,48,
    {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
    [],[]},
    {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
    [],[]}}},
    <<"t+">>}],
    [],
    {dict,1,16,16,8,80,48,
    {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
    {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
    [[clean|true]],
    []}}},
    undefined}
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 22 / 39
    

    View Slide

23. a state record9
    -record(state, {yz_idx_tree,
    kv_idx_tree,
    yz_idx_objects = dict:new(),
    kv_idx_objects = dict:new(),
    trees_updated = false,
    both = []}).
    %% Initialize State
    initial_state() ->
    #state{}.
    9eqc_util.erl (http://bit.ly/1K68zmd) and yz_index_hashtree_eqc.erl
    (http://bit.ly/1INKmwQ) [Daily, Lakhani]
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 23 / 39
    

    View Slide

24. a couple of tree processes begin
    we start a 1 riak key-value idx-tree process and a 1 yokozuna idx-tree
    process
    this happens per vnode (each vnode is responsible for a partition)
    each process contains multiple hashtrees due to preflist overlap
    some nodes contain data not part of preflist, otherwise always
    divergent10
    10Active Anti-Antropy [Blomstedt]
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 24 / 39
    

    View Slide

25. an insert into a kv-tree
    insert_kv_tree_command(S) ->
    {call, ?MODULE, insert_kv_tree,
    [insert_method(), eqc_util:riak_object(),
    S#state.kv_idx_tree]}.
    insert_kv_tree_next(S, _V, [_, RObj, _]) ->
    {ok, TreeData} = dict:find(?TEST_INDEX_N,
    S#state.kv_idx_objects),
    S#state{kv_idx_objects=dict:store(
    ?TEST_INDEX_N,
    set_treedata(RObj, TreeData),
    S#state.kv_idx_objects),
    trees_updated=false}.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 25 / 39
    

    View Slide

26. insert_kv_tree_pre(S, _Args) ->
    S#state.kv_idx_tree /= undefined.
    insert_kv_tree_post(_S, _Args, _Res) ->
    true.
    insert_kv_tree(Method, RObj, {ok, TreePid}) ->
    {Bucket, Key} = eqc_util:get_bkey_from_object(RObj),
    Items = [{object, {Bucket, Key}, RObj}],
    case Method of
    sync ->
    riak_kv_index_hashtree:insert(
    Items, [], TreePid);
    async ->
    riak_kv_index_hashtree:async_insert(
    Items, [], TreePid)
    end.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 26 / 39
    

    View Slide

27. a number of inserts and dirty segments
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 27 / 39
    

    View Slide

28. an insert into a yz-tree
    insert_yz_tree_command(S) ->
    {call, ?MODULE, insert_yz_tree,
    [insert_method(), eqc_util:riak_object(),
    S#state.yz_idx_tree]}.
    insert_yz_tree_next(S, _V, [_, RObj, _]) ->
    {ok, TreeData} = dict:find(?TEST_INDEX_N,
    S#state.yz_idx_objects),
    S#state{yz_idx_objects=dict:store(
    ?TEST_INDEX_N,
    set_treedata(RObj, TreeData),
    S#state.yz_idx_objects),
    trees_updated=false}.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 28 / 39
    

    View Slide

29. insert_yz_tree(Method, RObj, {ok, TreePid}) ->
    BKey = eqc_util:get_bkey_from_object(RObj),
    yz_index_hashtree:insert(
    Method, ?TEST_INDEX_N, BKey,
    yz_kv:hash_object(RObj), TreePid, []).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 29 / 39
    

    View Slide

30. an insert into both trees
    insert_both(Method, RObj, YZOkTree, KVOkTree) ->
    {insert_yz_tree(Method, RObj, YZOkTree),
    insert_kv_tree(Method, RObj, KVOkTree)}.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 30 / 39
    

    View Slide

31. an update
    update_command(S) ->
    {call, ?MODULE, update,
    [S#state.yz_idx_tree, S#state.kv_idx_tree]}.
    update_pre(S, _Args) ->
    S#state.yz_idx_tree /= undefined andalso
    S#state.kv_idx_tree /= undefined.
    update_next(S, _Value, _Args) ->
    S#state{trees_updated=true}.
    update_post(S, _Args, _Res) ->
    ...
    eq(ModelKVKeyCount, RealKVKeyCount) and
    eq(ModelYZKeyCount, RealYZKeyCount).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 31 / 39
    

    View Slide

32. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 32 / 39
    

    View Slide

33. a comparison of trees
    compare_command(S) ->
    {call, ?MODULE, compare, [S#state.yz_idx_tree,
    S#state.kv_idx_tree]}.
    compare({ok, YZTreePid}, {ok, KVTreePid}) ->
    Remote =
    fun(get_bucket, {L, B}) ->
    riak_kv_index_hashtree:exchange_bucket(?TEST_INDEX_N,
    L, B,
    KVTreePid);
    (key_hashes, Segment) ->
    riak_kv_index_hashtree:exchange_segment(?TEST_INDEX_N,
    Segment,
    KVTreePid);
    (_, _) -> ok
    end,
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 33 / 39
    

    View Slide

34. AccFun =
    fun(KeyDiff, Count) ->
    lists:foldl(fun(Diff, InnerCount) ->
    case repair(0, Diff) of
    full_repair -> InnerCount + 1;
    _ -> InnerCount
    end
    end, Count, KeyDiff)
    end,
    yz_index_hashtree:compare(?TEST_INDEX_N, Remote,
    AccFun, 0, YZTreePid).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 34 / 39
    

    View Slide

35. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 35 / 39
    

    View Slide

36. a key precondition
    compare_pre(S, _Args) ->
    S#state.yz_idx_tree /= undefined andalso
    S#state.kv_idx_tree /= undefined andalso
    S#state.trees_updated.
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 36 / 39
    

    View Slide

37. a key postcondition
    compare_post(S, _Args, Res) ->
    YZTreeData = dict:fetch(?TEST_INDEX_N,
    S#state.yz_idx_objects),
    KVTreeData = dict:fetch(?TEST_INDEX_N,
    S#state.kv_idx_objects),
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 37 / 39
    

    View Slide

38. LeftDiff = dict:fold(fun(BKey, Hash, Count) ->
    case dict:find(BKey, KVTreeData) of
    {ok, Hash} -> Count;
    {ok, _OtherHash} -> Count;
    error -> Count+1
    end
    end, 0, YZTreeData),
    RightDiff = dict:fold(fun(BKey, Hash, Count) ->
    case dict:find(BKey, YZTreeData) of
    {ok, Hash} -> Count;
    {ok, _OtherHash} -> Count;
    error -> Count+1
    end
    end, LeftDiff, KVTreeData),
    eq(RightDiff, Res).
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 38 / 39
    

    View Slide

39. a distribution of commands
    .................
    OK, passed 100 tests
    22.3% {yz_index_hashtree_eqc,insert_yz_tree,3}
    20.7% {yz_index_hashtree_eqc,insert_kv_tree,3}
    18.1% {yz_index_hashtree_eqc,insert_both,4}
    17.7% {yz_index_hashtree_eqc,update,2}
    9.2% {yz_index_hashtree_eqc,start_kv_tree,0}
    8.8% {yz_index_hashtree_eqc,start_yz_tree,0}
    3.3% {yz_index_hashtree_eqc,compare,2}
    Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing
    07-15-2015 (LambdaJam) 39 / 39
    

    View Slide