Modeling State Transitions with Specification-based Random Testing

What if you thought about tests only in terms of properties and counterexamples? Properties that may assert failures and/or successes. Counterexamples to a set of properties that can “shrink” to smaller failures and be better reasoned about. Properties and counterexamples are the foundation of QuickCheck, a tool to generate tests over concurrent and non-deterministic code.

The difficult component of most real-world approaches to generative testing is understanding the bounds and requirements surrounding a problem/feature/application. Using Erlang’s QuickCheck implementation, we’ll walk through an example which models a continuous, side-effecting, hashtree-based synchronization mechanism, called Active Anti-Entropy (AAE), as an abstract state machine. By being able to query the (Erlang) process state and compare it against our model state, we can assure that our system matches its intended specification – which is a whole lot more important than tests being green.

Transcript

1. Modeling State Transitions with Specification-based Random Testing Zeeshan Lakhani Software Engineer at Basho Technologies,Inc | Founder/Organizer Papers We Love @zeeshanlakhani 07-15-2015 (LambdaJam) Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 1 / 39
2. a beginning Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 2 / 39
3. a starter example1 membership in a list 1Testing Java with QuickCheck [Benac, Fredlund] http://bit.ly/1UYejBE Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 3 / 39
4. an initial state initial_state() -> [].
5. a group of function calls -spec add(list(), non_neg_integer()) -> list(). add(AList, N) -> [N|AList]. -spec is_member(list(), non_neg_integer()) -> boolean(). is_member(S, N) -> lists:member(N, S). Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 5 / 39
6. a few terms2, 3, 4 commands - symbolic calls to run during test sequences symbolic variables - generated during test generation dynamic values - generated during test execution next_state - operates during test generation and execution preconditions - only depend on the model, not execution time values postcondition - predicate called during test execution (that must hold) with the dynamic state before the call aggregate - Collects a list of values in each test, and shows the distribution of list elements 2eqc_statem notes [Stone] http://bit.ly/1dZ6qu2 3Testing Erlang Concurrency with QuickCheck [Cao] http://bit.ly/1UZQVnb 4Ulf Norell’s notes Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 6 / 39
7. a group of commands -spec add_command(S :: eqc_statem:symbolic_state()) -> eqc_gen:gen(eqc_statem:call()). add_command(S) -> {call, ?MODULE, add, [S, nat()]}. -spec is_member_command( S :: eqc_statem:symbolic_state()) -> eqc_gen:gen(eqc_statem:call()). is_member_command(S) -> {call, ?MODULE, is_member, [S, nat()]}. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 7 / 39
8. a group of preconditions -spec add_pre(S :: eqc_statem:symbolic_state(), Args :: [term()]) -> boolean(). add_pre(S, _Args) -> S /= undefined. -spec is_member_pre(S :: eqc_statem:symbolic_state(), Args :: [term()]) -> boolean(). is_member_pre(S, _Args) -> S /= undefined. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 8 / 39
9. a group of next states -spec add_next(S :: eqc_statem:symbolic_state(), V :: eqc_statem:var(), Args :: [term()]) -> eqc_statem:symbolic_state(). add_next(S, _Value, [_, N]) -> [N|S]. -spec is_member_next(S :: eqc_statem:symbolic_state(), V :: eqc_statem:var(), Args :: [term()]) -> eqc_statem:symbolic_state(). is_member_next(S, _Value, _Args) -> S. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 9 / 39
10. a group of postconditions -spec add_post(S :: eqc_statem:dynamic_state(), Args :: [term()], R :: term()) -> true | term(). add_post(S, [_, N], Res) -> [N|S] =:= Res. -spec is_member_post(S :: eqc_statem:dynamic_state(), Args :: [term()], R :: term()) -> true | term(). is_member_post(S, [_, N], Res) -> Res =:= lists:member(N, S). Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 10 / 39
11. a set of invariants -spec invariant(eqc_statem:dynamic_state()) -> boolean(). invariant(S) when length(S) >= 0 -> true; invariant(S) when length(S) > 0 -> FirstNum = hd(S), is_number(FirstNum) andalso FirstNum >= 0; invariant(_) -> false. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 11 / 39
12. a ?FORALL prop_test() -> ?FORALL(Cmds,commands(?MODULE), aggregate( command_names(Cmds), begin {H, S, Res} = run_commands(?MODULE, Cmds), pretty_commands(?MODULE, Cmds, {H, S, Res}, Res =:= ok) end)). Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 12 / 39
13. a fake failure .Failed! After 2 tests. [{set,{var,1},{call,stateful_sm,add,[[],0]}}, {set,{var,2},{call,stateful_sm,add,[[0],0]}}, {set,{var,3},{call,stateful_sm,is_member,[[0,0],0]}}, {set,{var,4},{call,stateful_sm,add,[[0,0],0]}}, {set,{var,5},{call,stateful_sm,is_member,[[0,0,0],0]}}] stateful_sm:add([], 0) -> [0] stateful_sm:add([0], 0) -> [0, 0] stateful_sm:is_member([0, 0], 0) -> true Reason: false Shrinking xxx...(3 times) [{set,{var,1},{call,stateful_sm,add,[[0,0],0]}}] stateful_sm:add([0, 0], 0) -> [0, 0, 0] Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 13 / 39
14. a search system curl "$RIAK_HOST/search/query/lambdajam?wt=json&q=name_s:lj_*" distributed across a cluster of nodes5 5Exploiting Parallelism in Query Processing for Web Document Search Using Shared-Memory and Cluster-Based Architectures [Aboutabl] http://bit.ly/1f2JElM Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 14 / 39
15. a way to index key-value data A liveness property guarantees that “something good eventually happens;” for example, all requests eventually receive a response.6 how it does it7 6Eventual Consistency Today [Bailis, Ghodsi] http://bit.ly/1K3Lgcw 7Riak Search 2.0 [Redmond] http://bit.ly/1HvOrE7 Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 15 / 39
16. a mechanism - active anti-entropy Merkle Treesa aA space- and time-efficient Implementation of the Merkle Tree Traversal Algorithm [Knecht, Nicola] http://bit.ly/1GjNLj7 a complete binary tree with an n-bit value associated with each node. Each internal node value is the result of a hash of the node values of its children (n is the number of bits returned by the hash function). Goals a aActive Anti-Antropy [Blomstedt] disk-based -> issues with in-memory trees containing billions of keys persistent -> build a tree once real-time updates - liveness non-blocking - can’t affect incoming write-rate Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 16 / 39
17. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 17 / 39
18. a difference riak search / yokozuna uses lucene’s termsenum8 to iterate over entropy data containing bucket/key & object hash riak search / yokozuna trees always repair entries remote_missing. . . key-value data is canonical 8http://bit.ly/1RyHewY Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 18 / 39
19. a problem riak_kv_vnode:actual_put:1460 vnode-kv: {r_object,<<"fruit_aae">>,<<"testfor spaces 342">>... and [info] <0.5785.0>@yz_solr:get_pairs:421 yz: [{struct,[{<<"vsn">>,<<"2">>}, {<<"riak_bucket_type">>,<<"default">>}, {<<"riak_bucket_name">>,<<"fruit_aae">>}, {<<"riak_key">>,<<"testfor spaces 342">>}... Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 19 / 39
20. a collection of generators vclock() -> ?LET(VclockSym, vclock_sym(), eval(VclockSym)). vclock_sym() -> ?LAZY( oneof([ {call, vclock, fresh, []}, ?LETSHRINK([Clock], [vclock_sym()], {call, ?MODULE, increment, [noshrink(binary(4)), nat(), Clock]}) ])). not_empty(G) -> ?SUCHTHAT(X, G, X /= [] andalso X /= <<>>). Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 20 / 39
21. increment(Actor, Count, Vclock) -> lists:foldl( fun vclock:increment/2, Vclock, lists:duplicate(Count, Actor)). riak_object() -> ?LET({{Bucket, Key}, Vclock, Value}, {bkey(), vclock(), binary()}, riak_object:set_vclock( riak_object:new(Bucket, Key, Value), Vclock)). bkey() -> {non_blank_string(), %% bucket non_blank_string()}. %% key Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 21 / 39
22. a generated riak object {r_object,<<"|plVWx&F">>,<<"?#sjiGS|">>, [{r_content,{dict,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[], [],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[], [],[]}}}, <<"t+">>}], [], {dict,1,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[], [[clean|true]], []}}}, undefined} Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 22 / 39
23. a state record9 -record(state, {yz_idx_tree, kv_idx_tree, yz_idx_objects = dict:new(), kv_idx_objects = dict:new(), trees_updated = false, both = []}). %% Initialize State initial_state() -> #state{}. 9eqc_util.erl (http://bit.ly/1K68zmd) and yz_index_hashtree_eqc.erl (http://bit.ly/1INKmwQ) [Daily, Lakhani] Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 23 / 39
24. a couple of tree processes begin we start a 1 riak key-value idx-tree process and a 1 yokozuna idx-tree process this happens per vnode (each vnode is responsible for a partition) each process contains multiple hashtrees due to preflist overlap some nodes contain data not part of preflist, otherwise always divergent10 10Active Anti-Antropy [Blomstedt] Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 24 / 39
25. an insert into a kv-tree insert_kv_tree_command(S) -> {call, ?MODULE, insert_kv_tree, [insert_method(), eqc_util:riak_object(), S#state.kv_idx_tree]}. insert_kv_tree_next(S, _V, [_, RObj, _]) -> {ok, TreeData} = dict:find(?TEST_INDEX_N, S#state.kv_idx_objects), S#state{kv_idx_objects=dict:store( ?TEST_INDEX_N, set_treedata(RObj, TreeData), S#state.kv_idx_objects), trees_updated=false}. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 25 / 39
26. insert_kv_tree_pre(S, _Args) -> S#state.kv_idx_tree /= undefined. insert_kv_tree_post(_S, _Args, _Res) -> true. insert_kv_tree(Method, RObj, {ok, TreePid}) -> {Bucket, Key} = eqc_util:get_bkey_from_object(RObj), Items = [{object, {Bucket, Key}, RObj}], case Method of sync -> riak_kv_index_hashtree:insert( Items, [], TreePid); async -> riak_kv_index_hashtree:async_insert( Items, [], TreePid) end. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 26 / 39
27. a number of inserts and dirty segments Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 27 / 39
28. an insert into a yz-tree insert_yz_tree_command(S) -> {call, ?MODULE, insert_yz_tree, [insert_method(), eqc_util:riak_object(), S#state.yz_idx_tree]}. insert_yz_tree_next(S, _V, [_, RObj, _]) -> {ok, TreeData} = dict:find(?TEST_INDEX_N, S#state.yz_idx_objects), S#state{yz_idx_objects=dict:store( ?TEST_INDEX_N, set_treedata(RObj, TreeData), S#state.yz_idx_objects), trees_updated=false}. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 28 / 39
29. insert_yz_tree(Method, RObj, {ok, TreePid}) -> BKey = eqc_util:get_bkey_from_object(RObj), yz_index_hashtree:insert( Method, ?TEST_INDEX_N, BKey, yz_kv:hash_object(RObj), TreePid, []). Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 29 / 39
30. an insert into both trees insert_both(Method, RObj, YZOkTree, KVOkTree) -> {insert_yz_tree(Method, RObj, YZOkTree), insert_kv_tree(Method, RObj, KVOkTree)}. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 30 / 39
31. an update update_command(S) -> {call, ?MODULE, update, [S#state.yz_idx_tree, S#state.kv_idx_tree]}. update_pre(S, _Args) -> S#state.yz_idx_tree /= undefined andalso S#state.kv_idx_tree /= undefined. update_next(S, _Value, _Args) -> S#state{trees_updated=true}. update_post(S, _Args, _Res) -> ... eq(ModelKVKeyCount, RealKVKeyCount) and eq(ModelYZKeyCount, RealYZKeyCount). Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 31 / 39
32. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 32 / 39
33. a comparison of trees compare_command(S) -> {call, ?MODULE, compare, [S#state.yz_idx_tree, S#state.kv_idx_tree]}. compare({ok, YZTreePid}, {ok, KVTreePid}) -> Remote = fun(get_bucket, {L, B}) -> riak_kv_index_hashtree:exchange_bucket(?TEST_INDEX_N, L, B, KVTreePid); (key_hashes, Segment) -> riak_kv_index_hashtree:exchange_segment(?TEST_INDEX_N, Segment, KVTreePid); (_, _) -> ok end, Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 33 / 39
34. AccFun = fun(KeyDiff, Count) -> lists:foldl(fun(Diff, InnerCount) -> case repair(0, Diff) of full_repair -> InnerCount + 1; _ -> InnerCount end end, Count, KeyDiff) end, yz_index_hashtree:compare(?TEST_INDEX_N, Remote, AccFun, 0, YZTreePid). Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 34 / 39
35. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 35 / 39
36. a key precondition compare_pre(S, _Args) -> S#state.yz_idx_tree /= undefined andalso S#state.kv_idx_tree /= undefined andalso S#state.trees_updated. Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 36 / 39
37. a key postcondition compare_post(S, _Args, Res) -> YZTreeData = dict:fetch(?TEST_INDEX_N, S#state.yz_idx_objects), KVTreeData = dict:fetch(?TEST_INDEX_N, S#state.kv_idx_objects), Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 37 / 39
38. LeftDiff = dict:fold(fun(BKey, Hash, Count) -> case dict:find(BKey, KVTreeData) of {ok, Hash} -> Count; {ok, _OtherHash} -> Count; error -> Count+1 end end, 0, YZTreeData), RightDiff = dict:fold(fun(BKey, Hash, Count) -> case dict:find(BKey, YZTreeData) of {ok, Hash} -> Count; {ok, _OtherHash} -> Count; error -> Count+1 end end, LeftDiff, KVTreeData), eq(RightDiff, Res). Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 38 / 39
39. a distribution of commands ................. OK, passed 100 tests 22.3% {yz_index_hashtree_eqc,insert_yz_tree,3} 20.7% {yz_index_hashtree_eqc,insert_kv_tree,3} 18.1% {yz_index_hashtree_eqc,insert_both,4} 17.7% {yz_index_hashtree_eqc,update,2} 9.2% {yz_index_hashtree_eqc,start_kv_tree,0} 8.8% {yz_index_hashtree_eqc,start_yz_tree,0} 3.3% {yz_index_hashtree_eqc,compare,2} Zeeshan Lakhani Modeling State Transitions with Specification-based Random Testing 07-15-2015 (LambdaJam) 39 / 39