Modeling State Transitions with Specification-based Random Testing 

Zeeshan Lakhani software engineer at Basho Technologies, Inc and founder/organizer of Papers We Love Talk by 

“It's really hard to explain that test infrastructure for distributed systems is actually a fantastically deep and cool problem.” Jay Kreps 05/15/2015 1 

No content

extremely painful for you 

The Inspiration: Dynamo 2 3 

No content

curl “$RIAK_HOST/search/query/cufp?wt=json&q=name_s:lj_*” • distributed across a cluster of nodes • all index entries for a given riak object are co-located on the same physical machine • b/c of replication, not all nodes need to be queried, only a coverage set 4 

“A liveness property guarantees that ‘something good eventually happens’; for example, all requests eventually receive a response.” Bailis, Ghodsi Eventual consistency today: limitations, extensions, and beyond Communications of the ACM #56 5 

• “To ensure convergence, replicas must exchange information with one another about which writes they have seen” • This exchange is called anti-entropy • read-repair is an anti-entropy mechanism to repair out-of-date replicas optimistic replication and resolution 7 8 6 

but what about “cold data?” 

or search queries that may return inconsistent results? 

read-repair all the keys? 

• background process using/storing merkle trees to keep divergent and missing data in sync • disk-based: issues with in-memory trees • containing billions of persistent keys: build a tree once
 • real-time updates - liveness
 • non-blocking - can’t affect incoming write-rate riak’s AAE (active anti-entropy)9 

• riak search / yokozuna iterates over entropy data containing {bucket_type, bucket}/key and object hash • riak search / yokozuna trees always repair entries remote_missing. . . key- value data is canonical 

riak_kv_vnode:actual_put:1460 vnode-kv: {r_object,<<"fruit_aae">>,<<"testfor spaces 342”>>... [info] <0.5785.0>@yz_solr:get_pairs:421 yz: [{struct,[{<<"vsn">>,<<"2">>}, {<<"riak_bucket_type">>,<<"default">>}, {<<"riak_bucket_name">>,<<"fruit_aae">>}, {<<"riak_key">>,<<"testfor spaces 342">>}... a bug in the wild 

• a complete binary tree with an n-bit value associated with each node • Each internal node value is the result of a hash of the node values of its children (n is the number of bits returned by the hash function) merkle trees 10 

11 

4 

4 

4 

4 

12 

No content

• Generators • SHRINKING • Controlled Randomness • Randomized scheduling to find race conditions (Pulse) 

%% @doc - Perform BAD_is_member action -spec bad_is_member(list(), non_neg_integer()) -> boolean(). bad_is_member(S, N) -> lists:member(N, lists:sublist(S, 5)). shrinking 

shrinking … stateful_sm:is_member([5, 0, 5, 1, 4], 3) -> false stateful_sm:is_member([5, 0, 5, 1, 4], 4) -> true stateful_sm:is_member([5, 0, 5, 1, 4], 2) -> false stateful_sm:add([5, 0, 5, 1, 4], 3) -> [3, 5, 0, 5, 1, 4] stateful_sm:is_member([3, 5, 0, 5, 1, 4], 2) -> false stateful_sm:add([3, 5, 0, 5, 1, 4], 0) -> [0, 3, 5, 0, 5, 1, 4] stateful_sm:add([0, 3, 5, 0, 5, 1, 4], 2) -> [2, 0, 3, 5, 0, 5, 1, 4] stateful_sm:add([2, 0, 3, 5, 0, 5, 1, 4], 0) -> [0, 2, 0, 3, 5, 0, 5, 1, 4] stateful_sm:is_member([0, 2, 0, 3, 5, 0, 5, 1, 4], 4) -> false Reason: false Shrinking xxxxx....x..(6 times) [{set,{var,1},{call,stateful_sm,add,[[5,1,4],0]}}] 

my generation vclock() -> ?LET(VclockSym, vclock_sym(), eval(VclockSym)). vclock_sym() -> ?LAZY( oneof([ {call, vclock, fresh, []}, ?LETSHRINK([Clock], [vclock_sym()], {call, ?MODULE, increment, [noshrink(binary(4)), nat(), Clock]}) ])). not_empty(G) -> ?SUCHTHAT(X, G, X /= [] andalso X /= <<>>). 

increment(Actor, Count, Vclock) -> lists:foldl( fun vclock:increment/2, Vclock, lists:duplicate(Count, Actor)). riak_object() -> ?LET({{Bucket, Key}, Vclock, Value}, {bkey(), vclock(), binary()}, riak_object:set_vclock( riak_object:new(Bucket, Key, Value), Vclock)). bkey() -> {non_blank_string(), %% bucket non_blank_string()}. %% key 

riak_object() -> ?LET({{Bucket, Key}, Vclock, Value}, {bkey(), vclock(), binary()}, riak_object:set_vclock( riak_object:new(Bucket, Key, Value), Vclock)). a generated riak object 

a generated riak object {r_object,<<"|plVWx&F">>,<<"?#sjiGS|">>, [{r_content,{dict,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[], [],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[], [],[]}}}, <<"t+">>}], [], {dict,1,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[], [[clean|true]], []}}}, undefined} 

-include_lib("eqc/include/eqc.hrl"). -include_lib("eqc/include/eqc_statem.hrl"). modeling state declaratively Provides functions for testing operations with side-effects, which are specified via an abstract state machine. 

•commands - symbolic calls to run during test sequences •symbolic variables - generated during test generation •dynamic values - generated during test execution •next_state - operates during test generation and execution •preconditions - ensure a logical precedence between operations, checks command in generation and used in shrinking •postcondition - predicate called during test execution (that must hold) with the dynamic state before the call •aggregate - Collects a list of values in each test, and shows the distribution of list elements a few terms 

-record(state, {yz_idx_tree, kv_idx_tree, yz_idx_objects = dict:new(), kv_idx_objects = dict:new(), trees_updated = false, both = []}). %% Initialize State initial_state() -> #state{}. the model’s state record13 

• we start a 1 riak key-value idx-tree process and a 1 yokozuna idx-tree process • this happens per vnode (each vnode is responsible for a partition) • each process contains multiple hashtrees due to preflist overlap • some nodes contain data not part of preflist, otherwise always divergent 

%% We want to constrain all data to fit in the last portion of our %% hash space so that it maps to partition 0. hashfun({_Bucket, _Key}=ID) -> %% Top valid hash value Top = (1 bsl 160) - 1, %% Calculate partition size PartitionSize = chash:ring_increment(?RING_SIZE), %% Generate an integer hash Hash = intify_sha(riak_core_util:chash_std_keyfun(ID)), %% Map the hash to 1/?RING_SIZE of the full hash space SmallHash = Hash rem PartitionSize, %% Force the hash into the last 1/?RING_SIZE block Top - SmallHash. 

• start • insert • update • compare tree (process) operations 

insert_kv_tree_pre(S, _Args) -> S#state.kv_idx_tree /= undefined. insert_kv_tree_command(S) -> {call, ?MODULE, insert_kv_tree, [insert_method(), eqc_util:riak_object(), S#state.kv_idx_tree]}. insert_kv_tree_next(S, _V, [_, RObj, _]) -> {ok, TreeData} = dict:find(?TEST_INDEX_N, S#state.kv_idx_objects), S#state{kv_idx_objects=dict:store( ?TEST_INDEX_N, set_treedata(RObj, TreeData), S#state.kv_idx_objects), trees_updated=false}. insert into the key-val hashtree 

insert_kv_tree(Method, RObj, {ok, TreePid}) -> {Bucket, Key} = eqc_util:get_bkey_from_object(RObj), Items = [{object, {Bucket, Key}, RObj}], case Method of sync -> riak_kv_index_hashtree:insert( Items, [], TreePid); async -> riak_kv_index_hashtree:async_insert( Items, [], TreePid) end. insert_kv_tree_post(_S, _Args, _Res) -> true. 

insert_yz_tree_command(S) -> {call, ?MODULE, insert_yz_tree, [insert_method(), eqc_util:riak_object(), S#state.yz_idx_tree]}. insert_yz_tree_next(S, _V, [_, RObj, _]) -> {ok, TreeData} = dict:find(?TEST_INDEX_N, S#state.yz_idx_objects), S#state{yz_idx_objects=dict:store( ?TEST_INDEX_N, set_treedata(RObj, TreeData), S#state.yz_idx_objects), trees_updated=false}. insert_yz_tree(Method, RObj, {ok, TreePid}) -> BKey = eqc_util:get_bkey_from_object(RObj), yz_index_hashtree:insert( Method, ?TEST_INDEX_N, BKey, yz_kv:hash_object(RObj), TreePid, []). insert into the search hashtree 

-spec insert_both(sync|async, obj(), {ok, tree()}, {ok, tree()}) -> {ok, ok}. insert_both(Method, RObj, YZOkTree, KVOkTree) -> {insert_yz_tree(Method, RObj, YZOkTree), insert_kv_tree(Method, RObj, KVOkTree)}. 

4 

update_pre(S, _Args) -> S#state.yz_idx_tree /= undefined andalso S#state.kv_idx_tree /= undefined. update_command(S) -> {call, ?MODULE, update, [S#state.yz_idx_tree, S#state.kv_idx_tree]}. update_next(S, _Value, _Args) -> S#state{trees_updated=true}. update({ok, YZTreePid}, {ok, KVTreePid}) -> yz_index_hashtree:update(?TEST_INDEX_N, YZTreePid), riak_kv_index_hashtree:update(?TEST_INDEX_N, KVTreePid), ok. update_post(S, _Args, _Res) -> ... eq(ModelKVKeyCount, RealKVKeyCount) and eq(ModelYZKeyCount, RealYZKeyCount). update the hashtrees 

4 

compare_pre(S, _Args) -> S#state.yz_idx_tree /= undefined andalso S#state.kv_idx_tree /= undefined andalso S#state.trees_updated. compare_command(S) -> {call, ?MODULE, compare, [S#state.yz_idx_tree, S#state.kv_idx_tree]}. compare({ok, YZTreePid}, {ok, KVTreePid}) -> Remote = fun(get_bucket, {L, B}) -> riak_kv_index_hashtree:exchange_bucket(?TEST_INDEX_N, L, B, KVTreePid); (key_hashes, Segment) -> riak_kv_index_hashtree:exchange_segment(?TEST_INDEX_N, Segment, KVTreePid); (_, _) -> ok end, AccFun = fun(KeyDiff, Count) -> lists:foldl(fun(Diff, InnerCount) -> case repair(0, Diff) of full_repair -> InnerCount + 1; _ -> InnerCount end end, Count, KeyDiff) end, yz_index_hashtree:compare(?TEST_INDEX_N, Remote, AccFun, 0, YZTreePid). compare the hashtrees 

4 

compare_post(S, _Args, Res) -> YZTreeData = dict:fetch(?TEST_INDEX_N, S#state.yz_idx_objects), KVTreeData = dict:fetch(?TEST_INDEX_N, S#state.kv_idx_objects), LeftDiff = dict:fold(fun(BKey, Hash, Count) -> case dict:find(BKey, KVTreeData) of {ok, Hash} -> Count; {ok, _OtherHash} -> Count; error -> Count+1 end end, 0, YZTreeData), RightDiff = dict:fold(fun(BKey, Hash, Count) -> case dict:find(BKey, YZTreeData) of {ok, Hash} -> Count; {ok, _OtherHash} -> Count; error -> Count+1 end end, LeftDiff, KVTreeData), eq(RightDiff, Res). 

prop_correct() -> ?FORALL(Cmds,commands(?MODULE, #state{}), aggregate(command_names(Cmds), ?TRAPEXIT(begin {H, S, Res} = run_commands(?MODULE, Cmds), catch yz_index_hashtree:destroy( element(2, S#state.yz_idx_tree)), catch riak_kv_index_hashtree:destroy( element(2, S#state.kv_idx_tree)), pretty_commands(?MODULE, Cmds, {H, S, Res}, Res == ok) end))). property test 

property test ................. OK, passed 100 tests 22.3% {yz_index_hashtree_eqc,insert_yz_tree,3} 20.7% {yz_index_hashtree_eqc,insert_kv_tree,3} 18.1% {yz_index_hashtree_eqc,insert_both,4} 17.7% {yz_index_hashtree_eqc,update,2} 9.2% {yz_index_hashtree_eqc,start_kv_tree,0} 8.8% {yz_index_hashtree_eqc,start_yz_tree,0} 3.3% {yz_index_hashtree_eqc,compare,2} 

John Daily Christopher Meiklejohn Sean Cribbs Jon Meredith Joe DeVivo thanks 

1. bit.ly/1JS3nRB 2. bit.ly/1Ne15if 3. bit.ly/1LZ5sxj 4. bit.ly/1f2JElM 5. bit.ly/1JG4cLU 6. bit.ly/1FmYpGS 7. bit.ly/1JG4cLU 8. bit.ly/1UqXAZW 9. bit.ly/1JEHCAX 10. bit.ly/1GjNLj7 11. bit.ly/1JDqdJ8 12. bit.ly/1EE9LLA 13. bit.ly/1INKmwQ footnotes