About Me • LOU Xun (楼洵) • Erlang since univ. |> Elixir ~4 years • Software Engineer @ CCP Games • ESI (player-facing APIs for game data) • Internal Tools and Pipelines • Chat System (ejabberd in Elixir!)
EVE Online • Sci-fi (spaceship!) MMO, sandbox by players • fleet fights! • large scale: 6000+ on a same battlefield • consequential: B-R cost $300,000+ (2014) • single Python process, Time Dilation (TiDi) • Elixir?!
LIAR Goals • Prototype to replace current impl. • Each Location in an Erlang Process (Actor) • Multicore parallelism (multi-node?) • faster cores ($$$) more cores ($) • Message passing, “eventual consistency” • DSL, give more power to Game Design
TDD • Defined APIs make it easy to adopt • Incremental, iterative development • Focus on single feature • local, then remote • Example-based • Most tests we write are example-based
property "new attribute have correct data" do forall {id, value} <- {integer(), float()} do attr = Attribute.new(id, value) assert Attribute.get_value(attr) == Attribute.get_base_value(attr) end end
• Generators instead of static input • Defines input boundary property "new attribute have correct data" do forall {id, value} <- {integer(), float()} do attr = Attribute.new(id, value) assert Attribute.get_value(attr) == Attribute.get_base_value(attr) end end
• Generators instead of static input • Defines input boundary • Randomize input from large search spaces property "new attribute have correct data" do forall {id, value} <- {integer(), float()} do attr = Attribute.new(id, value) assert Attribute.get_value(attr) == Attribute.get_base_value(attr) end end
• Generators instead of static input • Defines input boundary • Randomize input from large search spaces • Find (minimal) counter examples for you property "new attribute have correct data" do forall {id, value} <- {integer(), float()} do attr = Attribute.new(id, value) assert Attribute.get_value(attr) == Attribute.get_base_value(attr) end end
• Generators instead of static input • Defines input boundary • Randomize input from large search spaces • Find (minimal) counter examples for you • How to define useful properties? property "new attribute have correct data" do forall {id, value} <- {integer(), float()} do attr = Attribute.new(id, value) assert Attribute.get_value(attr) == Attribute.get_base_value(attr) end end
One More Thi… Flaw • TDD: rarely cross-feature test cases! • load_item… unload and load again, does it work? • (hint: it doesn’t) • (hint2: no one would ever think of this) • Most other forms of testing as well • How is the system used in real world? • Generator, but for user behaviours?
Stateful PBT • Simulate real world usage of a system • Model the system with an “abstract statem” • Generate a sequence of commands • Execute all the commands • Check result / invariants • or, even just running all commands can fail
Almost Stateless… property "Liar top level APIs" do forall cmds in commands(__MODULE__) do ...setup ... {history, state, result} = run_commands(__MODULE__, cmds) ...tear down ... result == :ok ...custom output ... end end commands and run_commands • use defined callbacks • represents 2 steps in stateful PBT
Library Example • init: {[], []} (library, user) • command: new_book, borrow, return • precondition: true, library/user have the book • next_state: {[A], []} -> {[], [A]} • postcondition: only one A exist! (invariant)
Case Study: Concurrency Bug • Not live demo… (just for the look) • Read and use test output • Effectiveness vs. example-based tests • Tips on writing a stateful PBT • Inspiration for finding system property
Shrinking • As important as generating • removes inconsequential commands (noise) • focus on real problems • Tries to minimize the counter example • originally 27 commands… • shrank to 9 (1/3)
Captured Logs • Direct cause: trying to get attribute from nil item • First line: which actor crashed (“Location 7”) • Last line: crashed when handling what message • “remove item modifier at target location” • no “remove modifier” commands… • must happened during item unload! [error] GenServer {Liar.Runtime.LocationRegistry, 7} terminating ** (FunctionClauseError) ... (liar) lib/liar/item.ex:52: Liar.Item.get_attribute(nil, 46) ... Last message: {:"$gen_cast", {:rim_target, {92, 1}, {88, 46}}}
Observe • Erlang (thus Elixir) provides strong isolation • one crashed Actor doesn’t damage any other • neither the VM Liar.load_item(9, Liar.Item) Liar.add_item_modifier(:dr_add, {77, 11}, {77, 9})
Observe • Erlang (thus Elixir) provides strong isolation • one crashed Actor doesn’t damage any other • neither the VM • PropEr shows us all and only necessary steps • to produce and observe failure Liar.load_item(9, Liar.Item) Liar.add_item_modifier(:dr_add, {77, 11}, {77, 9})
Validating the Fix ◊ mix test test/liar_pbt_test.exs Excluding tags: [skip: true] OK: The input passed the test. . Finished in 0.1 seconds 1 property, 0 failures Randomized with seed 667925
Other Bugs Revealed • Item still registered after unload • Leftover outgoing modifiers after unload • Wrong return format • Bug in dependent package • …
Lines of Code Blank Comment Code code 278 228 1081 example test 86 1 370 stateful PBT 99 10 386 • more commands, even Process.exit! • refactor for readability • ~100 lines for debug output!
propertesting.com • by Fred Hebert • this talk highly inspired by him • Free for online reading • Learn You Some Erlang • Erlang in Anger • The Zen of Erlang • and more…
command “filtering” • No locations: only generate start_location • No items: only start_location or load_item • Has items: most functions are valid • Has modifiers: can remove modifiers def command(%__MODULE__{items: items} = state) when map_size(items) == 0 do frequency([ {1, {:call, Liar, :start_location, [gen_new_lid(state)]}}, {50, {:call, Liar, :load_item, [gen_loaded_lid(state), gen_new_item(state)]}} ]) end
command “filtering” • No locations: only generate start_location • No items: only start_location or load_item • Has items: most functions are valid • Has modifiers: can remove modifiers • Forces you to think how flexible the system is • NOT used during shrinking!
precondition • Validate arguments (exist in StateM…) • Correct shrinking relies on this • WTH no locations! • Shrink: remove several commands, then use precondition to valid the remaining sequence Commands: [ {:set, {:var, 2}, {:call, Liar, :load_item, [10, Liar.Item]}} ]
Stateful Test • Don’t repeat your logic! • Use simple state • Use inefficient algorithm • Check (partial) invariants • “only 1 book exist in library + user”
General Notes • “Fixing” the model (test) is normal • mix propcheck.clean • Adjust frequency to expose different bugs • And number / size of tests • especially helpful if setup is heavy • Nevertheless, great tools help (mix test )
LIAR Specific • Testing all Locations (multiple actors) • Requires synchronization • LIAR’s “consistency guarantee” • sync “call” after certain commands • reason for the observe step in case study
–Fred, propertesting.com Stateful property tests are particularly useful when “what the code should do”—what the user perceives —is simple, but “how the code does it”—how it is implemented—is complex.
EVE Rules • What is user’s perspective for attributes? • Actually quite simple: • base_value + all modifiers -> real value • modifiers carry source values • recursively apply the same simple rule!
Inspired? • Think properties from a user’s perspective • does it have a “simple” mental model? • Real impl. can’t afford the simple model • calculate once, store the result, propagate • Caching vs. recursive calculation
PBT and co. • Complement not replacement • (example-based) TDD for dev., PBT for verification • Better understanding of your system & domain • Do require some effort to get comfortable with • not suitable for all problems • but gives a lot of satisfaction when useful :)
aquarhead
gotok365
microcms
syumai
77web
nsuz
ledsun
shinmiy
akatsuki174
yui_knk
munetoshi
punchdrunker
zakiyama
lauravandoore
speakerdeck
mthomps
eileencodes
brettharned
gr2m
lara
jrom
michaelherold
tanoku
aleyda