Simulation Tests for API Based Systems

Presented at IN/Clojure 2018

Transcript

1. Simulation Tests for API Based Systems Shafeeq Kunnakkadan | nilenso
3. Agenda • Problem statement • About generative testing and Prior Art • Libraries and tools • Building the test suite
4. simulation sɪmjuːˈleɪʃ(ə)n imitation of a situation or process. the production of a computer model of something, especially for the purpose of study.
5. System Under Test • API Based Payment System • Designed to be distributed • Fast paced development • Slow (Manual) QA
6. Simple Checkout Store Smart PoS Shopper
7. Actual Checkout
8. n x Checkouts
9. Problem • Simulate all possible flows in a checkout process • Automate creation, execution, and reporting of tests • Run simulations in parallel, catch concurrency bugs
12. types of testing • Example Based Testing • Generative Testing • Property Based Testing • Simulation Testing
13. Example Based Testing • Input, Execute, Output, Validate • Unit, Functional, Integration
14. Example Based Testing
15. Example Based Testing • Limited coverage • Not scalable
16. Generative Testing • Generate Inputs for tests • Random • Validate against properties
17. Property Based Testing • Properties of the system • Invariants • Mathematical properties: Idempotence, Commutativity, Associativity etc
18. Simulation Testing • System model • Random • Simulate the user
19. Prior Art
20. QuickCheck John Hughes - Testing the Hard Stuff and Staying Sane
21. QuickCheck John Hughes - Testing the Hard Stuff and Staying Sane
22. test.check Gary Fredericks - Building test check Generators https://github.com/clojure/test.check
23. Simulant Michael Nygard - Simulation Testing https://github.com/Datomic/simulant
24. Libraries and Tools
25. Tools • Clojure • Schema • test.check • Datascript, datalog
27. test.check
32. Building the Test Suite
33. Stages
34. Generation: Params • From schema • Test.check generators • Input Constraints
35. Generation: Flow • Flow • Flow constraints • Idempotent operations (immediate, distant)
41. Execution • Action multimethods • Stateful Params • Known exceptions • Continue?
42. Assertion • Properties • CRUD • All ok? • State transitions
45. State machines make for great properties.
46. Reporting & Diagnostics • Assertion reports • Check points
47. Caveats
49. Conclusion • Use Simulation Tests • Properties are hard: State machine to the rescue • Grow your tests with minimal cost
50. Simulation Tests for API Based Systems Shafeeq Kunnakkadan | nilenso @shafeeq_k Icons from the Noun Project: Shop by Chanut is Industries; POS by Ben Davis; Shopper by H Alberto Gongora