Model-based testing: gone and back again - .NET Perth March 2020

My first job in software was as a tester using model-based testing. After switching to development, I've found that writing tests is more work, and gives me less confidence than before. In this talk, I'll explain what's missing in the way we write tests now, some more benefits of model-based testing, and why I'm going back to my roots.

Links:
https://saucelabs.com/blog/the-challenges-and-benefits-of-model-based-testing
https://statecharts.github.io/
http://graphwalker.github.io/
https://altom.gitlab.io/altwalker/altwalker/
https://xstate.js.org/
https://gannettdigital.github.io/simulato/
http://mit.bme.hu/~micskeiz/pages/modelbased_testing.html#tools

Transcript

1. Model-based testing Gone and back again
2. Model-based testing Gone and back again
3. This is a talk about ME
4. Ben Lowry @bplowry Consultant
5. Tester Developer
6. The trouble with testing • Hard to find time to write good tests • Avoid writing tests because I don’t want to maintain them • Don’t know what to test • Edge cases are hard to find • Still end up with bugs no matter how many I write
7. Model-based testing
8. MBT fixes the problems I’m having • Hard to find time to write good tests • Avoid writing tests because I don’t want to maintain them • Don’t know what to test • Edge cases are hard to find • Still end up with bugs no matter how many I write ✅ ✅ ✅ ✅
9. Step one: create an abstract model
10. Step one: create an abstract model • Markov chains • Theorem proving • Finite state machines • Statecharts
12. Zero Running Stopped
13. Zero Press main button Running Stopped Press second button
14. Zero Press main button Running Stopped Press second button Press main button Press main button
15. Zero Press main button Running
16. Zero Press main button Running GIVEN WHEN THEN
17. You can model just about anything as a finite state machine
19. Machine-friendly formats stateDiagram [*] --> Sleeping Sleeping --> Idle : Wake up Idle --> Sleeping : Bored Idle --> Eating : Start eating Eating --> Idle : Done eating Idle --> Zoomies : ??? Zoomies --> Sleeping : Tired https://mermaid-js.github.io/mermaid/#/stateDiagram
20. Step two: fill in actions and assertions
21. Step two: fill in actions and assertions • Tell the test executor how to figure out if it’s in the right state • Write code to perform the actions
22. Zero Press main button Running Stopped Press second button Press main button Press main button
23. Step three: generate the tests
24. Step three: generate the tests • Generate paths: • Shortest paths from start to end • Random choice of action • Weighted choice of action (Dijkstra’s algorithm) • Define stop conditions • 100% node/edge coverage • Number of iterations • Test duration • Never
25. Zero Press main button Running Stopped Press second button Press main button Press main button
26. Zero Press main button Running Stopped Press second button Press main button Press main button
27. Zero Press main button Running Stopped Press second button Press main button Press main button
28. Zero Press main button Running Stopped Press second button Press main button Press main button
29. Zero Press main button Running Stopped Press second button Press main button Press main button
30. Zero Press main button Running Stopped Press second button Press main button Press main button
31. Zero Press main button Running Stopped Press second button Press main button Press main button
32. Zero Press main button Running Stopped Press second button Press main button Press main button
33. Updating the tests when there are changes
34. Zero Press main button Running Stopped Press second button Press main button Press main button
35. Zero Press main button Running Stopped Press second button
36. Zero Press main button Running Stopped Press second button Running Running Zero Stopped Zero ???
37. Zero Press main button Running Stopped Press second button Running Running Zero Stopped Zero Lap
38. Zero Press main button Running Stopped Press second button Press main button Press main button
39. Zero Running Stopped Press second button Press main button Press main button Press main button
40. Zero Press second button Running Stopped Press second button Press main button Press main button Lap Press second button Press main button
41. Challenges with model-based testing • Learning curve – it’s a new approach for most • Hard to find tools • Can be hard to get the model right • When you’ve found a bug, it’ll keep popping up until it’s fixed
42. Tools • NModel • Spec Explorer • GraphWalker graphwalker.github.io • AltWalker altom.gitlab.io/altwalker/altwalker • XState xstate.js.org
43. Benefits of model-based testing • Better understanding of your app, or part of your app • Quicker to write tests, because you don’t • Easy to make changes • Tests won’t break because of refactoring • Better coverage and finding edge cases
44. Model-based testing is the logical progression from automated testing
45. Write Execute Manual testing Automated testing Model-based testing
46. Write Execute Manual testing Automated testing Model-based testing
47. Write Execute Manual testing Automated testing Model-based testing
48. Stop writing tests Because computers do it better than you can