Development of Test Suite with Formally Verified FSM Coverage: A Case Study

Rostislav Yavorskiy and Iosif Itkin

International Conference on Software Testing, Machine Learning and Complex Process Analysis (TMPA-2019)
7-9 November 2019, Tbilisi

Video: https://youtu.be/0FasFDcwYh8

TMPA Conference website https://tmpaconf.org/
TMPA Conference on Facebook https://www.facebook.com/groups/tmpaconf/

Transcript

1. Development of Test Suite with Formally Verified FSM Coverage: A Case Study Iosif Itkin, Rostislav Yavorskiy TMPA-2019: Software Testing, Machine Learning and Complex Process Analysis Tbilisi, Georgia on 7-9 November 2019
2. Order book 2
3. The model restrictions • Bid queue < N1 • Ask queue < N2 • Number of the order price levels < N3 3
4. Hyper-states (equivalence classes) Property definition: [0,1, 2, M] 0|M 0|M 0|0 0|0 M|0 M|0 M|0 4
5. Questions Which hyper-states are reachable? Which one-step transitions between the hyper-states are possible? 0|M M|M M|0 0|0 0|M M|0 M|0 5
6. The goal Create test suite with formally verified 100% coverage 6
7. Phase 1: Random generation of the tests Advantage: - Fast - Easy to implement Disadvantage: - No way to guarantee the completeness 7
8. Time to discover the hyper-graph links (number of scenarios) 8
9. Time to discover the hyper-graph links (number of scenarios) 9
10. Phase 2: Formal verification of the completeness 10
11. Phase 2: Formal verification of the completeness 11
12. Conclusion Two reasons of the incompleteness: 1. Time to discover reachable transitions between hyper-states grows up exponentially 2. Bugs in the random test generation module. 12