Development of Test Suite with Formally
Verified FSM Coverage: A Case Study
Iosif Itkin, Rostislav Yavorskiy
TMPA-2019: Software Testing, Machine Learning and Complex Process Analysis
Tbilisi, Georgia on 7-9 November 2019
Slide 2
Slide 2 text
Order book
2
Slide 3
Slide 3 text
The model restrictions
● Bid queue < N1
● Ask queue < N2
● Number of the order price levels < N3
3
Questions
Which hyper-states are reachable?
Which one-step transitions between the hyper-states are
possible?
0|M
M|M
M|0
0|0
0|M
M|0
M|0
5
Slide 6
Slide 6 text
The goal
Create test suite with formally verified 100% coverage
6
Slide 7
Slide 7 text
Phase 1: Random generation of the tests
Advantage:
- Fast
- Easy to implement
Disadvantage:
- No way to guarantee the completeness
7
Slide 8
Slide 8 text
Time to discover the hyper-graph links
(number of scenarios)
8
Slide 9
Slide 9 text
Time to discover the hyper-graph links
(number of scenarios)
9
Slide 10
Slide 10 text
Phase 2: Formal verification of the completeness
10
Slide 11
Slide 11 text
Phase 2: Formal verification of the completeness
11
Slide 12
Slide 12 text
Conclusion
Two reasons of the incompleteness:
1. Time to discover reachable transitions between
hyper-states grows up exponentially
2. Bugs in the random test generation module.
12