Development of Test Suite with Formally Verified FSM Coverage: A Case Study

Rostislav Yavorskiy and Iosif Itkin

International Conference on Software Testing, Machine Learning and Complex Process Analysis (TMPA-2019)
7-9 November 2019, Tbilisi

Video: https://youtu.be/0FasFDcwYh8

TMPA Conference website https://tmpaconf.org/
TMPA Conference on Facebook https://www.facebook.com/groups/tmpaconf/

Transcript

1. Development of Test Suite with Formally
   Verified FSM Coverage: A Case Study
   Iosif Itkin, Rostislav Yavorskiy
   TMPA-2019: Software Testing, Machine Learning and Complex Process Analysis
   Tbilisi, Georgia on 7-9 November 2019
   

   View Slide

2. Order book
   2
   

   View Slide

3. The model restrictions
   ● Bid queue < N1
   ● Ask queue < N2
   ● Number of the order price levels < N3
   3
   

   View Slide

4. Hyper-states (equivalence classes)
   Property definition: [0,1, 2, M]
   0|M
   0|M
   0|0
   0|0
   M|0
   M|0
   M|0
   4
   

   View Slide

5. Questions
   Which hyper-states are reachable?
   Which one-step transitions between the hyper-states are
   possible?
   0|M
   M|M
   M|0
   0|0
   0|M
   M|0
   M|0
   5
   

   View Slide

6. The goal
   Create test suite with formally verified 100% coverage
   6
   

   View Slide

7. Phase 1: Random generation of the tests
   Advantage:
   - Fast
   - Easy to implement
   Disadvantage:
   - No way to guarantee the completeness
   7
   

   View Slide

8. Time to discover the hyper-graph links
   (number of scenarios)
   8
   

   View Slide

9. Time to discover the hyper-graph links
   (number of scenarios)
   9
   

   View Slide

10. Phase 2: Formal verification of the completeness
    10
    

    View Slide

11. Phase 2: Formal verification of the completeness
    11
    

    View Slide

12. Conclusion
    Two reasons of the incompleteness:
    1. Time to discover reachable transitions between
    hyper-states grows up exponentially
    2. Bugs in the random test generation module.
    12
    

    View Slide