Relation between Test Coverage and Timed Automata Model Structure

Lukas Krejci, Jan Sobotka and Jiri Novak

International Conference on Software Testing, Machine Learning and Complex Process Analysis (TMPA-2019)
7-9 November 2019, Tbilisi

Video: https://youtu.be/9b3m3QFaXwY

TMPA Conference website https://tmpaconf.org/
TMPA Conference on Facebook https://www.facebook.com/groups/tmpaconf/

Transcript

1. Relation between Test Coverage and Timed Automata Model Structure Lukáš Krejčí, Jan Sobotka and Jiří Novák lukas.krejci@fel.cvut.cz TMPA 2019
2. Outline Background Automotive integration testing Model–Based Testing with Timed Automata models Problem description Case study Experiment Results Conclusion
3. Background Integration testing and Model-Based Testing
4. Integration Testing  Evaluation of interactions in a cluster of ECUs  Distributed functions  Bus communication  Done independently by the car manufacturer  ECUs usually come from different suppliers  With real hardware using HiL testing method  Complete car electronics or relevant part of electronic system  Test cases (sequences) implemented manually by test engineers  Our team is trying to deploy test generation using Model-Based Testing principles  Developed test cases maintained during car life cycle
5. MBT Concept Overview
6. HiL Testing Platform
7. HiL Testing Platform
8. Case Study Automatic trunk doors control and keyless locking systems
9. Modeling Language  System and its environment are modeled as a network of Timed Automata
10. System Under Test  Opening and closing of the automatic trunk doors using the buttons  Locking and unlocking the car using the remote control in keys, the key position detection and door handle
11. Problem Outline Different approaches of modeling
12. Problem Outline  Observer model is created according to System Specification  There are multiple approaches to modeling of both SUT and the environment  Fully permissive  Equivalent of random stimuli  Useful for discovering of corner cases  Fully restrictive  Reduces the possible traces  Allows more accurate models  Question is how different resulting model structure influences coverage of observer model
13. Simple Environment Model  Easy to create  Each input button is modeled as separate automaton
14. Complex Environment Model  Based on behavior of a real driver  Generates more realistic test cases
15. Simple Observer Model  Each subsystem is modeled as individual automaton  More accurate and permissive description of an SUT
16. Simple Observer Model  Each subsystem is modeled as individual automaton  More accurate and permissive description of an SUT
17. Complex Observer Model  Models the full system  More restrictive description of the entire SUT
18. Experiment Comparison of modeling approaches using structural criteria
19. Experiment Overview  Compare all model variants by structural coverage criteria  Coverage of nodes  Coverage of edges  Coverage of edge pairs  Test runs were driven by different strategies  Random  Systematic  Heuristic  Find the most suitable modeling approach
20. Taster Tool  Tool developed by our team MBT for online testing with Timed Automata models
21. Results Experiment results and comparison of modeling approaches
22. Nodes Coverage
23. Edges Coverage
24. Edge Pairs Coverage
25. Conclusions and Future Work Result of comparison and future research
26. Conclusions  Combination of simple environment model and simple observer model provided most consisted results  Worse performance of complex environment model is expected  Complex observer model provides good results as well, expect for edge-based criteria and heuristic strategy  Results suggest that coverage of observer model depends on structure of the environment model  It is more beneficial to create simpler, divided models, which are significantly easier to create and maintain
27. Future Work  Extension of the case study with additional subsystems  Propulsion  Intrusion detection  Evaluation test cases generation strategies  Extension of existing strategies in Taster tool  Utilization of machine learning
28. If you have any questions, feel free to ask Thank you for your attention