Relation between Test Coverage and Timed Automata Model Structure Lukáš Krejčí, Jan Sobotka and Jiří Novák [email protected] TMPA 2019

Outline Background Automotive integration testing Model–Based Testing with Timed Automata models Problem description Case study Experiment Results Conclusion

Background Integration testing and Model-Based Testing

Integration Testing  Evaluation of interactions in a cluster of ECUs  Distributed functions  Bus communication  Done independently by the car manufacturer  ECUs usually come from different suppliers  With real hardware using HiL testing method  Complete car electronics or relevant part of electronic system  Test cases (sequences) implemented manually by test engineers  Our team is trying to deploy test generation using Model-Based Testing principles  Developed test cases maintained during car life cycle

MBT Concept Overview

HiL Testing Platform

HiL Testing Platform

Case Study Automatic trunk doors control and keyless locking systems

Modeling Language  System and its environment are modeled as a network of Timed Automata

System Under Test  Opening and closing of the automatic trunk doors using the buttons  Locking and unlocking the car using the remote control in keys, the key position detection and door handle

Problem Outline Different approaches of modeling

Problem Outline  Observer model is created according to System Specification  There are multiple approaches to modeling of both SUT and the environment  Fully permissive  Equivalent of random stimuli  Useful for discovering of corner cases  Fully restrictive  Reduces the possible traces  Allows more accurate models  Question is how different resulting model structure influences coverage of observer model

Simple Environment Model  Easy to create  Each input button is modeled as separate automaton

Complex Environment Model  Based on behavior of a real driver  Generates more realistic test cases

Simple Observer Model  Each subsystem is modeled as individual automaton  More accurate and permissive description of an SUT

Simple Observer Model  Each subsystem is modeled as individual automaton  More accurate and permissive description of an SUT

Complex Observer Model  Models the full system  More restrictive description of the entire SUT

Experiment Comparison of modeling approaches using structural criteria

Experiment Overview  Compare all model variants by structural coverage criteria  Coverage of nodes  Coverage of edges  Coverage of edge pairs  Test runs were driven by different strategies  Random  Systematic  Heuristic  Find the most suitable modeling approach

Taster Tool  Tool developed by our team MBT for online testing with Timed Automata models

Results Experiment results and comparison of modeling approaches

Nodes Coverage

Edges Coverage

Edge Pairs Coverage

Conclusions and Future Work Result of comparison and future research

Conclusions  Combination of simple environment model and simple observer model provided most consisted results  Worse performance of complex environment model is expected  Complex observer model provides good results as well, expect for edge-based criteria and heuristic strategy  Results suggest that coverage of observer model depends on structure of the environment model  It is more beneficial to create simpler, divided models, which are significantly easier to create and maintain

Future Work  Extension of the case study with additional subsystems  Propulsion  Intrusion detection  Evaluation test cases generation strategies  Extension of existing strategies in Taster tool  Utilization of machine learning

If you have any questions, feel free to ask Thank you for your attention