Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Relation between Test Coverage and Timed Automata Model Structure

Exactpro
November 08, 2019

Relation between Test Coverage and Timed Automata Model Structure

Lukas Krejci, Jan Sobotka and Jiri Novak

International Conference on Software Testing, Machine Learning and Complex Process Analysis (TMPA-2019)
7-9 November 2019, Tbilisi

Video: https://youtu.be/9b3m3QFaXwY

TMPA Conference website https://tmpaconf.org/
TMPA Conference on Facebook https://www.facebook.com/groups/tmpaconf/

Exactpro

November 08, 2019
Tweet

More Decks by Exactpro

Other Decks in Technology

Transcript

  1. Relation between Test
    Coverage and Timed
    Automata Model Structure
    Lukáš Krejčí, Jan Sobotka and Jiří Novák
    [email protected]
    TMPA 2019

    View full-size slide

  2. Outline
    Background Automotive integration testing
    Model–Based Testing with Timed Automata models
    Problem description
    Case study
    Experiment
    Results
    Conclusion

    View full-size slide

  3. Background
    Integration testing and Model-Based Testing

    View full-size slide

  4. Integration Testing
     Evaluation of interactions in a cluster of ECUs
     Distributed functions
     Bus communication
     Done independently by the car manufacturer
     ECUs usually come from different suppliers
     With real hardware using HiL testing method
     Complete car electronics or relevant part of electronic system
     Test cases (sequences) implemented manually by test engineers
     Our team is trying to deploy test generation using Model-Based
    Testing principles
     Developed test cases maintained during car life cycle

    View full-size slide

  5. MBT Concept Overview

    View full-size slide

  6. HiL Testing Platform

    View full-size slide

  7. HiL Testing Platform

    View full-size slide

  8. Case Study
    Automatic trunk doors control and keyless locking systems

    View full-size slide

  9. Modeling Language
     System and its environment are modeled
    as a network of Timed Automata

    View full-size slide

  10. System Under Test
     Opening and closing of the automatic trunk doors using the buttons
     Locking and unlocking the car using the remote control in keys, the
    key position detection and door handle

    View full-size slide

  11. Problem Outline
    Different approaches of modeling

    View full-size slide

  12. Problem Outline
     Observer model is created according to System Specification
     There are multiple approaches to modeling of both SUT and the environment
     Fully permissive
     Equivalent of random stimuli
     Useful for discovering of corner cases
     Fully restrictive
     Reduces the possible traces
     Allows more accurate models
     Question is how different resulting model structure influences coverage of
    observer model

    View full-size slide

  13. Simple Environment Model
     Easy to create
     Each input button is modeled as separate automaton

    View full-size slide

  14. Complex Environment Model
     Based on behavior of a real driver
     Generates more realistic test cases

    View full-size slide

  15. Simple Observer Model
     Each subsystem is modeled as individual automaton
     More accurate and permissive description of an SUT

    View full-size slide

  16. Simple Observer Model
     Each subsystem is modeled as individual automaton
     More accurate and permissive description of an SUT

    View full-size slide

  17. Complex Observer Model
     Models the full system
     More restrictive description of the entire SUT

    View full-size slide

  18. Experiment
    Comparison of modeling approaches using structural criteria

    View full-size slide

  19. Experiment Overview
     Compare all model variants by structural coverage criteria
     Coverage of nodes
     Coverage of edges
     Coverage of edge pairs
     Test runs were driven by different strategies
     Random
     Systematic
     Heuristic
     Find the most suitable modeling approach

    View full-size slide

  20. Taster Tool
     Tool developed by our team MBT for online testing with Timed
    Automata models

    View full-size slide

  21. Results
    Experiment results and comparison of modeling approaches

    View full-size slide

  22. Nodes Coverage

    View full-size slide

  23. Edges Coverage

    View full-size slide

  24. Edge Pairs Coverage

    View full-size slide

  25. Conclusions and Future Work
    Result of comparison and future research

    View full-size slide

  26. Conclusions
     Combination of simple environment model and simple observer model
    provided most consisted results
     Worse performance of complex environment model is expected
     Complex observer model provides good results as well, expect for
    edge-based criteria and heuristic strategy
     Results suggest that coverage of observer model depends on structure
    of the environment model
     It is more beneficial to create simpler, divided models, which are
    significantly easier to create and maintain

    View full-size slide

  27. Future Work
     Extension of the case study with additional subsystems
     Propulsion
     Intrusion detection
     Evaluation test cases generation strategies
     Extension of existing strategies in Taster tool
     Utilization of machine learning

    View full-size slide

  28. If you have any questions, feel free to ask
    Thank you for your attention

    View full-size slide