Passive testing of production systems based on model inference (MEMOCODE 2015)

This paper tackles the problem of testing production systems, i.e. systems that run in industrial environments, and that are distributed over several devices and sensors. Usually, such systems lack of models, or are expressed with models that are not up to date. Without any model, the testing process is often done by hand, and tends to be an heavy and tedious task. This paper contributes to this issue by proposing a framework called Autofunk, which combines different fields such as model inference, expert systems, and machine learning. This framework, designed with the collaboration of our industrial partner Michelin, infers formal models that can be used as specifications to perform offline passive testing. Given a large set of production messages, it infers exact models that only capture the functional behaviours of a system under analysis. Thereafter, inferred models are used as input by a passive tester, which checks whether a system under test conforms to these models. Since inferred models do not express all the possible behaviours that should happen, we define conformance with two implementation relations. We evaluate our framework on real production systems and show that it can be used in practice.

Transcript

1. Passive testing of
   production systems based
   on model inference.
   William Durand, Sébastien Salva
   September 22, 2015
   Ǯ
   / MEMOCODE'15
   

   View Slide

2. View Slide

3. Quick Tour @ Michelin
   

   View Slide

4. A factory is divided into several workshops,
   one for each step of the manufacturing process.
   

   View Slide

5. A production system is composed of devices, production
   machines, and one or more software to control them.
   q In our case, we target a single workshop only.
   

   View Slide

6. Software exchange information with points and machines by
   sending and receiving production messages.
   17-Sep-2015 23:29:59.50|17011|MSG_IN [pid: 1] [nsec: 8] [point: 1] ...
   17-Sep-2015 23:29:59.61|17021|MSG_OUT [pid: 1] [nsec: 8] [point: 3] ...
   17-Sep-2015 23:29:59.70|17011|MSG_IN [pid: 2] [nsec: 8] [point: 2] ...
   A simple example of 3 messages in a human readable format.
   

   View Slide

7. Production messages are exchanged in a binary format (custom
   protocols), through centralized exchanging systems.
   

   View Slide

8. Each production message is tied to a product (e.g. tire),
   identified by a product identifier (pid).
   Gathering all production messages related to a product
   allows to retrieve what happened to it (behaviours).
   

   View Slide

9. Background
   

   View Slide

10. Developement Teams POV
    100+ applications running in production
    Not (fully) covered by tests
    Documentation most likely outdated
    MUST be maintained for ~20 years!
    

    View Slide

11. Customers (Factories) POV
    Stability over anything else
    Maintenance periods are planned,
    but rather long (> 1 week)
    1h (unexpected) downtime = 50k $
    

    View Slide

12. Testing such production systems is complex,
    and takes a lot of time as it implies the physical
    devices, and there are numerous behaviours.
    

    View Slide

13. These behaviours could be formally described into a model.
    But writing such models would be complicated and error prone.
    q Not suitable for Michelin applications.
    

    View Slide

14. Our Approach (1/3)
    By leveraging the information carried by the messages, we
    build formal and exact models (STS) that describe functional
    behaviours of a production System Under Analysis (SUA).
    

    View Slide

15. Our Approach (2/3)
    Michelin's exchanging systems guarantee the order in which
    the production messages occured. We capture the messages
    directly into these systems to avoid message loss, reordering,
    and/or duplication of the production messages.
    

    View Slide

16. Our Approach (3/3)
    We take production messages from another System Under Test
    (SUT), and we check whether SUT conforms with SUA (using two
    implementation relations to define the notion of conformance).
    

    View Slide

17. The Big Picture
    

    View Slide

18. Model Inference
    1. We collect production system traces (monitoring)
    2. We segment these traces to create different complete
    trace sets (outlier detection approach)
    3. We build (rather large) STS models from these sets
    4. We reduce the models to obtain "usable" models
    Durand, W., & Salva, S. (2015). Autofunk: An Inference‐Based Formal Model Generation
    Framework for Production Systems. In FM 2015: Formal Methods (pp. 577‐580). Springer
    International Publishing.
    

    View Slide

19. Model Reduction
    y
    

    View Slide

20. Model Inference Experimentation
    10 million production messages (20 days)
    y
    161,035 traces
    y
    S R(S)
    77,058 branches 1,587 branches
    43,536 branches 1,585 branches
    q It took 6 minutes to build the two models.
    

    View Slide

21. In Depth Testing
    

    View Slide

22. Offline Passive Testing
    Two
    implementation relations
    :
    Trace preorder
    relation
    and our own
    weaker
    implementation relation
    Our testing algorithm relies on both to give verdicts
    † Partial models = No Fail verdict
    

    View Slide

23. The Need for a Weaker Impl. Relation
    "Since I know that my model is not complete, I am willing
    to accept not standard behaviours till a certain point."
    

    View Slide

24. Experimentation
    SUA: 53,996 traces
    SUT: 25,047 traces
    y
    98% are Pass traces. The remaining 2% are new
    behaviours that never occured before.
    q It took 10 minutes to check conformance.
    

    View Slide

25. Now, What?
    2% still represents many traces, and can contain many false
    positive. For Michelin engineers, it is still "better than nothing".
    Larger sets of traces to build the models should reduce the
    number of false positive
    But we should find a way to refine this possibly fail trace set
    

    View Slide

26. Conclusion
    Fast passive testing framework for a specific context
    Model inference: the more production messages, the better!
    Testing: still too many possibly fail traces
    

    View Slide

27. Future Work
    Online
    passive testing (just-in-time fault detection?)
    Active testing
    by leveraging the inferred models again
    Developing a way to focus on specific parts of the system
    

    View Slide

28. Thank You.
    Questions?
    

    View Slide