This paper tackles the problem of testing production systems, i.e. systems that run in industrial environments, and that are distributed over several devices and sensors. Usually, such systems lack of models, or are expressed with models that are not up to date. Without any model, the testing process is often done by hand, and tends to be an heavy and tedious task. This paper contributes to this issue by proposing a framework called Autofunk, which combines different fields such as model inference, expert systems, and machine learning. This framework, designed with the collaboration of our industrial partner Michelin, infers formal models that can be used as specifications to perform offline passive testing. Given a large set of production messages, it infers exact models that only capture the functional behaviours of a system under analysis. Thereafter, inferred models are used as input by a passive tester, which checks whether a system under test conforms to these models. Since inferred models do not express all the possible behaviours that should happen, we define conformance with two implementation relations. We evaluate our framework on real production systems and show that it can be used in practice.
Passive testing of
production systems based
on model inference.
William Durand, Sébastien Salva
September 22, 2015
Quick Tour @ Michelin
A factory is divided into several workshops,
one for each step of the manufacturing process.
A production system is composed of devices, production
machines, and one or more software to control them.
q In our case, we target a single workshop only.
Software exchange information with points and machines by
sending and receiving production messages.
17-Sep-2015 23:29:59.50|17011|MSG_IN [pid: 1] [nsec: 8] [point: 1] ...
17-Sep-2015 23:29:59.61|17021|MSG_OUT [pid: 1] [nsec: 8] [point: 3] ...
17-Sep-2015 23:29:59.70|17011|MSG_IN [pid: 2] [nsec: 8] [point: 2] ...
A simple example of 3 messages in a human readable format.
Production messages are exchanged in a binary format (custom
protocols), through centralized exchanging systems.
Each production message is tied to a product (e.g. tire),
identified by a product identifier (pid).
Gathering all production messages related to a product
allows to retrieve what happened to it (behaviours).
Developement Teams POV
100+ applications running in production
Not (fully) covered by tests
Documentation most likely outdated
MUST be maintained for ~20 years!
Customers (Factories) POV
Stability over anything else
Maintenance periods are planned,
but rather long (> 1 week)
1h (unexpected) downtime = 50k $
Testing such production systems is complex,
and takes a lot of time as it implies the physical
devices, and there are numerous behaviours.
These behaviours could be formally described into a model.
But writing such models would be complicated and error prone.
q Not suitable for Michelin applications.
Our Approach (1/3)
By leveraging the information carried by the messages, we
build formal and exact models (STS) that describe functional
behaviours of a production System Under Analysis (SUA).
Our Approach (2/3)
Michelin's exchanging systems guarantee the order in which
the production messages occured. We capture the messages
directly into these systems to avoid message loss, reordering,
and/or duplication of the production messages.
Our Approach (3/3)
We take production messages from another System Under Test
(SUT), and we check whether SUT conforms with SUA (using two
implementation relations to define the notion of conformance).
The Big Picture
1. We collect production system traces (monitoring)
2. We segment these traces to create different complete
trace sets (outlier detection approach)
3. We build (rather large) STS models from these sets
4. We reduce the models to obtain "usable" models
Durand, W., & Salva, S. (2015). Autofunk: An Inference‐Based Formal Model Generation
Framework for Production Systems. In FM 2015: Formal Methods (pp. 577‐580). Springer
Model Inference Experimentation
10 million production messages (20 days)
77,058 branches 1,587 branches
43,536 branches 1,585 branches
q It took 6 minutes to build the two models.
In Depth Testing
Offline Passive Testing
and our own
Our testing algorithm relies on both to give verdicts
Partial models = No Fail verdict
The Need for a Weaker Impl. Relation
"Since I know that my model is not complete, I am willing
to accept not standard behaviours till a certain point."
SUA: 53,996 traces
SUT: 25,047 traces
98% are Pass traces. The remaining 2% are new
behaviours that never occured before.
q It took 10 minutes to check conformance.
2% still represents many traces, and can contain many false
positive. For Michelin engineers, it is still "better than nothing".
Larger sets of traces to build the models should reduce the
number of false positive
But we should find a way to refine this possibly fail trace set
Fast passive testing framework for a specific context
Model inference: the more production messages, the better!
Testing: still too many possibly fail traces
passive testing (just-in-time fault detection?)
by leveraging the inferred models again
Developing a way to focus on specific parts of the system