Passive testing of
production systems based
on model inference.
William Durand, Sébastien Salva
September 22, 2015
Ǯ
/ MEMOCODE'15
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
Quick Tour @ Michelin
Slide 4
Slide 4 text
A factory is divided into several workshops,
one for each step of the manufacturing process.
Slide 5
Slide 5 text
A production system is composed of devices, production
machines, and one or more software to control them.
q In our case, we target a single workshop only.
Slide 6
Slide 6 text
Software exchange information with points and machines by
sending and receiving production messages.
17-Sep-2015 23:29:59.50|17011|MSG_IN [pid: 1] [nsec: 8] [point: 1] ...
17-Sep-2015 23:29:59.61|17021|MSG_OUT [pid: 1] [nsec: 8] [point: 3] ...
17-Sep-2015 23:29:59.70|17011|MSG_IN [pid: 2] [nsec: 8] [point: 2] ...
A simple example of 3 messages in a human readable format.
Slide 7
Slide 7 text
Production messages are exchanged in a binary format (custom
protocols), through centralized exchanging systems.
Slide 8
Slide 8 text
Each production message is tied to a product (e.g. tire),
identified by a product identifier (pid).
Gathering all production messages related to a product
allows to retrieve what happened to it (behaviours).
Slide 9
Slide 9 text
Background
Slide 10
Slide 10 text
Developement Teams POV
100+ applications running in production
Not (fully) covered by tests
Documentation most likely outdated
MUST be maintained for ~20 years!
Slide 11
Slide 11 text
Customers (Factories) POV
Stability over anything else
Maintenance periods are planned,
but rather long (> 1 week)
1h (unexpected) downtime = 50k $
Slide 12
Slide 12 text
Testing such production systems is complex,
and takes a lot of time as it implies the physical
devices, and there are numerous behaviours.
Slide 13
Slide 13 text
These behaviours could be formally described into a model.
But writing such models would be complicated and error prone.
q Not suitable for Michelin applications.
Slide 14
Slide 14 text
Our Approach (1/3)
By leveraging the information carried by the messages, we
build formal and exact models (STS) that describe functional
behaviours of a production System Under Analysis (SUA).
Slide 15
Slide 15 text
Our Approach (2/3)
Michelin's exchanging systems guarantee the order in which
the production messages occured. We capture the messages
directly into these systems to avoid message loss, reordering,
and/or duplication of the production messages.
Slide 16
Slide 16 text
Our Approach (3/3)
We take production messages from another System Under Test
(SUT), and we check whether SUT conforms with SUA (using two
implementation relations to define the notion of conformance).
Slide 17
Slide 17 text
The Big Picture
Slide 18
Slide 18 text
Model Inference
1. We collect production system traces (monitoring)
2. We segment these traces to create different complete
trace sets (outlier detection approach)
3. We build (rather large) STS models from these sets
4. We reduce the models to obtain "usable" models
Durand, W., & Salva, S. (2015). Autofunk: An Inference‐Based Formal Model Generation
Framework for Production Systems. In FM 2015: Formal Methods (pp. 577‐580). Springer
International Publishing.
Slide 19
Slide 19 text
Model Reduction
y
Slide 20
Slide 20 text
Model Inference Experimentation
10 million production messages (20 days)
y
161,035 traces
y
S R(S)
77,058 branches 1,587 branches
43,536 branches 1,585 branches
q It took 6 minutes to build the two models.
Slide 21
Slide 21 text
In Depth Testing
Slide 22
Slide 22 text
Offline Passive Testing
Two
implementation relations
:
Trace preorder
relation
and our own
weaker
implementation relation
Our testing algorithm relies on both to give verdicts
Partial models = No Fail verdict
Slide 23
Slide 23 text
The Need for a Weaker Impl. Relation
"Since I know that my model is not complete, I am willing
to accept not standard behaviours till a certain point."
Slide 24
Slide 24 text
Experimentation
SUA: 53,996 traces
SUT: 25,047 traces
y
98% are Pass traces. The remaining 2% are new
behaviours that never occured before.
q It took 10 minutes to check conformance.
Slide 25
Slide 25 text
Now, What?
2% still represents many traces, and can contain many false
positive. For Michelin engineers, it is still "better than nothing".
Larger sets of traces to build the models should reduce the
number of false positive
But we should find a way to refine this possibly fail trace set
Slide 26
Slide 26 text
Conclusion
Fast passive testing framework for a specific context
Model inference: the more production messages, the better!
Testing: still too many possibly fail traces
Slide 27
Slide 27 text
Future Work
Online
passive testing (just-in-time fault detection?)
Active testing
by leveraging the inferred models again
Developing a way to focus on specific parts of the system