The Quest for Average Response Time
Tom Henzinger
IST Austria
Joint work with Krishnendu Chatterjee and Jan Otop
Slide 2
Slide 2 text
Yes/No
Program Analysis
Property
Program
Formal Verification
Slide 3
Slide 3 text
Yes/No
Model Checker
Transition
System (r ) } g)
Property
Program
Formal Verification
Slide 4
Slide 4 text
Yes/No
Model Checker
Timed
Automaton (r ) }· 5
g)
Quantitative
Property
Program
Formal Verification
Slide 5
Slide 5 text
Yes/No
Model Checker
Markov
Process
Quantitative
Property
Program 8 (r ) Pr(}g) ¸ 0.5)
Formal Verification
Slide 6
Slide 6 text
Model Checker
Timed
Automaton Program (r ) } g)
Property
Quantitative Answer R
Worst or average response time
Quantitative Analysis
Slide 7
Slide 7 text
Quantitative Analysis
From checking correctness
to measuring performance and robustness of software systems:
Quantitative temporal logics
Quantitative automata
Quantitative abstractions
Quantitative synthesis
etc.
Slide 8
Slide 8 text
Quantitative Analysis
From checking correctness
to measuring performance and robustness of software systems:
Quantitative temporal logics
Quantitative automata
Quantitative abstractions
Quantitative synthesis
etc.
None of this has captured average response time.
Slide 9
Slide 9 text
Observations
r request
g grant
t tick
x neither
§ = {r,g,t,x}
Slide 10
Slide 10 text
Program Behavior = Observation Sequence
x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x …
Slide 11
Slide 11 text
Response Times
x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x … 4,3,2
Slide 12
Slide 12 text
Response Times
x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x … 4,3,2
r t t t t t t t … 1
Slide 13
Slide 13 text
Response Property
r
g
r,t,x
g,t,x
Slide 14
Slide 14 text
Response Monitor
r
r,t,x
g,t,x
S
S
g
Slide 15
Slide 15 text
Response Monitor
r
r,t,x
g,t,x
S
S
g
Decomposing model checking [Pnueli et al.]
Alternating automata
Run-time verification
Slide 16
Slide 16 text
Bounded Response
r
g
r,x
g,t,x
C := 0
C · 3
t
C := C+1
Slide 17
Slide 17 text
Bounded Response
r
g
r,x
g,t,x
C := 0
C · 3
t
C := C+1
g
C > 3
(Discrete) clocks exponentially succinct,
but not more expressive than finite state.
Slide 18
Slide 18 text
Bounded Response Monitor
r r,x
g,t,x
S
S
g
t
C := C+1
C := 0
C · 3
Slide 19
Slide 19 text
Maximal Response
r
g r,x
g,t,x
C := 0
V := max(V,C)
t
C := C+1
V := 0
Slide 20
Slide 20 text
Maximal Response
r
g r,x
g,t,x
C := 0
V := max(V,C)
t
C := C+1
V := 0
Value of an infinite run is liminf of V.
Slide 21
Slide 21 text
Maximal Response Monitor
r r,x
g,t,x
S
S
g
t
V := V+1
V := 0
V := 0
V := max(V, )
is final value of V.
S
Slide 22
Slide 22 text
Average Response
r
g r,x
g,t,x
C := 0
N := N+1
V := avg(V,C,N)
t
C := C+1
V := 0
N := 0
avg(V,C,N) = (V¢(N-1)+C) / N
Slide 23
Slide 23 text
Average Response Monitor
r r,x
g,t,x
S
S
g
t
V := V+1
V := 0
V := avg(V, ,N)
N := N+1
V := 0
N := 0
Slide 24
Slide 24 text
(max,inc) automata:
Master automaton maintains the max of values
returned by slaves (1 max register).
Each slave automaton counts occurrences of t
(1 inc register).
(avg,inc) automata:
Master automaton maintains the avg of values
returned by slaves (1 avg register).
Slaves as above.
Slide 25
Slide 25 text
Nested Weighted Automata
r r,x
0
g,t,x
0
S
g
0
t
1
S
limavg of weights
sum of weights
Function on weights instead of registers.
Slide 26
Slide 26 text
Unlike in the qualitative case,
nested weighted automata (“quantitative monitors”) are
more expressive than flat weighted automata:
1. value of flat limavg automaton bounded by largest weight
(cannot specify average response time)
2. flat automata have constant “width” (number of registers)
Slide 27
Slide 27 text
Deterministic qualitative automaton A: §! ! B
Deterministic quantitative automaton A: §! ! R
Slide 28
Slide 28 text
Deterministic qualitative automaton A: §! ! B
Deterministic quantitative automaton A: §! ! R
! = x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x …
Response(!) = 1
BoundedResponse(!) = 0
MaximalResponse(!) = 4
AverageResponse(!) = 3
4
3 2
Slide 29
Slide 29 text
t
r,g,t,x
Nondeterministic Automaton
t
t t t t t t t t t … values {0, 1}
Slide 30
Slide 30 text
Nondeterministic qualitative automaton A: §! ! B
A(!) = max{ value(½) | ½ run of A and obs(½) = ! }
Nondeterministic quantitative automaton A: §! ! R
A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }
Slide 31
Slide 31 text
Nondeterministic qualitative automaton A: §! ! B
A(!) = max{ value(½) | ½ run of A and obs(½) = ! }
Emptiness: 9!. A(!) = 1
Universality: 8!. A(!) = 1
Nondeterministic quantitative automaton A: §! ! R
A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }
Slide 32
Slide 32 text
Nondeterministic qualitative automaton A: §! ! B
A(!) = max{ value(½) | ½ run of A and obs(½) = ! }
Emptiness: 9!. A(!) = 1
Universality: 8!. A(!) = 1
Nondeterministic quantitative automaton A: §! ! R
A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }
Emptiness: 9!. A(!) ¸ ¸
Universality: 8!. A(!) ¸ ¸
Slide 33
Slide 33 text
Transition System = Labeled Graph
r
r
t
x
x
x
t
g
g
g
t
t
t
t
x
x
t
Defines a set of behaviors.
Slide 34
Slide 34 text
Qualitative Analysis
Given a transition system A and a qualitative property B,
Q1. does some run of A correspond to a run of B ?
[emptiness of A £ B ]
Q2. does every run of A correspond to a run of B ?
[as hard as universality of B ]
Slide 35
Slide 35 text
Quantitative Analysis
Given a transition system A and a quantitative property B,
Q1. does some run of A correspond to a run of B with value V ¸ ¸ ?
[emptiness of A £ B ]
Q2. does every run of A correspond to a run of B with V ¸ ¸ ?
[as hard as universality of B ]
Slide 36
Slide 36 text
Qualitative Analysis
Given a transition system A and a qualitative property B,
Q1. does some run of A correspond to a run of B ?
[emptiness of A £ B ]
Q2. does every run of A correspond to a run of B ?
Equivalently: does some run of A correspond to a run of :B ?
[emptiness of A £ :B ]
Slide 37
Slide 37 text
Qualitative Analysis
Given a transition system A and a qualitative property B,
Q1. does some run of A correspond to a run of B ?
[emptiness of A £ B ]
Q2. does every run of A correspond to a run of B ?
Equivalently: does some run of A correspond to a run of :B ?
[emptiness of A £ :B ]
For deterministic B, the complement :B is easy to compute.
Slide 38
Slide 38 text
Nondeterministic quantitative automaton A: §! ! R
A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }
Monitor: obs(½1
) = obs(½2
) ) value(½1
) = value(½2
)
Deterministic automata are monitors.
Slide 39
Slide 39 text
Quantitative Analysis
Given a transition system A and a quantitative property B,
Q1. does some run of A correspond to a run of B with value V ¸ ¸ ?
[emptiness of A £ B ]
Q2. does every run of A correspond to a run of B with V ¸ ¸ ?
For monitor B, equivalently:
does some run of A correspond to a run of B with V < ¸ ?
[emptiness of A £ -B ]
Slide 40
Slide 40 text
Example
r r
t
t
t
t
t
r
g
g
g
t
t
t
Slide 41
Slide 41 text
Example
r r
t
t
t
t
t
r
g
g
g
t
t
t
Best maximal response time: 2
Worst maximal response time: 3
Emptiness of (max,inc) automata
Slide 42
Slide 42 text
Example
r r
t
t
t
t
t
r
g
g
g
t
t
t
Best maximal response time: 2
Worst maximal response time: 3
Emptiness of (max,inc) automata
Best average response time: 1.5
Worst average response time: 3
Emptiness of (avg,inc) automata
Probabilistic System = Markov Chain
r
r
t
x
x
x
t
g
g
g
t
t
t
t
x
x
t
0.5 0.3
0.2
0.5
0.5
Defines probability for every
finite observation sequence,
and prob density function on
infinite observation sequences.
0.9
0.1
Slide 47
Slide 47 text
Probabilistic System = Markov Chain
r
r
t
x
x
x
t
g
g
g
t
t
t
t
x
x
t
0.5 0.3
0.2
0.5
0.5
Defines probability for every
finite observation sequence,
and prob density function on
infinite observation sequences.
0.9
0.1
Given prob density function on §!,
monitor specifies random variable V.
Slide 48
Slide 48 text
Probabilistic Analysis
Given a probabilistic system A and a functional quantitative property B,
Q1. compute the expected value of V on the runs of A £ B
[moment analysis]
Q2. compute the probability of V ¸ ¸ on the runs of A £ B
[distribution analysis]
Slide 49
Slide 49 text
Probabilistic Example
r r
t
t
t
t
t
r
g
g
g
t
t
t
0.5 0.5
Slide 50
Slide 50 text
Probabilistic Example
r r
t
t
t
t
t
r
g
g
g
t
t
t
Expected maximal response time: 2.5
Prob of maximal response time at most 2: 0.5
Probabilistic analysis of (max,inc) automata
0.5 0.5
Slide 51
Slide 51 text
Probabilistic Example
r r
t
t
t
t
t
r
g
g
g
t
t
t
Expected maximal response time: 2.5
Prob of maximal response time at most 2: 0.5
Probabilistic analysis of (max,inc) automata
Expected average response time: 2.25
Prob of average response time at most 2: 0.5
Probabilistic analysis of (avg,inc) automata
0.5 0.5
Markov Decision Process
r
r
t
x
x
x
t
g
g
g
t
t
t
t
x
x
t
0.5
0.5
0.9
0.1
Slide 55
Slide 55 text
Markov Decision Process
r
r
t
x
x
x
t
g
g
g
t
t
t
t
x
x
t
0.5
0.5
0.9
0.1
Given a policy p:§!!{ $,$,$ },
monitor specifies random
variable V.
Slide 56
Slide 56 text
Many Open Questions …
E.g., given an MDP A and a monitor B,
compute the policy that maximizes the expected value of B on A
(generalization of mean-payoff game).
Slide 57
Slide 57 text
r t r t r t t t g t g t g
5,5,5
Matching Requests and Grants
Slide 58
Slide 58 text
r t r t r t t t g t g t g
5,5,5
7,5,3
Matching Requests and Grants
Quantitative pushdown monitors?
Slide 59
Slide 59 text
Counter Machine
r
x
g,t,x
S
S
g
C = 0
t
V := V+1
V := 0
C := 0
V := 0
V := max(V, )
r
C := C+1
g
C > 0
C := C-1
Emptiness for two counters is in general undecidable.
Slide 60
Slide 60 text
Counter Monitor
t
x
r,g,x
S
S
t
V := 0
V := 0
V := max(V, )
r
V := V+1
g
V := V-1
No test on counter values.
Slide 61
Slide 61 text
Counter Monitor
t
x
r,g,x
S
S
t
V := 0
V := 0
V := max(V, )
r
V := V+1
g
V := V-1
No test on counter values. width = 1
Slide 62
Slide 62 text
Register Automaton
x
V := 0
C := 0
r
C := C+1
g
C := C-1
Emptiness of (max,inc+dec) decidable [flat, constant width: Alur et al.].
t
V := max(V,C)
C := 0
Slide 63
Slide 63 text
Results on (max,inc+dec) Automata
Nondet Monitor
(max,inc+dec) (max,inc+dec)
Emptiness PSPACE PSPACE
Universality undecidable undecidable
Expectation undecidable
Probability undecidable
Slide 64
Slide 64 text
Results on (avg,inc+dec) Automata
Nondet Monitor
(avg,inc+dec) (avg,inc+dec)
Emptiness open open
Universality undecidable open
Expectation PTIME
Probability PTIME
Slide 65
Slide 65 text
Quantitative Monitors = Nested Weighted Automata
Unbounded width allows for natural decomposition of specifications
(incl. average response time).
More expressive and more succinct than flat weighted automata.
Emptiness decidable and sufficient for
monitor verification, model measuring, and model repair
(universality can be undecidable, even for constant width).
Probabilistic analysis polynomial for (avg,inc+dec) monitors.
Slide 66
Slide 66 text
Model Measuring:
How much can system A be perturbed without violating qualitative property B ?
Model Repair:
How much must system A be changed to satisfy qualitative property B ?
Slide 67
Slide 67 text
Model Measuring:
How much can system A be perturbed without violating qualitative property B ?
For an observation sequence ! we can define
distance d(A,!) (e.g. edit distance) by constructing from A
monitor FA
such that FA
(!) = d(A,!).
Robustness of A with respect to B:
r(A,B) = sup{ e | 8!. d(A,!) · e ) B(!) = 1 }.
Model Repair:
How much must system A be changed to satisfy qualitative property B ?