TMPA-2017: The Quest for Average Response Time

TMPA-2017: Tools and Methods of Program Analysis
3-4 March, 2017, Hotel Holiday Inn Moscow Vinogradovo, Moscow

The Quest for Average Response Time
Thomas A. Henzinger (President, IST, Austria Institute of Science and Technology)

For video follow the link: https://youtu.be/bCMj2toH1b4
Would like to know more?
Visit our website:
www.tmpaconf.org
www.exactprosystems.com/events/tmpa

Follow us:
https://www.linkedin.com/company/exactpro-systems-llc?trk=biz-companies-cym
https://twitter.com/exactpro

Transcript

1. The Quest for Average Response Time Tom Henzinger IST Austria Joint work with Krishnendu Chatterjee and Jan Otop
2. Yes/No Program Analysis Property Program Formal Verification
3. Yes/No Model Checker Transition System  (r ) } g) Property Program Formal Verification
4. Yes/No Model Checker Timed Automaton  (r ) }· 5 g) Quantitative Property Program Formal Verification
5. Yes/No Model Checker Markov Process Quantitative Property Program 8 (r ) Pr(}g) ¸ 0.5) Formal Verification
6. Model Checker Timed Automaton Program  (r ) } g) Property Quantitative Answer R Worst or average response time Quantitative Analysis
7. Quantitative Analysis From checking correctness to measuring performance and robustness of software systems: Quantitative temporal logics Quantitative automata Quantitative abstractions Quantitative synthesis etc.
8. Quantitative Analysis From checking correctness to measuring performance and robustness of software systems: Quantitative temporal logics Quantitative automata Quantitative abstractions Quantitative synthesis etc. None of this has captured average response time.
9. Observations r request g grant t tick x neither § = {r,g,t,x}
10. Program Behavior = Observation Sequence x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x …
11. Response Times x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x … 4,3,2
12. Response Times x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x … 4,3,2 r t t t t t t t … 1
13. Response Property r g r,t,x g,t,x
14. Response Monitor r r,t,x g,t,x S S g
15. Response Monitor r r,t,x g,t,x S S g Decomposing model checking [Pnueli et al.] Alternating automata Run-time verification
16. Bounded Response r g r,x g,t,x C := 0 C · 3 t C := C+1
17. Bounded Response r g r,x g,t,x C := 0 C · 3 t C := C+1 g C > 3 (Discrete) clocks exponentially succinct, but not more expressive than finite state.
18. Bounded Response Monitor r r,x g,t,x S S g t C := C+1 C := 0 C · 3
19. Maximal Response r g r,x g,t,x C := 0 V := max(V,C) t C := C+1 V := 0
20. Maximal Response r g r,x g,t,x C := 0 V := max(V,C) t C := C+1 V := 0 Value of an infinite run is liminf of V.
21. Maximal Response Monitor r r,x g,t,x S S g t V := V+1 V := 0 V := 0 V := max(V, ) is final value of V. S
22. Average Response r g r,x g,t,x C := 0 N := N+1 V := avg(V,C,N) t C := C+1 V := 0 N := 0 avg(V,C,N) = (V¢(N-1)+C) / N
23. Average Response Monitor r r,x g,t,x S S g t V := V+1 V := 0 V := avg(V, ,N) N := N+1 V := 0 N := 0
24. (max,inc) automata: Master automaton maintains the max of values returned by slaves (1 max register). Each slave automaton counts occurrences of t (1 inc register). (avg,inc) automata: Master automaton maintains the avg of values returned by slaves (1 avg register). Slaves as above.
25. Nested Weighted Automata r r,x 0 g,t,x 0 S g 0 t 1 S limavg of weights sum of weights Function on weights instead of registers.
26. Unlike in the qualitative case, nested weighted automata (“quantitative monitors”) are more expressive than flat weighted automata: 1. value of flat limavg automaton bounded by largest weight (cannot specify average response time) 2. flat automata have constant “width” (number of registers)
27. Deterministic qualitative automaton A: §! ! B Deterministic quantitative automaton A: §! ! R
28. Deterministic qualitative automaton A: §! ! B Deterministic quantitative automaton A: §! ! R ! = x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x … Response(!) = 1 BoundedResponse(!) = 0 MaximalResponse(!) = 4 AverageResponse(!) = 3 4 3 2
29. t r,g,t,x Nondeterministic Automaton t t t t t t t t t t … values {0, 1}
30. Nondeterministic qualitative automaton A: §! ! B A(!) = max{ value(½) | ½ run of A and obs(½) = ! } Nondeterministic quantitative automaton A: §! ! R A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }
31. Nondeterministic qualitative automaton A: §! ! B A(!) = max{ value(½) | ½ run of A and obs(½) = ! } Emptiness: 9!. A(!) = 1 Universality: 8!. A(!) = 1 Nondeterministic quantitative automaton A: §! ! R A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }
32. Nondeterministic qualitative automaton A: §! ! B A(!) = max{ value(½) | ½ run of A and obs(½) = ! } Emptiness: 9!. A(!) = 1 Universality: 8!. A(!) = 1 Nondeterministic quantitative automaton A: §! ! R A(!) = sup{ value(½) | ½ run of A and obs(½) = ! } Emptiness: 9!. A(!) ¸ ¸ Universality: 8!. A(!) ¸ ¸
33. Transition System = Labeled Graph r r t x x x t g g g t t t t x x t Defines a set of behaviors.
34. Qualitative Analysis Given a transition system A and a qualitative property B, Q1. does some run of A correspond to a run of B ? [emptiness of A £ B ] Q2. does every run of A correspond to a run of B ? [as hard as universality of B ]
35. Quantitative Analysis Given a transition system A and a quantitative property B, Q1. does some run of A correspond to a run of B with value V ¸ ¸ ? [emptiness of A £ B ] Q2. does every run of A correspond to a run of B with V ¸ ¸ ? [as hard as universality of B ]
36. Qualitative Analysis Given a transition system A and a qualitative property B, Q1. does some run of A correspond to a run of B ? [emptiness of A £ B ] Q2. does every run of A correspond to a run of B ? Equivalently: does some run of A correspond to a run of :B ? [emptiness of A £ :B ]
37. Qualitative Analysis Given a transition system A and a qualitative property B, Q1. does some run of A correspond to a run of B ? [emptiness of A £ B ] Q2. does every run of A correspond to a run of B ? Equivalently: does some run of A correspond to a run of :B ? [emptiness of A £ :B ] For deterministic B, the complement :B is easy to compute.
38. Nondeterministic quantitative automaton A: §! ! R A(!) = sup{ value(½) | ½ run of A and obs(½) = ! } Monitor: obs(½1 ) = obs(½2 ) ) value(½1 ) = value(½2 ) Deterministic automata are monitors.
39. Quantitative Analysis Given a transition system A and a quantitative property B, Q1. does some run of A correspond to a run of B with value V ¸ ¸ ? [emptiness of A £ B ] Q2. does every run of A correspond to a run of B with V ¸ ¸ ? For monitor B, equivalently: does some run of A correspond to a run of B with V < ¸ ? [emptiness of A £ -B ]
40. Example r r t t t t t r g g g t t t
41. Example r r t t t t t r g g g t t t Best maximal response time: 2 Worst maximal response time: 3 Emptiness of (max,inc) automata
42. Example r r t t t t t r g g g t t t Best maximal response time: 2 Worst maximal response time: 3 Emptiness of (max,inc) automata Best average response time: 1.5 Worst average response time: 3 Emptiness of (avg,inc) automata
43. Results on (max,inc) Automata Nondet Monitor (max,inc) (max,inc) Emptiness PSPACE PSPACE Universality · EXPSPACE PSPACE ¸ PSPACE Qualitative nondeterministic universality PSPACE, emptiness and deterministic universality PTIME.
44. Results on (avg,inc) Automata Nondet Monitor (avg,inc) (avg,inc) Emptiness · EXPSPACE · EXPSPACE ¸ PSPACE ¸ PSPACE Universality undecidable · EXPSPACE ¸ PSPACE
45. Results on (avg,inc) Automata Nondet Monitor Monitor Monitor (avg,inc) (avg,inc) bounded width constant width (avg,inc) (avg,inc) Emptiness · EXPSPACE · EXPSPACE PSPACE PTIME ¸ PSPACE ¸ PSPACE Universality undecidable · EXPSPACE PSPACE PTIME ¸ PSPACE How many overlapping requests?
46. Probabilistic System = Markov Chain r r t x x x t g g g t t t t x x t 0.5 0.3 0.2 0.5 0.5 Defines probability for every finite observation sequence, and prob density function on infinite observation sequences. 0.9 0.1
47. Probabilistic System = Markov Chain r r t x x x t g g g t t t t x x t 0.5 0.3 0.2 0.5 0.5 Defines probability for every finite observation sequence, and prob density function on infinite observation sequences. 0.9 0.1 Given prob density function on §!, monitor specifies random variable V.
48. Probabilistic Analysis Given a probabilistic system A and a functional quantitative property B, Q1. compute the expected value of V on the runs of A £ B [moment analysis] Q2. compute the probability of V ¸ ¸ on the runs of A £ B [distribution analysis]
49. Probabilistic Example r r t t t t t r g g g t t t 0.5 0.5
50. Probabilistic Example r r t t t t t r g g g t t t Expected maximal response time: 2.5 Prob of maximal response time at most 2: 0.5 Probabilistic analysis of (max,inc) automata 0.5 0.5
51. Probabilistic Example r r t t t t t r g g g t t t Expected maximal response time: 2.5 Prob of maximal response time at most 2: 0.5 Probabilistic analysis of (max,inc) automata Expected average response time: 2.25 Prob of average response time at most 2: 0.5 Probabilistic analysis of (avg,inc) automata 0.5 0.5
52. Results on (max,inc) Automata Nondet Monitor (max,inc) (max,inc) Emptiness PSPACE PSPACE Universality · EXPSPACE PSPACE ¸ PSPACE Expectation · EXPSPACE ¸ PSPACE Probability · EXPSPACE ¸ PSPACE
53. Results on (avg,inc) Automata Nondet Monitor (avg,inc) (avg,inc) Emptiness · EXPSPACE · EXPSPACE ¸ PSPACE ¸ PSPACE Universality undecidable · EXPSPACE ¸ PSPACE Expectation PTIME Probability PTIME
54. Markov Decision Process r r t x x x t g g g t t t t x x t 0.5 0.5 0.9 0.1
55. Markov Decision Process r r t x x x t g g g t t t t x x t 0.5 0.5 0.9 0.1 Given a policy p:§!!{ $,$,$ }, monitor specifies random variable V.
56. Many Open Questions … E.g., given an MDP A and a monitor B, compute the policy that maximizes the expected value of B on A (generalization of mean-payoff game).
57. r t r t r t t t g t g t g 5,5,5 Matching Requests and Grants
58. r t r t r t t t g t g t g 5,5,5 7,5,3 Matching Requests and Grants Quantitative pushdown monitors?
59. Counter Machine r x g,t,x S S g C = 0 t V := V+1 V := 0 C := 0 V := 0 V := max(V, ) r C := C+1 g C > 0 C := C-1 Emptiness for two counters is in general undecidable.
60. Counter Monitor t x r,g,x S S t V := 0 V := 0 V := max(V, ) r V := V+1 g V := V-1 No test on counter values.
61. Counter Monitor t x r,g,x S S t V := 0 V := 0 V := max(V, ) r V := V+1 g V := V-1 No test on counter values. width = 1
62. Register Automaton x V := 0 C := 0 r C := C+1 g C := C-1 Emptiness of (max,inc+dec) decidable [flat, constant width: Alur et al.]. t V := max(V,C) C := 0
63. Results on (max,inc+dec) Automata Nondet Monitor (max,inc+dec) (max,inc+dec) Emptiness PSPACE PSPACE Universality undecidable undecidable Expectation undecidable Probability undecidable
64. Results on (avg,inc+dec) Automata Nondet Monitor (avg,inc+dec) (avg,inc+dec) Emptiness open open Universality undecidable open Expectation PTIME Probability PTIME
65. Quantitative Monitors = Nested Weighted Automata Unbounded width allows for natural decomposition of specifications (incl. average response time). More expressive and more succinct than flat weighted automata. Emptiness decidable and sufficient for monitor verification, model measuring, and model repair (universality can be undecidable, even for constant width). Probabilistic analysis polynomial for (avg,inc+dec) monitors.
66. Model Measuring: How much can system A be perturbed without violating qualitative property B ? Model Repair: How much must system A be changed to satisfy qualitative property B ?
67. Model Measuring: How much can system A be perturbed without violating qualitative property B ? For an observation sequence ! we can define distance d(A,!) (e.g. edit distance) by constructing from A monitor FA such that FA (!) = d(A,!). Robustness of A with respect to B: r(A,B) = sup{ e | 8!. d(A,!) · e ) B(!) = 1 }. Model Repair: How much must system A be changed to satisfy qualitative property B ?
68. References Nested Weighted Automata: LICS 2015 Quantitative Automata under Probabilistic Semantics: LICS 2016 Nested Weighted Automata of Bounded Width: MFCS 2016
69. IST (Institute of Science and Technology) Austria looking for PhD students: phd.ist.ac.at