Habilitation exam: seminar

The second part of the habilitation exam consisted of a presentation -- and discussion -- of a seminar on my research topics. This is the slide deck.

https://www.ulisboa.pt/prova-academica/measuring-and-improving-software-fault-diagnosis

Transcript

1. Measuring and Improving SW Fault Diagnosis @rmaranhao —Seminar— Rui Abreu (aka Rui Maranhão) habilitation exam
2. Influencers Arjan. J.C. van Gemund Joost Visser João Saraiva
3. Influencers
4. Predict and Recover from SW Failures Influencers
5. Influencers Fault Localization?
6. Influencers
7. Influencers
8. Influencers Not necessarily an exhaustive list… Not necessarily an exhaustive list…
9. Relevance
10. Relevance
11. Relevance
12. Relevance
13. Relevance
14. Estimated 20% downtime due to fault diagnosis Relevance
15. History
16. Software Errors mentioned in Ada Byron’s notes on Charles Bababage’s analytical engine 1840
17. First actual bug and actual debugging: Admiral Grace Hopper’s associates working on Mark II Computer at Harvard University 1947
18. UNIVAC 1100 FLIT: Fault Location by Interpretive Testing 1962
19. Weiser’s Breakthrough paper. Input: source code and program point 1981 Static Slicing
20. Weiser’s Breakthrough paper. Input: source code and program point 1981 Static Slicing
21. Weiser’s Breakthrough paper. Input: source code and program point 1981 Static Slicing
22. Weiser’s Breakthrough paper. Input: source code and program point 1981 Static Slicing
23. Stallman’s GDB Input: faulty program and 1 failed test case 1986
24. 1987 Model-based Diagnosis
25. Korel and Laski’s dynamic slicing Input: source code and failed test case 1988 Dynamic Slicing int i; int sum = 0; int product = 1; for(i = 0; i < N; ++i) sum = sum + i; product = product *i; } write(sum); write(product); int i; int sum = 0; int product = 1; for(i = 0; i < N; ++i) sum = sum + i; product = product *i; } write(sum); write(product); i = 0
26. DDD Input: faulty program and failed test case 1993
27. Input: faulty program, 1 failed and 1 passed test case 1996 Delta Debugging
28. Input: faulty program, test suite 2002-… Statistical Debugging
29. EZUNIT 2007
30. 2008-2009
31. 2008-2009
32. 2008-2009
33. VIDA 2009
41. • Techniques that take into account spectra • aka abstraction of program traces • Spectrum-based Fault Localization (SFL) • Statistical vs. reasoning • Lightweight, scalable This talk 
 — Measuring and Improving SW Fault Diagnosis — Focus
42. SFL: Principle (1) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 6 7 8 1 2 3 4 9 10 11 12 1 2 3 5 4 6 7 8 9 10 11 12 Not touched Touched, pass Touched, fail Test suite t1 t2 t3 t4 t5 Integrates well with testing
43. SFL: Principle (2) Test suite t2 t3 t4 t5 0 1 1 1 1 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 5 6 7 8 1 2 3 4 9 10 11 12 1 2 3 5 4 6 7 8 9 10 11 12 Not touched Touched, pass Touched, fail Status t1 ! Integrates well with testing
44. Test suite t3 t4 t5 1 2 2 2 2 1 1 0 2 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 5 6 7 8 1 2 3 4 9 10 11 12 1 2 3 5 4 6 7 8 9 10 11 12 Not touched Touched, pass Touched, fail Status t1 ! t2 ! SFL: Principle (3) Integrates well with testing
45. Test suite t4 t5 1 2 2 2 2 1 1 0 2 1 1 1 1 1 1 0 1 0 0 1 1 0 0 0 5 6 7 8 1 2 3 4 9 10 11 12 1 2 3 5 4 6 7 8 9 10 11 12 Not touched Touched, pass Touched, fail Status t1 ! t2 ! t3 " SFL: Principle (4) Integrates well with testing
46. Test suite t5 1 3 3 2 3 1 1 0 3 1 3 3 1 1 1 0 1 0 0 1 1 0 0 0 5 6 7 8 1 2 3 4 9 10 11 12 1 2 3 5 4 6 7 8 9 10 11 12 Not touched Touched, pass Touched, fail Status t1 ! t2 ! t3 " t4 ! SFL: Principle (5) Integrates well with testing
47. Test suite 1 3 3 2 3 1 1 0 3 1 3 3 1 2 2 0 2 0 0 2 2 1 0 0 5 6 7 8 1 2 3 4 9 10 11 12 1 2 3 5 4 6 7 8 9 10 11 12 Not touched Touched, pass Touched, fail Status t1 ! t2 ! t3 " t4 ! t5 " SFL: Principle (6) Integrates well with testing
48. 1 3 3 2 3 1 1 0 3 1 3 3 1 2 2 0 2 0 0 2 2 1 0 0 5 6 7 8 1 2 3 4 9 10 11 12 1 2 3 5 4 6 7 8 9 10 11 12 Not touched Touched, fail Status t1 ! t2 ! t3 " t4 ! t5 " SFL: Principle (7) Components are ranked according to the likelihood of causing detected errors Integrates well with testing
49. Fault Program Spectra Test Suite class Triangle {… static int type(int a, int b, int c) { t1 t2 t3 t4 t5 t6 Suspiciousness int type = SCALENE; 0.09998 if ( (a == b) && (b == c) ) 0.09998 type = EQUILATERAL; 0.10001 else if ( (a*a) == ((b*b) + (c*c)) ) 0.09999 type = RIGHT; 0.10001 else if ( (a == b) || (b == a) ) /* FAULT */ 0.10000 type = ISOSCELES; 0.10001 return type; } 0.09998 static double area(int a, int b, int c) { double s = (a+b+c)/2.0; 0.10000 return Math.sqrt(s*(s-a)*(s-b)*(s-c)); } ... } 0.10000 PASS/FAIL Error Vector
50. Suspiciousness score • Each component (row) is ranked according to their similarity to the error vector • Many similarity coefficients exist. • Ochiai similarity is equivalent to the cosine of the angle between two vectors in a n-dimensional space Abreu, R., Zoeteweij, P., Golsteijn, R., & Van Gemund, A. J. (2009). A practical evaluation of spectrum-based fault localization. Journal of Systems and Software, 82(11), 1780-1792. Lucia, L., Lo, D., Jiang, L., Thung, F., & Budi, A. (2014). Extended comprehensive study of association measures for fault localization. Journal of Software: Evolution and Process, 26(2).
51. Rank Position Suspicious Statement Line number Suspiciousnes s 1º type = EQUILATERAL; 3 0.10001 2º type = RIGHT; 5 0.10001 3º type = ISOSCELES; 7 0.10001 4º else if ( (a == b) || (b == a) ) /* FAULT */ 6 0.10000 5º double s = (a+b+c)/2.0; 9 0.10000 6º return Math.sqrt(s*(s-a)*(s-b)*(s-c)); 10 0.10000 7º else if ( (a*a) == ((b*b) + (c*c)) ) 4 0.09999 8º int type = SCALENE; 1 0.09998 9º if ( (a == b) && (b == c) ) 2 0.09998 10º return type; } 8 0.09998 R. Abreu, P. Zoeteweij, and A. J. van Gemund, “Spectrum-Based Multiple Fault Localization”, ASE ’09 Diagnostic Performance — aka Effort —
52. Rank Position Suspicious Statement Line number Suspiciousnes s 1º type = EQUILATERAL; 3 0.10001 2º type = RIGHT; 5 0.10001 3º type = ISOSCELES; 7 0.10001 4º else if ( (a == b) || (b == a) ) /* FAULT */ 6 0.10000 5º double s = (a+b+c)/2.0; 9 0.10000 6º return Math.sqrt(s*(s-a)*(s-b)*(s-c)); 10 0.10000 7º else if ( (a*a) == ((b*b) + (c*c)) ) 4 0.09999 8º int type = SCALENE; 1 0.09998 9º if ( (a == b) && (b == c) ) 2 0.09998 10º return type; } 8 0.09998 R. Abreu, P. Zoeteweij, and A. J. van Gemund, “Spectrum-Based Multiple Fault Localization”, ASE ’09 Diagnostic Performance — aka Effort —
53. Can we do better? • Statistics-based SFL does not reason in terms of multiple faults c1 c2 c3 P/F 1 0 0 1 (F) 0 1 0 1 (F) 1 0 1 1 (F) 0 1 1 1 (F) 1 1 0 0 (P) Diagnostic report = < c3, c1, c2 >
54. Reasoning • Barinel is a spectrum-based reasoning approach • Integrates the best of model-based diagnosis and spectra c1 c2 c3 P/F 1 0 0 1 (F) 0 1 0 1 (F) 1 0 1 1 (F) 0 1 1 1 (F) 1 1 0 0 (P) c1 must be faulty c2 cannot be single fault c3 cannot be single fault c2, c3 cannot be double fault
55. Reasoning • Barinel is a spectrum-based reasoning approach • Integrates the best of model-based diagnosis with spectra c1 c2 c3 P/F 1 0 0 1 (F) 0 1 0 1 (F) 1 0 1 1 (F) 0 1 1 1 (F) 1 1 0 0 (P) c2 must be faulty c1 cannot be single fault c3 cannot be single fault c1, c3 cannot be double fault
56. Reasoning • Barinel is a spectrum-based reasoning approach • Integrates the best of model-based diagnosis with spectra c1 c2 c3 P/F 1 0 0 1 (F) 0 1 0 1 (F) 1 0 1 1 (F) 0 1 1 1 (F) 1 1 0 0 (P) Summary: c1, c2 faulty, but not single-fault c1, c2 can be double-fault c1,c3 nor c2,c3 can be double-fault so {c1,c2} is the only diagnosis possible (subsuming the triple fault {c1,c2,c3}) R. Abreu, P. Zoeteweij, and A. J. van Gemund, “Spectrum-Based Multiple Fault Localization”, ASE ’09 **https://github.com/npcardoso/MHS2 (citable via https://zenodo.org/record/10037) ➤ contribute to the project; send pull requests; email us!
57. Best performing techniques still require to inspect 10% of the code… • 100 LOC ➤ 10LOC • 10,000 LOC ➤ 1,000LOC • 1,000,000 LOC ➤ 10,000LOC Are we there yet?
60. Case To Inspect Out of / Previous Load Problem 2 logical threads 315 Teletext Lock-Up 2 blocks 60K NVM corrupt 96 blocks, 10 files 150K, 1.8K Scrolling Bug 5 blocks 150K Invisible Pages 12 blocks 150K Tuner Problem 2 files 1.8K Zapping Crash 1 run (15 mins) 1 day (develop) Wrong Audio 1 run (15 mins) ½ day (expert) Industrial Case Studies (NXP)
61. Case To Inspect Out of / Previous Load Problem 2 logical threads 315 Teletext Lock-Up 2 blocks 60K NVM corrupt 96 blocks, 10 files 150K, 1.8K Scrolling Bug 5 blocks 150K Invisible Pages 12 blocks 150K Tuner Problem 2 files 1.8K Zapping Crash 1 run (15 mins) 1 day (develop) Wrong Audio 1 run (15 mins) ½ day (expert) Industrial Case Studies (NXP)
62. Case To Inspect Out of / Previous Load Problem 2 logical threads 315 Teletext Lock-Up 2 blocks 60K NVM corrupt 96 blocks, 10 files 150K, 1.8K Scrolling Bug 5 blocks 150K Invisible Pages 12 blocks 150K Tuner Problem 2 files 1.8K Zapping Crash 1 run (15 mins) 1 day (develop) Wrong Audio 1 run (15 mins) ½ day (expert) Industrial Case Studies (NXP)
63. • Are we properly quantifying diagnostic accuracy? • Is there an ecosystem offering these techniques? So… are we there?
64. http://www.coloradd.net GZoltar — www.gzoltar.com —
66. Visualizations
67. Visualizations
68. Visualizations
69. • 40 participants • Intention: GZoltar vs. IDE’s features • Program: Xtream • 17,389 LOC • 306 classes and 22 packages • 1418 unit test cases • Injected 1 logical fault Gouveia, C., Campos, J., & Abreu, R.. Using HTML5 visualizations in software fault localization. VISSOFT’13 User Study: Setup
70. RQ1: Do the proposed visualizations efficiently aid the user to quickly find a fault? User Study: Results
71. class Triangle {… static int type(int a, int b, int c) { t1 t2 t3 t4 t5 t6 Suspiciousness int type = SCALENE; 0.09998 if ( (a == b) && (b == c) ) 0.09998 type = EQUILATERAL; 0.10001 else if ( (a*a) == ((b*b) + (c*c)) ) 0.09999 type = RIGHT; 0.10001 else if ( (a == b) || (b == a) ) /* FAULT */ 0.10000 type = ISOSCELES; 0.10001 return type; } 0.09998 static double area(int a, int b, int c) { double s = (a+b+c)/2.0; 0.10000 return Math.sqrt(s*(s-a)*(s-b)*(s-c)); } ... } 0.10000 “A confounding factor for the usefulness of SFL is the dependency on the quality of the existing test suite” Importance of Testing
72. Rank Position Suspicious Statement Line number Suspiciousnes s 1º type = EQUILATERAL; 3 0.10001 2º type = RIGHT; 5 0.10001 3º type = ISOSCELES; 7 0.10001 4º else if ( (a == b) || (b == a) ) /* FAULT */ 6 0.10000 5º double s = (a+b+c)/2.0; 9 0.10000 6º return Math.sqrt(s*(s-a)*(s-b)*(s-c)); 10 0.10000 7º else if ( (a*a) == ((b*b) + (c*c)) ) 4 0.09999 8º int type = SCALENE; 1 0.09998 9º if ( (a == b) && (b == c) ) 2 0.09998 10º return type; } 8 0.09998 Diagnostic Performance — aka Effort — R. Abreu, P. Zoeteweij, and A. J. van Gemund, “Spectrum-Based Multiple Fault Localization”, ASE ’09
73. Rank Position Suspicious Statement Line number Suspiciousnes s 1º type = EQUILATERAL; 3 0.10001 2º type = RIGHT; 5 0.10001 3º type = ISOSCELES; 7 0.10001 4º else if ( (a == b) || (b == a) ) /* FAULT */ 6 0.10000 5º double s = (a+b+c)/2.0; 9 0.10000 6º return Math.sqrt(s*(s-a)*(s-b)*(s-c)); 10 0.10000 7º else if ( (a*a) == ((b*b) + (c*c)) ) 4 0.09999 8º int type = SCALENE; 1 0.09998 9º if ( (a == b) && (b == c) ) 2 0.09998 10º return type; } 8 0.09998 Diagnostic Performance — aka Effort — R. Abreu, P. Zoeteweij, and A. J. van Gemund, “Spectrum-Based Multiple Fault Localization”, ASE ’09
74. The variety of test cases is the major factor to reduce uncertainty in the ranking.
75. class Triangle {… static int type(int a, int b, int c) { t1 t2 t3 t4 t5 t6 Suspiciousness int type = SCALENE; 0.09998 if ( (a == b) && (b == c) ) 0.09998 type = EQUILATERAL; 0.10001 else if ( (a*a) == ((b*b) + (c*c)) ) 0.09999 type = RIGHT; 0.10001 else if ( (a == b) || (b == a) ) /* FAULT */ 0.10000 type = ISOSCELES; 0.10001 return type; } 0.09998 static double area(int a, int b, int c) { double s = (a+b+c)/2.0; 0.10000 return Math.sqrt(s*(s-a)*(s-b)*(s-c)); } ... } 0.10000 Density of a Test Suite A. Gonzalez-Sanchez, R. Abreu, H.-G. Gross, and A. J. van Gemund, “Prioritizing tests for fault localization through ambiguity group reduction”, ASE ’11
76. class Triangle {… static int type(int a, int b, int c) { t1 t2 t3 t4 t5 t6 Suspiciousness int type = SCALENE; 0.09998 if ( (a == b) && (b == c) ) 0.09998 type = EQUILATERAL; 0.10001 else if ( (a*a) == ((b*b) + (c*c)) ) 0.09999 type = RIGHT; 0.10001 else if ( (a == b) || (b == a) ) /* FAULT */ 0.10000 type = ISOSCELES; 0.10001 return type; } 0.09998 static double area(int a, int b, int c) { double s = (a+b+c)/2.0; 0.10000 return Math.sqrt(s*(s-a)*(s-b)*(s-c)); } ... } 0.10000 = 0.4 ¯ ⇢ Density of a Test Suite A. Gonzalez-Sanchez, R. Abreu, H.-G. Gross, and A. J. van Gemund, “Prioritizing tests for fault localization through ambiguity group reduction”, ASE ’11
77. cannot construct suitable test cases systematically, this is an ideal application for search-based software testing (SBST). In SBST, an optimization goal is formulated as a fitness function, and then efficient meta-heuristic search algorithms are guided by the fitness function to generate tests. A fitness function takes as input a candidate solution, and calculates a numerical value representing the quality of the candidate, such that there is a strict ordering. In our case, the optimal solution is a test case that leads to ¯ ⇢ = 0.5, consequently our fitness function for test case t for a given test suite T is: fitness(t) = |0.5 ¯ ⇢(T [ {t})| (9) This fitness function turns the problem into a minimization problem, i.e., the optimization aims to achieve a fitness value of 0, which is the case if a solution is found such that ¯ ⇢ = 0.5. density is ¯ ⇢ = 0.4+...+0.2 6 ! ¯ ⇢ = 0.400, i.e., the test cases yield a coverage matrix density of 40% (¯ ⇢ = 0.400). B. Optimal Coverage Matrix Density Our aim is to reduce the entropy of a test suite, and the drop in entropy is known as information gain [25]. The information gain that a (new) test case provides is determined by the reduction of the size of the top-ranked suspect set. Assuming there are |D| top-ranked suspects, a test ti with coverage density ⇢(ti) reduces the top-ranked set to |D|·⇢(ti) components if ti fails, and to |D|·(1 ⇢(ti)) if ti passes. Under these conditions, it has been previously demonstrated [24] that the information gain can be modeled as follows: IG(¯ ⇢) = ¯ ⇢ · log2 (¯ ⇢) (1 ¯ ⇢) · log2 (1 ¯ ⇢) (8) For our running example, the value of IG is equal to 0.400· log2 (0.400) (1 0.400)·log2 (1 0.400) = 0.971. The value c t T p o D d t a b t p 0,0 1,0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 IG(¯ ⇢) R. A. Johnson, “An information theory approach to diagnosis”, IRE Transactions on Reliability and Quality Control, no. 1, pp. 35–35, 1960
78. Campos, J., Abreu, R., Fraser, G., & d'Amorim, M. Entropy-based test generation for improved fault localization. ASE’13.
79. class Triangle {… static int type(int a, int b, int c) { int type = SCALENE; if ( (a == b) && (b == c) ) type = EQUILATERAL; else if ( (a*a) == ((b*b) + (c*c)) ) type = RIGHT; else if ( (a == b) || (b == a) ) /* FAULT */ type = ISOSCELES; return type; } static double area(int a, int b, int c) { double s = (a+b+c)/2.0; return Math.sqrt(s*(s-a)*(s-b)*(s-c)); } ... } Test case outcome (pass = , fail = ) T º 8 9 1 7 2 4 3 10 5 6 0,400 4,000 Suspiciousness 0.09998 0.09998 0.10001 0.09999 0.10001 0.10000 0.10001 0.09998 0.10000 0.10000 matrix. Tests Guided by the Coverage Matrix Density e matrix density ¯ ⇢ gives us a measurable goal generation. As we can measure the effect but ct suitable test cases systematically, this is an on for search-based software testing (SBST). In mization goal is formulated as a fitness function, ent meta-heuristic search algorithms are guided unction to generate tests. nction takes as input a candidate solution, and umerical value representing the quality of the h that there is a strict ordering. In our case, olution is a test case that leads to ¯ ⇢ = 0.5, our fitness function for test case t for a given fitness(t) = |0.5 ¯ ⇢(T [ {t})| (9) nction turns the problem into a minimization he optimization aims to achieve a fitness value he case if a solution is found such that ¯ ⇢ = 0.5. sed Test Suite Extension Cd
80. class Triangle {… static int type(int a, int b, int c) { int type = SCALENE; if ( (a == b) && (b == c) ) type = EQUILATERAL; else if ( (a*a) == ((b*b) + (c*c)) ) type = RIGHT; else if ( (a == b) || (b == a) ) /* FAULT */ type = ISOSCELES; return type; } static double area(int a, int b, int c) { double s = (a+b+c)/2.0; return Math.sqrt(s*(s-a)*(s-b)*(s-c)); } ... } Test case outcome (pass = , fail = ) T º 8 9 1 7 2 4 3 10 5 6 0,400 4,000 Suspiciousness 0.09998 0.09998 0.10001 0.09999 0.10001 0.10000 0.10001 0.09998 0.10000 0.10000 T + {t7} t7 1 1 1 1 1 1 1 1 0,457 2,000 º 6 7 5 1 2 8 3 4 Suspiciousness 0.03629 0.03629 0.08466 0.29033 0.17204 0.03629 0.17204 0.17204 matrix. Tests Guided by the Coverage Matrix Density e matrix density ¯ ⇢ gives us a measurable goal generation. As we can measure the effect but ct suitable test cases systematically, this is an on for search-based software testing (SBST). In mization goal is formulated as a fitness function, ent meta-heuristic search algorithms are guided unction to generate tests. nction takes as input a candidate solution, and umerical value representing the quality of the h that there is a strict ordering. In our case, olution is a test case that leads to ¯ ⇢ = 0.5, our fitness function for test case t for a given fitness(t) = |0.5 ¯ ⇢(T [ {t})| (9) nction turns the problem into a minimization he optimization aims to achieve a fitness value he case if a solution is found such that ¯ ⇢ = 0.5. sed Test Suite Extension Cd
81. class Triangle {… static int type(int a, int b, int c) { int type = SCALENE; if ( (a == b) && (b == c) ) type = EQUILATERAL; else if ( (a*a) == ((b*b) + (c*c)) ) type = RIGHT; else if ( (a == b) || (b == a) ) /* FAULT */ type = ISOSCELES; return type; } static double area(int a, int b, int c) { double s = (a+b+c)/2.0; return Math.sqrt(s*(s-a)*(s-b)*(s-c)); } ... } Test case outcome (pass = , fail = ) T º 8 9 1 7 2 4 3 10 5 6 0,400 4,000 Suspiciousness 0.09998 0.09998 0.10001 0.09999 0.10001 0.10000 0.10001 0.09998 0.10000 0.10000 T + {t7} t7 1 1 1 1 1 1 1 1 0,457 2,000 º 6 7 5 1 2 8 3 4 Suspiciousness 0.03629 0.03629 0.08466 0.29033 0.17204 0.03629 0.17204 0.17204 T + {t7, t8} t8 1 1 1 1 1 1 0,475 1,000 º 6 7 3 1 2 8 4 5 Suspiciousness 0.02354 0.02354 0.10983 0.37666 0.22320 0.02354 0.10983 0.10983 matrix. Tests Guided by the Coverage Matrix Density e matrix density ¯ ⇢ gives us a measurable goal generation. As we can measure the effect but ct suitable test cases systematically, this is an on for search-based software testing (SBST). In mization goal is formulated as a fitness function, ent meta-heuristic search algorithms are guided unction to generate tests. nction takes as input a candidate solution, and umerical value representing the quality of the h that there is a strict ordering. In our case, olution is a test case that leads to ¯ ⇢ = 0.5, our fitness function for test case t for a given fitness(t) = |0.5 ¯ ⇢(T [ {t})| (9) nction turns the problem into a minimization he optimization aims to achieve a fitness value he case if a solution is found such that ¯ ⇢ = 0.5. sed Test Suite Extension Cd
82. class Triangle {… static int type(int a, int b, int c) { int type = SCALENE; if ( (a == b) && (b == c) ) type = EQUILATERAL; else if ( (a*a) == ((b*b) + (c*c)) ) type = RIGHT; else if ( (a == b) || (b == a) ) /* FAULT */ type = ISOSCELES; return type; } static double area(int a, int b, int c) { double s = (a+b+c)/2.0; return Math.sqrt(s*(s-a)*(s-b)*(s-c)); } ... } Test case outcome (pass = , fail = ) T º 8 9 1 7 2 4 3 10 5 6 0,400 4,000 Suspiciousness 0.09998 0.09998 0.10001 0.09999 0.10001 0.10000 0.10001 0.09998 0.10000 0.10000 T + {t7} t7 1 1 1 1 1 1 1 1 0,457 2,000 º 6 7 5 1 2 8 3 4 Suspiciousness 0.03629 0.03629 0.08466 0.29033 0.17204 0.03629 0.17204 0.17204 T + {t7, t8} t8 1 1 1 1 1 1 0,475 1,000 º 6 7 3 1 2 8 4 5 Suspiciousness 0.02354 0.02354 0.10983 0.37666 0.22320 0.02354 0.10983 0.10983 T + {t7, t8, t9} t9 1 1 1 1 1 1 1 0,500 0,000 º 5 6 2 1 7 3 4 Suspiciousness 0.04347 0.04347 0.17391 0.34782 0.04347 0.17391 0.17391 matrix. Tests Guided by the Coverage Matrix Density e matrix density ¯ ⇢ gives us a measurable goal generation. As we can measure the effect but ct suitable test cases systematically, this is an on for search-based software testing (SBST). In mization goal is formulated as a fitness function, ent meta-heuristic search algorithms are guided unction to generate tests. nction takes as input a candidate solution, and umerical value representing the quality of the h that there is a strict ordering. In our case, olution is a test case that leads to ¯ ⇢ = 0.5, our fitness function for test case t for a given fitness(t) = |0.5 ¯ ⇢(T [ {t})| (9) nction turns the problem into a minimization he optimization aims to achieve a fitness value he case if a solution is found such that ¯ ⇢ = 0.5. sed Test Suite Extension Cd
83. class Triangle {… static int type(int a, int b, int c) { int type = SCALENE; if ( (a == b) && (b == c) ) type = EQUILATERAL; else if ( (a*a) == ((b*b) + (c*c)) ) type = RIGHT; else if ( (a == b) || (b == a) ) /* FAULT */ type = ISOSCELES; return type; } static double area(int a, int b, int c) { double s = (a+b+c)/2.0; return Math.sqrt(s*(s-a)*(s-b)*(s-c)); } ... } Test case outcome (pass = , fail = ) T º 8 9 1 7 2 4 3 10 5 6 0,400 4,000 Suspiciousness 0.09998 0.09998 0.10001 0.09999 0.10001 0.10000 0.10001 0.09998 0.10000 0.10000 T + {t7} t7 1 1 1 1 1 1 1 1 0,457 2,000 º 6 7 5 1 2 8 3 4 Suspiciousness 0.03629 0.03629 0.08466 0.29033 0.17204 0.03629 0.17204 0.17204 T + {t7, t8} t8 1 1 1 1 1 1 0,475 1,000 º 6 7 3 1 2 8 4 5 Suspiciousness 0.02354 0.02354 0.10983 0.37666 0.22320 0.02354 0.10983 0.10983 T + {t7, t8, t9} t9 1 1 1 1 1 1 1 0,500 0,000 º 5 6 2 1 7 3 4 Suspiciousness 0.04347 0.04347 0.17391 0.34782 0.04347 0.17391 0.17391 matrix. Tests Guided by the Coverage Matrix Density e matrix density ¯ ⇢ gives us a measurable goal generation. As we can measure the effect but ct suitable test cases systematically, this is an on for search-based software testing (SBST). In mization goal is formulated as a fitness function, ent meta-heuristic search algorithms are guided unction to generate tests. nction takes as input a candidate solution, and umerical value representing the quality of the h that there is a strict ordering. In our case, olution is a test case that leads to ¯ ⇢ = 0.5, our fitness function for test case t for a given fitness(t) = |0.5 ¯ ⇢(T [ {t})| (9) nction turns the problem into a minimization he optimization aims to achieve a fitness value he case if a solution is found such that ¯ ⇢ = 0.5. sed Test Suite Extension Cd = 0.500 ¯ ⇢
84. class Triangle {… static int type(int a, int b, int c) { int type = SCALENE; if ( (a == b) && (b == c) ) type = EQUILATERAL; else if ( (a*a) == ((b*b) + (c*c)) ) type = RIGHT; else if ( (a == b) || (b == a) ) /* FAULT */ type = ISOSCELES; return type; } static double area(int a, int b, int c) { double s = (a+b+c)/2.0; return Math.sqrt(s*(s-a)*(s-b)*(s-c)); } ... } Test case outcome (pass = , fail = ) T º 8 9 1 7 2 4 3 10 5 6 0,400 4,000 Suspiciousness 0.09998 0.09998 0.10001 0.09999 0.10001 0.10000 0.10001 0.09998 0.10000 0.10000 T + {t7} t7 1 1 1 1 1 1 1 1 0,457 2,000 º 6 7 5 1 2 8 3 4 Suspiciousness 0.03629 0.03629 0.08466 0.29033 0.17204 0.03629 0.17204 0.17204 T + {t7, t8} t8 1 1 1 1 1 1 0,475 1,000 º 6 7 3 1 2 8 4 5 Suspiciousness 0.02354 0.02354 0.10983 0.37666 0.22320 0.02354 0.10983 0.10983 T + {t7, t8, t9} t9 1 1 1 1 1 1 1 0,500 0,000 º 5 6 2 1 7 3 4 Suspiciousness 0.04347 0.04347 0.17391 0.34782 0.04347 0.17391 0.17391 matrix. Tests Guided by the Coverage Matrix Density e matrix density ¯ ⇢ gives us a measurable goal generation. As we can measure the effect but ct suitable test cases systematically, this is an on for search-based software testing (SBST). In mization goal is formulated as a fitness function, ent meta-heuristic search algorithms are guided unction to generate tests. nction takes as input a candidate solution, and umerical value representing the quality of the h that there is a strict ordering. In our case, olution is a test case that leads to ¯ ⇢ = 0.5, our fitness function for test case t for a given fitness(t) = |0.5 ¯ ⇢(T [ {t})| (9) nction turns the problem into a minimization he optimization aims to achieve a fitness value he case if a solution is found such that ¯ ⇢ = 0.5. sed Test Suite Extension Cd = 0.0 Cd = 0.500 ¯ ⇢
85. Density is not enough: diversity and uniqueness also important!
86. c1 c2 c3 c4 t1 1 1 0 0 t2 0 0 1 1 t3 0 1 0 0 t4 0 1 0 0 Ideal density… alas ⇢0 = 1.0 <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit>
87. c1 c2 c3 c4 t1 1 1 0 0 t2 0 0 1 1 t3 0 1 0 0 t4 0 1 0 0 Ideal density… alas ⇢0 = 1.0 <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> ⇢0 = 1 |1 2 · ¯ ⇢| <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> Normalized density:
88. c1 c2 c3 c4 t1 1 1 0 0 t2 0 0 1 1 t3 0 1 0 0 t4 0 1 0 0 Ideal density… alas ⇢0 = 1.0 <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> ⇢0 = 1 |1 2 · ¯ ⇢| <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> Normalized density:
89. c1 c2 c3 c4 t1 1 1 0 0 t2 0 0 1 1 t3 0 1 0 0 t4 0 1 0 0 Ideal density… alas SFL will no further indict/exonerate candidates! ⇢0 = 1.0 <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> <latexit sha1_base64="4l6MyAjVH52w1fb5VEuuvYTvvMs=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbRU0hE0ItQ9OKxgrWFNJTNdtMu3WTD7kQopT/DiwdFvPprvPlv3LY5aOuDgcd7M8zMizIpDHret1NaWV1b3yhvVra2d3b3qvsHj0blmvEmU1LpdkQNlyLlTRQoeTvTnCaR5K1oeDv1W09cG6HSBxxlPExoPxWxYBStFHT0QJ2Sa+K7Xrda81xvBrJM/ILUoECjW/3q9BTLE54ik9SYwPcyDMdUo2CSTyqd3PCMsiHt88DSlCbchOPZyRNyYpUeiZW2lSKZqb8nxjQxZpREtjOhODCL3lT8zwtyjK/CsUizHHnK5oviXBJUZPo/6QnNGcqRJZRpYW8lbEA1ZWhTqtgQ/MWXl8njuet7rn9/UavfFHGU4QiO4Qx8uIQ63EEDmsBAwTO8wpuDzovz7nzMW0tOMXMIf+B8/gACpo/A</latexit> ⇢0 = 1 |1 2 · ¯ ⇢| <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> <latexit sha1_base64="fsb0q9eDcL5wQcQ8gp2pK0AI21w=">AAACDnicbVDLSgMxFM3UV62vUZdugqXoxjJTBN0IRTcuK9gHdIaSyaRtaCYZkoxQpv0CN/6KGxeKuHXtzr8x085CWw/cy+Gce0nuCWJGlXacb6uwsrq2vlHcLG1t7+zu2fsHLSUSiUkTCyZkJ0CKMMpJU1PNSCeWBEUBI+1gdJP57QciFRX8Xo9j4kdowGmfYqSN1LMrnhyKE3gFXXgG4STrNejhUGjoBUimmT2d9OyyU3VmgMvEzUkZ5Gj07C8vFDiJCNeYIaW6rhNrP0VSU8zItOQlisQIj9CAdA3lKCLKT2fnTGHFKCHsC2mKazhTf2+kKFJqHAVmMkJ6qBa9TPzP6ya6f+mnlMeJJhzPH+onDGoBs2xgSCXBmo0NQVhS81eIh0girE2CJROCu3jyMmnVqq5Tde/Oy/XrPI4iOALH4BS44ALUwS1ogCbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB6NlmWA=</latexit> Normalized density:
90. c1 c2 c3 c4 t1 1 1 0 0 t2 0 1 1 0 t3 1 0 1 1 t4 0 0 0 1 transactions diversity: components uniqueness: ⇢0 = 1.0 G = 1.0 U = 1.0 <latexit sha1_base64="Sb5yD0ZNi//XUJj+ab5+ZcCJPaE=">AAACInicbZDNSsNAFIUn/tb6F3XpZrCIrkIigroQii50WcG0hSaUyWTSDp1M4sxEKKHP4sZXceNCUVeCD+OkzULbHhg4fPde5t4TpIxKZdvfxsLi0vLKamWtur6xubVt7uw2ZZIJTFycsES0AyQJo5y4iipG2qkgKA4YaQWD66LeeiRC0oTfq2FK/Bj1OI0oRkqjrnnhiX5yBC+hY9neQ4ZC6MVI9TFi+c1oLnZL3DVrtmWPBWeNU5oaKNXomp9emOAsJlxhhqTsOHaq/BwJRTEjo6qXSZIiPEA90tGWo5hIPx+fOIKHmoQwSoR+XMEx/TuRo1jKYRzozmJROV0r4LxaJ1PRuZ9TnmaKcDz5KMoYVAks8oIhFQQrNtQGYUH1rhD3kUBY6VSrOgRn+uRZ0zyxHNty7k5r9asyjgrYBwfgGDjgDNTBLWgAF2DwBF7AG3g3no1X48P4mrQuGOXMHvgn4+cXZ5+hGw==</latexit> <latexit sha1_base64="Sb5yD0ZNi//XUJj+ab5+ZcCJPaE=">AAACInicbZDNSsNAFIUn/tb6F3XpZrCIrkIigroQii50WcG0hSaUyWTSDp1M4sxEKKHP4sZXceNCUVeCD+OkzULbHhg4fPde5t4TpIxKZdvfxsLi0vLKamWtur6xubVt7uw2ZZIJTFycsES0AyQJo5y4iipG2qkgKA4YaQWD66LeeiRC0oTfq2FK/Bj1OI0oRkqjrnnhiX5yBC+hY9neQ4ZC6MVI9TFi+c1oLnZL3DVrtmWPBWeNU5oaKNXomp9emOAsJlxhhqTsOHaq/BwJRTEjo6qXSZIiPEA90tGWo5hIPx+fOIKHmoQwSoR+XMEx/TuRo1jKYRzozmJROV0r4LxaJ1PRuZ9TnmaKcDz5KMoYVAks8oIhFQQrNtQGYUH1rhD3kUBY6VSrOgRn+uRZ0zyxHNty7k5r9asyjgrYBwfgGDjgDNTBLWgAF2DwBF7AG3g3no1X48P4mrQuGOXMHvgn4+cXZ5+hGw==</latexit> <latexit sha1_base64="Sb5yD0ZNi//XUJj+ab5+ZcCJPaE=">AAACInicbZDNSsNAFIUn/tb6F3XpZrCIrkIigroQii50WcG0hSaUyWTSDp1M4sxEKKHP4sZXceNCUVeCD+OkzULbHhg4fPde5t4TpIxKZdvfxsLi0vLKamWtur6xubVt7uw2ZZIJTFycsES0AyQJo5y4iipG2qkgKA4YaQWD66LeeiRC0oTfq2FK/Bj1OI0oRkqjrnnhiX5yBC+hY9neQ4ZC6MVI9TFi+c1oLnZL3DVrtmWPBWeNU5oaKNXomp9emOAsJlxhhqTsOHaq/BwJRTEjo6qXSZIiPEA90tGWo5hIPx+fOIKHmoQwSoR+XMEx/TuRo1jKYRzozmJROV0r4LxaJ1PRuZ9TnmaKcDz5KMoYVAks8oIhFQQrNtQGYUH1rhD3kUBY6VSrOgRn+uRZ0zyxHNty7k5r9asyjgrYBwfgGDjgDNTBLWgAF2DwBF7AG3g3no1X48P4mrQuGOXMHvgn4+cXZ5+hGw==</latexit> <latexit sha1_base64="Sb5yD0ZNi//XUJj+ab5+ZcCJPaE=">AAACInicbZDNSsNAFIUn/tb6F3XpZrCIrkIigroQii50WcG0hSaUyWTSDp1M4sxEKKHP4sZXceNCUVeCD+OkzULbHhg4fPde5t4TpIxKZdvfxsLi0vLKamWtur6xubVt7uw2ZZIJTFycsES0AyQJo5y4iipG2qkgKA4YaQWD66LeeiRC0oTfq2FK/Bj1OI0oRkqjrnnhiX5yBC+hY9neQ4ZC6MVI9TFi+c1oLnZL3DVrtmWPBWeNU5oaKNXomp9emOAsJlxhhqTsOHaq/BwJRTEjo6qXSZIiPEA90tGWo5hIPx+fOIKHmoQwSoR+XMEx/TuRo1jKYRzozmJROV0r4LxaJ1PRuZ9TnmaKcDz5KMoYVAks8oIhFQQrNtQGYUH1rhD3kUBY6VSrOgRn+uRZ0zyxHNty7k5r9asyjgrYBwfgGDjgDNTBLWgAF2DwBF7AG3g3no1X48P4mrQuGOXMHvgn4+cXZ5+hGw==</latexit> G = 1 P i pi ⇥ (pi 1) N ⇥ (N 1) <latexit sha1_base64="/0DSOw3Hbde0S5+I68IorX3Z5sA=">AAACK3icbVDLSgMxFM34rPU16tJNsAh10TIRQTdCqQtdlQr2AZ1hyKSZNjTzIMkIZej/uPFXXOjCB279DzPtINp64MLJufdyc44XcyaVZb0bS8srq2vrhY3i5tb2zq65t9+WUSIIbZGIR6LrYUk5C2lLMcVpNxYUBx6nHW90lfU791RIFoV3ahxTJ8CDkPmMYKUl16zbAVZDgnl6PYGXEMEKtH2BSWrLJHAZjHXZigVUwrLmFXQySRs/SiN7u2bJqlpTwEWCclICOZqu+Wz3I5IENFSEYyl7yIqVk2KhGOF0UrQTSWNMRnhAe5qGWN9y0qnXCTzWSh/6kdAVKjhVf2+kOJByHHh6MnMm53uZ+F+vlyj/wklZGCeKhmR2yE84VBHMgoN9JihRfKwJJoLpv0IyxDoqpeMt6hDQvOVF0j6tIquKbs9KtXoeRwEcgiNQBgicgxq4AU3QAgQ8gCfwCt6MR+PF+DA+Z6NLRr5zAP7A+PoGR+alNg==</latexit> <latexit sha1_base64="/0DSOw3Hbde0S5+I68IorX3Z5sA=">AAACK3icbVDLSgMxFM34rPU16tJNsAh10TIRQTdCqQtdlQr2AZ1hyKSZNjTzIMkIZej/uPFXXOjCB279DzPtINp64MLJufdyc44XcyaVZb0bS8srq2vrhY3i5tb2zq65t9+WUSIIbZGIR6LrYUk5C2lLMcVpNxYUBx6nHW90lfU791RIFoV3ahxTJ8CDkPmMYKUl16zbAVZDgnl6PYGXEMEKtH2BSWrLJHAZjHXZigVUwrLmFXQySRs/SiN7u2bJqlpTwEWCclICOZqu+Wz3I5IENFSEYyl7yIqVk2KhGOF0UrQTSWNMRnhAe5qGWN9y0qnXCTzWSh/6kdAVKjhVf2+kOJByHHh6MnMm53uZ+F+vlyj/wklZGCeKhmR2yE84VBHMgoN9JihRfKwJJoLpv0IyxDoqpeMt6hDQvOVF0j6tIquKbs9KtXoeRwEcgiNQBgicgxq4AU3QAgQ8gCfwCt6MR+PF+DA+Z6NLRr5zAP7A+PoGR+alNg==</latexit> <latexit sha1_base64="/0DSOw3Hbde0S5+I68IorX3Z5sA=">AAACK3icbVDLSgMxFM34rPU16tJNsAh10TIRQTdCqQtdlQr2AZ1hyKSZNjTzIMkIZej/uPFXXOjCB279DzPtINp64MLJufdyc44XcyaVZb0bS8srq2vrhY3i5tb2zq65t9+WUSIIbZGIR6LrYUk5C2lLMcVpNxYUBx6nHW90lfU791RIFoV3ahxTJ8CDkPmMYKUl16zbAVZDgnl6PYGXEMEKtH2BSWrLJHAZjHXZigVUwrLmFXQySRs/SiN7u2bJqlpTwEWCclICOZqu+Wz3I5IENFSEYyl7yIqVk2KhGOF0UrQTSWNMRnhAe5qGWN9y0qnXCTzWSh/6kdAVKjhVf2+kOJByHHh6MnMm53uZ+F+vlyj/wklZGCeKhmR2yE84VBHMgoN9JihRfKwJJoLpv0IyxDoqpeMt6hDQvOVF0j6tIquKbs9KtXoeRwEcgiNQBgicgxq4AU3QAgQ8gCfwCt6MR+PF+DA+Z6NLRr5zAP7A+PoGR+alNg==</latexit> <latexit sha1_base64="/0DSOw3Hbde0S5+I68IorX3Z5sA=">AAACK3icbVDLSgMxFM34rPU16tJNsAh10TIRQTdCqQtdlQr2AZ1hyKSZNjTzIMkIZej/uPFXXOjCB279DzPtINp64MLJufdyc44XcyaVZb0bS8srq2vrhY3i5tb2zq65t9+WUSIIbZGIR6LrYUk5C2lLMcVpNxYUBx6nHW90lfU791RIFoV3ahxTJ8CDkPmMYKUl16zbAVZDgnl6PYGXEMEKtH2BSWrLJHAZjHXZigVUwrLmFXQySRs/SiN7u2bJqlpTwEWCclICOZqu+Wz3I5IENFSEYyl7yIqVk2KhGOF0UrQTSWNMRnhAe5qGWN9y0qnXCTzWSh/6kdAVKjhVf2+kOJByHHh6MnMm53uZ+F+vlyj/wklZGCeKhmR2yE84VBHMgoN9JihRfKwJJoLpv0IyxDoqpeMt6hDQvOVF0j6tIquKbs9KtXoeRwEcgiNQBgicgxq4AU3QAgQ8gCfwCt6MR+PF+DA+Z6NLRr5zAP7A+PoGR+alNg==</latexit> U = |G| M <latexit sha1_base64="tn8YU8tOE4rFNLl7v07ncYnx9Zo=">AAACBHicbVBNS8NAEJ34WetX1GMvi0XwVBIR9CIUPehFqGDaQlPKZrtpl24+2N0IJc3Bi3/FiwdFvPojvPlv3LQ5aOuDgcd7M8zM82LOpLKsb2NpeWV1bb20Ud7c2t7ZNff2mzJKBKEOiXgk2h6WlLOQOoopTtuxoDjwOG15o6vcbz1QIVkU3qtxTLsBHoTMZwQrLfXMihtgNSSYp06GLpDrC0zSyfUkS2+znlm1atYUaJHYBalCgUbP/HL7EUkCGirCsZQd24pVN8VCMcJpVnYTSWNMRnhAO5qGOKCym06fyNCRVvrIj4SuUKGp+nsixYGU48DTnfnJct7Lxf+8TqL8827KwjhRNCSzRX7CkYpQngjqM0GJ4mNNMBFM34rIEOsclM6trEOw519eJM2Tmm3V7LvTav2yiKMEFTiEY7DhDOpwAw1wgMAjPMMrvBlPxovxbnzMWpeMYuYA/sD4/AHJQpgt</latexit> <latexit sha1_base64="tn8YU8tOE4rFNLl7v07ncYnx9Zo=">AAACBHicbVBNS8NAEJ34WetX1GMvi0XwVBIR9CIUPehFqGDaQlPKZrtpl24+2N0IJc3Bi3/FiwdFvPojvPlv3LQ5aOuDgcd7M8zM82LOpLKsb2NpeWV1bb20Ud7c2t7ZNff2mzJKBKEOiXgk2h6WlLOQOoopTtuxoDjwOG15o6vcbz1QIVkU3qtxTLsBHoTMZwQrLfXMihtgNSSYp06GLpDrC0zSyfUkS2+znlm1atYUaJHYBalCgUbP/HL7EUkCGirCsZQd24pVN8VCMcJpVnYTSWNMRnhAO5qGOKCym06fyNCRVvrIj4SuUKGp+nsixYGU48DTnfnJct7Lxf+8TqL8827KwjhRNCSzRX7CkYpQngjqM0GJ4mNNMBFM34rIEOsclM6trEOw519eJM2Tmm3V7LvTav2yiKMEFTiEY7DhDOpwAw1wgMAjPMMrvBlPxovxbnzMWpeMYuYA/sD4/AHJQpgt</latexit> <latexit sha1_base64="tn8YU8tOE4rFNLl7v07ncYnx9Zo=">AAACBHicbVBNS8NAEJ34WetX1GMvi0XwVBIR9CIUPehFqGDaQlPKZrtpl24+2N0IJc3Bi3/FiwdFvPojvPlv3LQ5aOuDgcd7M8zM82LOpLKsb2NpeWV1bb20Ud7c2t7ZNff2mzJKBKEOiXgk2h6WlLOQOoopTtuxoDjwOG15o6vcbz1QIVkU3qtxTLsBHoTMZwQrLfXMihtgNSSYp06GLpDrC0zSyfUkS2+znlm1atYUaJHYBalCgUbP/HL7EUkCGirCsZQd24pVN8VCMcJpVnYTSWNMRnhAO5qGOKCym06fyNCRVvrIj4SuUKGp+nsixYGU48DTnfnJct7Lxf+8TqL8827KwjhRNCSzRX7CkYpQngjqM0GJ4mNNMBFM34rIEOsclM6trEOw519eJM2Tmm3V7LvTav2yiKMEFTiEY7DhDOpwAw1wgMAjPMMrvBlPxovxbnzMWpeMYuYA/sD4/AHJQpgt</latexit> <latexit sha1_base64="tn8YU8tOE4rFNLl7v07ncYnx9Zo=">AAACBHicbVBNS8NAEJ34WetX1GMvi0XwVBIR9CIUPehFqGDaQlPKZrtpl24+2N0IJc3Bi3/FiwdFvPojvPlv3LQ5aOuDgcd7M8zM82LOpLKsb2NpeWV1bb20Ud7c2t7ZNff2mzJKBKEOiXgk2h6WlLOQOoopTtuxoDjwOG15o6vcbz1QIVkU3qtxTLsBHoTMZwQrLfXMihtgNSSYp06GLpDrC0zSyfUkS2+znlm1atYUaJHYBalCgUbP/HL7EUkCGirCsZQd24pVN8VCMcJpVnYTSWNMRnhAO5qGOKCym06fyNCRVvrIj4SuUKGp+nsixYGU48DTnfnJct7Lxf+8TqL8827KwjhRNCSzRX7CkYpQngjqM0GJ4mNNMBFM34rIEOsclM6trEOw519eJM2Tmm3V7LvTav2yiKMEFTiEY7DhDOpwAw1wgMAjPMMrvBlPxovxbnzMWpeMYuYA/sD4/AHJQpgt</latexit>
95. Defects4J BC DDU Run Tests SFL
96. A. Perez, R. Abreu, and A. van Deursen. A Theoretical and Empirical Analysis of Program Spectra Diagnosability. TSE 2019 (also ICSE17)
97. A. Perez, R. Abreu, and A. van Deursen. A Theoretical and Empirical Analysis of Program Spectra Diagnosability. TSE 2019 (also ICSE17)
98. • Available as an Eclipse plug-in • Also available as a library • Instrumentation and diagnosis • (Some) Testing features are yet to be deployed
99. Trends / Opportunities • Integration with software repository mining • E.g., Estimate component reliability • Test Generation: the oracle problem. • Human in the loop? • Program invariants? • Integration with Program Repair • Include other aspects/domains • Humans, psychology, hardware, security
100. @rmaranhao
101. @rmaranhao
102. @rmaranhao
103. @rmaranhao
104. @rmaranhao
105. @rmaranhao
106. @rmaranhao
107. Influencer, the one