Efficient mutation analysis of relational database structure using mutant schemata and parallelisation

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Wright2013/

Transcript

1. Efficient Mutation Analysis of Relational Database Structure Using Mutant Schemata and Parallelisation Chris J. Wright Gregory M. Kapfhammer Phil McMinn Mutation 2013
2. Chris J. Wright - c.wright@dcs.shef.ac.uk Relational Database Management Systems (RDBMS)
3. Chris J. Wright - c.wright@dcs.shef.ac.uk Relational Database Management Systems (RDBMS)
4. Chris J. Wright - c.wright@dcs.shef.ac.uk
5. Chris J. Wright - c.wright@dcs.shef.ac.uk Many different RDBMSs...
6. Chris J. Wright - c.wright@dcs.shef.ac.uk Many different RDBMSs... ...same specification of structure
7. Chris J. Wright - c.wright@dcs.shef.ac.uk 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); Database Schema
8. Chris J. Wright - c.wright@dcs.shef.ac.uk 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); Database Schema Tables
9. Chris J. Wright - c.wright@dcs.shef.ac.uk 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); Database Schema Columns
10. Chris J. Wright - c.wright@dcs.shef.ac.uk 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); Database Schema Constraints
11. Chris J. Wright - c.wright@dcs.shef.ac.uk 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); Database Schema How do we know this is correct?
12. Chris J. Wright - c.wright@dcs.shef.ac.uk Why Test Database Structure?
13. Chris J. Wright - c.wright@dcs.shef.ac.uk Database Schema Why Test Database Structure?
14. Chris J. Wright - c.wright@dcs.shef.ac.uk DBMS Database Schema Why Test Database Structure?
15. Chris J. Wright - c.wright@dcs.shef.ac.uk DBMS Database Schema Why Test Database Structure?
16. Chris J. Wright - c.wright@dcs.shef.ac.uk DBMS Application Database Schema Why Test Database Structure?
17. Chris J. Wright - c.wright@dcs.shef.ac.uk DBMS Application Web Server Database Schema Why Test Database Structure?
18. Chris J. Wright - c.wright@dcs.shef.ac.uk DBMS Application Web Server Third Party Database Schema Why Test Database Structure?
19. Chris J. Wright - c.wright@dcs.shef.ac.uk DBMS Application Web Server Third Party Database Schema ✗ Why Test Database Structure?
20. Chris J. Wright - c.wright@dcs.shef.ac.uk DBMS Application Web Server Third Party Database Schema ✗ ✗ Why Test Database Structure?
21. Chris J. Wright - c.wright@dcs.shef.ac.uk DBMS Application Web Server Third Party Database Schema ✗ ✗ ✗ Why Test Database Structure?
22. Chris J. Wright - c.wright@dcs.shef.ac.uk DBMS Application Web Server Third Party Database Schema ✗ ✗ ✗ ✗ Why Test Database Structure?
23. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis
24. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Test Suite Application
25. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Database Test Suite
26. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Database Insert Statements
27. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Database Insert Statements
28. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); Database Insert Statements
29. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); Database Insert Statements
30. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); Database Insert Statements
31. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ Database Insert Statements
32. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ Database Insert Statements
33. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); Database Insert Statements
34. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ Database Insert Statements
35. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ Database Insert Statements
36. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); Database Insert Statements
37. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database Insert Statements
38. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database Insert Statements
39. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database Insert Statements INSERT INTO S(X, Y, Z) VALUES('a', 'b', 'a');
40. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database Insert Statements INSERT INTO S(X, Y, Z) VALUES('a', 'b', 'a'); ✗
41. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database Insert Statements INSERT INTO S(X, Y, Z) VALUES('a', 'b', 'a'); ✗
42. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutating the Structure 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, ) 9 REFERENCES T (A, B) 10 ); Y Z
43. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutating the Structure 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, ) 9 REFERENCES T (A, B) 10 ); Y Z
44. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutating the Structure 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, ) 9 REFERENCES T (A, B) 10 ); Y Z
45. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutating the Structure 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, ) 9 REFERENCES T (A, B) 10 ); Y Z
46. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 );
47. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 );
48. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); Database (mutated)
49. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); Database (mutated) Insert Statements
50. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); Database (mutated) Insert Statements
51. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ Database (mutated) Insert Statements
52. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ Database (mutated) Insert Statements
53. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); Database (mutated) Insert Statements
54. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ Database (mutated) Insert Statements
55. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ Database (mutated) Insert Statements
56. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); Database (mutated) Insert Statements
57. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database (mutated) Insert Statements
58. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database (mutated) Insert Statements
59. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database (mutated) Insert Statements INSERT INTO S(X, Y, Z) VALUES('a', 'b', 'a');
60. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database (mutated) Insert Statements INSERT INTO S(X, Y, Z) VALUES('a', 'b', 'a'); ✓
61. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database (mutated) Insert Statements INSERT INTO S(X, Y, Z) VALUES('a', 'b', 'a'); ✓
62. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database (mutated) Insert Statements INSERT INTO S(X, Y, Z) VALUES('a', 'b', 'a'); ✓ Results are different
63. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✓ INSERT INTO T(A, B, C) VALUES('a', 'a', 'a'); ✗ INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'b'); ✓ Database (mutated) Insert Statements INSERT INTO S(X, Y, Z) VALUES('a', 'b', 'a'); ✓ Mutant is killed
64. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Operators
65. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Operators Primary Key
66. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Operators Primary Key Not Null
67. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Operators Check Primary Key Not Null
68. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Operators Check Primary Key Not Null Unique
69. Chris J. Wright - c.wright@dcs.shef.ac.uk 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); The Problem? 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 );
70. Chris J. Wright - c.wright@dcs.shef.ac.uk 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); The Problem? 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 );
71. Chris J. Wright - c.wright@dcs.shef.ac.uk 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); The Problem? Many mutants to analyse
72. Chris J. Wright - c.wright@dcs.shef.ac.uk 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); The Problem? A time consuming process
73. Chris J. Wright - c.wright@dcs.shef.ac.uk The Solution?
74. Chris J. Wright - c.wright@dcs.shef.ac.uk The Solution? Mutant Schemata
75. Chris J. Wright - c.wright@dcs.shef.ac.uk The Solution? Mutant Schemata Combine mutants into a ‘meta-mutant’
76. Chris J. Wright - c.wright@dcs.shef.ac.uk The Solution? Parallelisation Mutant Schemata Combine mutants into a ‘meta-mutant’
77. Chris J. Wright - c.wright@dcs.shef.ac.uk The Solution? Parallelisation Mutant Schemata Combine mutants into a ‘meta-mutant’ Analyse multiple mutants simultaneously
78. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Approaches Mutant Representation Parallelisation Strategy Normal “Original”
79. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Approaches Mutant Representation Parallelisation Strategy Schemata Normal “Original” Full “Full Schemata”
80. Chris J. Wright - c.wright@dcs.shef.ac.uk Original Approach
81. Chris J. Wright - c.wright@dcs.shef.ac.uk Original Approach Create structure in database
82. Chris J. Wright - c.wright@dcs.shef.ac.uk Original Approach Create structure in database Execute insert statements
83. Chris J. Wright - c.wright@dcs.shef.ac.uk Original Approach Create structure in database Execute insert statements Drop structure from database
84. Chris J. Wright - c.wright@dcs.shef.ac.uk Original Approach Create structure in database Execute insert statements Drop structure from database
85. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutant Schemata Approach Create structure in database Execute insert statements Drop structure from database
86. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutant Schemata Approach Create structure in database Execute insert statements Drop structure from database
87. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutant Schemata Approach Create structure in database Execute insert statements Drop structure from database Reduce time creating/ dropping database
88. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach
89. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 );
90. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES T (A, B) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 );
91. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE mutant_1_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE mutant_1_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES mutant_1_T (A, B) 10 ); 1 CREATE TABLE mutant_2_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 4 ); 5 6 CREATE TABLE mutant_2_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES mutant_2_T (A, B) 10 );
92. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE mutant_1_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE mutant_1_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES mutant_1_T (A, B) 10 ); 11 12 CREATE TABLE mutant_2_T ( 13 A CHAR, B CHAR, C CHAR, 14 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 15 ); 16 17 CREATE TABLE mutant_2_S ( 18 X CHAR, Y CHAR, Z CHAR, 19 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 20 REFERENCES mutant_2_T (A, B) 21 );
93. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE mutant_1_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE mutant_1_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES mutant_1_T (A, B) 10 ); 11 12 CREATE TABLE mutant_2_T ( 13 A CHAR, B CHAR, C CHAR, 14 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 15 ); 16 17 CREATE TABLE mutant_2_S ( 18 X CHAR, Y CHAR, Z CHAR, 19 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 20 REFERENCES mutant_2_T (A, B) 21 );
94. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE mutant_1_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE mutant_1_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES mutant_1_T (A, B) 10 ); 11 12 CREATE TABLE mutant_2_T ( 13 A CHAR, B CHAR, C CHAR, 14 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 15 ); 16 17 CREATE TABLE mutant_2_S ( 18 X CHAR, Y CHAR, Z CHAR, 19 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 20 REFERENCES mutant_2_T (A, B) 21 ); Database (mutated) Insert Statements
95. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE mutant_1_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE mutant_1_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES mutant_1_T (A, B) 10 ); 11 12 CREATE TABLE mutant_2_T ( 13 A CHAR, B CHAR, C CHAR, 14 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 15 ); 16 17 CREATE TABLE mutant_2_S ( 18 X CHAR, Y CHAR, Z CHAR, 19 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 20 REFERENCES mutant_2_T (A, B) 21 ); Database (mutated) INSERT INTO S(X, Y, Z) VALUES('a', 'a', 'a'); Insert Statements
96. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE mutant_1_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE mutant_1_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES mutant_1_T (A, B) 10 ); 11 12 CREATE TABLE mutant_2_T ( 13 A CHAR, B CHAR, C CHAR, 14 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 15 ); 16 17 CREATE TABLE mutant_2_S ( 18 X CHAR, Y CHAR, Z CHAR, 19 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 20 REFERENCES mutant_2_T (A, B) 21 ); Database (mutated) INSERT INTO mutant_1_S(X, Y, Z) VALUES('a', 'a', 'a'); Insert Statements
97. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE mutant_1_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE mutant_1_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES mutant_1_T (A, B) 10 ); 11 12 CREATE TABLE mutant_2_T ( 13 A CHAR, B CHAR, C CHAR, 14 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 15 ); 16 17 CREATE TABLE mutant_2_S ( 18 X CHAR, Y CHAR, Z CHAR, 19 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 20 REFERENCES mutant_2_T (A, B) 21 ); Database (mutated) INSERT INTO mutant_1_S(X, Y, Z) VALUES('a', 'a', 'a'); ✓ Insert Statements
98. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE mutant_1_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE mutant_1_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES mutant_1_T (A, B) 10 ); 11 12 CREATE TABLE mutant_2_T ( 13 A CHAR, B CHAR, C CHAR, 14 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 15 ); 16 17 CREATE TABLE mutant_2_S ( 18 X CHAR, Y CHAR, Z CHAR, 19 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 20 REFERENCES mutant_2_T (A, B) 21 ); Database (mutated) INSERT INTO mutant_1_S(X, Y, Z) VALUES('a', 'a', 'a'); ✓ Insert Statements
99. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach
100. Chris J. Wright - c.wright@dcs.shef.ac.uk Create tables once... Full Schemata Approach
101. Chris J. Wright - c.wright@dcs.shef.ac.uk Create tables once... ...rewrite queries to match Full Schemata Approach
102. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Approaches Mutant Representation Parallelisation Strategy Schemata Normal “Original” Full “Full Schemata”
103. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Approaches Mutant Representation Parallelisation Strategy Schemata Normal “Original” Full “Full Schemata” Minimal “Minimal Schemata”
104. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE mutant_1_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE mutant_1_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES mutant_1_T (A, B) 10 ); 11 12 CREATE TABLE mutant_2_T ( 13 A CHAR, B CHAR, C CHAR, 14 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 15 ); 16 17 CREATE TABLE mutant_2_S ( 18 X CHAR, Y CHAR, Z CHAR, 19 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 20 REFERENCES mutant_2_T (A, B) 21 );
105. Chris J. Wright - c.wright@dcs.shef.ac.uk Full Schemata Approach 1 CREATE TABLE mutant_1_T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE mutant_1_S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 9 REFERENCES mutant_1_T (A, B) 10 ); 11 12 CREATE TABLE mutant_2_T ( 13 A CHAR, B CHAR, C CHAR, 14 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 15 ); 16 17 CREATE TABLE mutant_2_S ( 18 X CHAR, Y CHAR, Z CHAR, 19 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 20 REFERENCES mutant_2_T (A, B) 21 ); Mutated Tables
106. Chris J. Wright - c.wright@dcs.shef.ac.uk Minimal Schemata Approach 1 CREATE TABLE mutant_1_S ( 2 X CHAR, Y CHAR, Z CHAR, 3 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 4 REFERENCES mutant_1_T (A, B) 5 ); 6 7 CREATE TABLE mutant_2_T ( 8 A CHAR, B CHAR, C CHAR, 9 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 10 ); Mutated Tables
107. Chris J. Wright - c.wright@dcs.shef.ac.uk Minimal Schemata Approach 1 CREATE TABLE mutant_1_S ( 2 X CHAR, Y CHAR, Z CHAR, 3 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 4 REFERENCES mutant_1_T (A, B) 5 ); 6 7 CREATE TABLE mutant_2_T ( 8 A CHAR, B CHAR, C CHAR, 9 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 10 );
108. Chris J. Wright - c.wright@dcs.shef.ac.uk Minimal Schemata Approach 1 CREATE TABLE mutant_1_S ( 2 X CHAR, Y CHAR, Z CHAR, 3 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 4 REFERENCES mutant_1_T (A, B) 5 ); 6 7 CREATE TABLE mutant_2_T ( 8 A CHAR, B CHAR, C CHAR, 9 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 10 );
109. Chris J. Wright - c.wright@dcs.shef.ac.uk Minimal Schemata Approach 1 CREATE TABLE mutant_1_S ( 2 X CHAR, Y CHAR, Z CHAR, 3 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 4 REFERENCES mutant_1_T (A, B) 5 ); 6 7 CREATE TABLE mutant_2_T ( 8 A CHAR, B CHAR, C CHAR, 9 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 10 );
110. Chris J. Wright - c.wright@dcs.shef.ac.uk Minimal Schemata Approach 1 CREATE TABLE mutant_1_S ( 2 X CHAR, Y CHAR, Z CHAR, 3 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 4 REFERENCES mutant_1_T (A, B) 5 ); 6 7 CREATE TABLE mutant_2_T ( 8 A CHAR, B CHAR, C CHAR, 9 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 10 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 );
111. Chris J. Wright - c.wright@dcs.shef.ac.uk Minimal Schemata Approach 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 ); 11 12 CREATE TABLE mutant_1_S ( 13 X CHAR, Y CHAR, Z CHAR, 14 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Z) 15 REFERENCES T (A, B) 16 ); 17 18 CREATE TABLE mutant_2_T ( 19 A CHAR, B CHAR, C CHAR, 20 CONSTRAINT UniqueOnColsAandB UNIQUE (A, C) 21 );
112. Chris J. Wright - c.wright@dcs.shef.ac.uk Minimal Schemata Approach
113. Chris J. Wright - c.wright@dcs.shef.ac.uk Create tables once... Minimal Schemata Approach
114. Chris J. Wright - c.wright@dcs.shef.ac.uk Create tables once... ...only mutated tables... Minimal Schemata Approach
115. Chris J. Wright - c.wright@dcs.shef.ac.uk Create tables once... ...only mutated tables... Minimal Schemata Approach ...reduce queries executed
116. Chris J. Wright - c.wright@dcs.shef.ac.uk Create tables once... ...only mutated tables... Minimal Schemata Approach ...reduce queries executed Plus one copy for foreign keys
117. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Approaches Mutant Representation Parallelisation Strategy Schemata Normal “Original” Full “Full Schemata” Minimal “Minimal Schemata”
118. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Approaches Mutant Representation Parallelisation Strategy Schemata Normal “Original” Full “Full Schemata” Minimal “Minimal Schemata” Up front “Up-Front Schemata”
119. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutation Analysis Approaches Mutant Representation Parallelisation Strategy Schemata Normal “Original” Full “Full Schemata” Minimal “Minimal Schemata” Up front “Up-Front Schemata” Just in time “Just-in-Time Schemata”
120. Chris J. Wright - c.wright@dcs.shef.ac.uk Parallelisation ‘Just-in-Time Schemata’
121. Chris J. Wright - c.wright@dcs.shef.ac.uk Parallelisation ‘Just-in-Time Schemata’ Make the ‘Original’ approach parallel
122. Chris J. Wright - c.wright@dcs.shef.ac.uk Parallelisation ‘Up-Front Schemata’ ‘Just-in-Time Schemata’ Make the ‘Original’ approach parallel
123. Chris J. Wright - c.wright@dcs.shef.ac.uk Parallelisation ‘Up-Front Schemata’ ‘Just-in-Time Schemata’ Make the ‘Original’ approach parallel Make the ‘Full Schemata’ approach parallel
124. Chris J. Wright - c.wright@dcs.shef.ac.uk Original Approach Create structure in database Execute insert statements Drop structure from database
125. Chris J. Wright - c.wright@dcs.shef.ac.uk ‘Just-in-Time’ Approach Create structure in database Execute insert statements Drop structure from database Parallel
126. Chris J. Wright - c.wright@dcs.shef.ac.uk ‘Just-in-Time’ Approach Create structure in database Execute insert statements Drop structure from database Parallel
127. Chris J. Wright - c.wright@dcs.shef.ac.uk Mutant Schemata Approach Create structure in database Execute insert statements Drop structure from database
128. Chris J. Wright - c.wright@dcs.shef.ac.uk ‘Up-Front’ Approach Create structure in database Execute insert statements Drop structure from database Parallel
129. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study
130. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study Evaluation Metric Mutation Time Approaches 5 Case Studies 6 DBMSs 2 Repetitions 30
131. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study Evaluation Metric Mutation Time Approaches 5 Case Studies 6 DBMSs 2 Repetitions 30
132. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study Evaluation Metric Mutation Time Approaches 5 Case Studies 6 DBMSs 2 Repetitions 30
133. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study Evaluation Metric Mutation Time Approaches 5 Case Studies 6 DBMSs 2 Repetitions 30
134. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study Evaluation Metric Mutation Time Approaches 5 Case Studies 6 DBMSs 2 Repetitions 30
135. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: Approaches Mutant Representation Parallelisation Strategy Schemata Normal “Original” Full “Full Schemata” Minimal “Minimal Schemata” Up front “Up-Front Schemata” Just in time “Just-in-Time Schemata”
136. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: Case Studies Case Study Tables Columns Primary Keys Foreign Keys Unique Constraints Cloc 2 10 0 0 0 JWhoisServer 6 49 6 0 0 NistDML182 2 32 1 1 0 NistDML183 2 6 0 1 1 RiskIt 13 56 11 10 0 UnixUsage 8 32 7 7 0
137. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: Case Studies Case Study Tables Columns Primary Keys Foreign Keys Unique Constraints Cloc 2 10 0 0 0 JWhoisServer 6 49 6 0 0 NistDML182 2 32 1 1 0 NistDML183 2 6 0 1 1 RiskIt 13 56 11 10 0 UnixUsage 8 32 7 7 0
138. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: Case Studies Case Study Tables Columns Primary Keys Foreign Keys Unique Constraints Cloc 2 10 0 0 0 JWhoisServer 6 49 6 0 0 NistDML182 2 32 1 1 0 NistDML183 2 6 0 1 1 RiskIt 13 56 11 10 0 UnixUsage 8 32 7 7 0
139. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: Case Studies Case Study Tables Columns Primary Keys Foreign Keys Unique Constraints Cloc 2 10 0 0 0 JWhoisServer 6 49 6 0 0 NistDML182 2 32 1 1 0 NistDML183 2 6 0 1 1 RiskIt 13 56 11 10 0 UnixUsage 8 32 7 7 0
140. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: Case Studies Case Study Tables Columns Primary Keys Foreign Keys Unique Constraints Cloc 2 10 0 0 0 JWhoisServer 6 49 6 0 0 NistDML182 2 32 1 1 0 NistDML183 2 6 0 1 1 RiskIt 13 56 11 10 0 UnixUsage 8 32 7 7 0
141. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: Case Studies Case Study Tables Columns Primary Keys Foreign Keys Unique Constraints Cloc 2 10 0 0 0 JWhoisServer 6 49 6 0 0 NistDML182 2 32 1 1 0 NistDML183 2 6 0 1 1 RiskIt 13 56 11 10 0 UnixUsage 8 32 7 7 0
142. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: Case Studies Case Study Tables Columns Primary Keys Foreign Keys Unique Constraints Cloc 2 10 0 0 0 JWhoisServer 6 49 6 0 0 NistDML182 2 32 1 1 0 NistDML183 2 6 0 1 1 RiskIt 13 56 11 10 0 UnixUsage 8 32 7 7 0
143. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: Case Studies Case Study Total Constraints Total Mutants Cloc 0 30 JWhoisServer 50 184 NistDML182 2 66 NistDML183 2 18 RiskIt 36 160 UnixUsage 23 69
144. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: DBMSs
145. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: DBMSs PostgreSQL
146. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: DBMSs SQLite PostgreSQL
147. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: DBMSs SQLite PostgreSQL Client-Server Model
148. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: DBMSs SQLite PostgreSQL Client-Server Model Local Client Model
149. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: DBMSs SQLite PostgreSQL Client-Server Model Local Client Model Simultaneous Read/Write
150. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: DBMSs SQLite PostgreSQL Client-Server Model Local Client Model Simultaneous Read/Write Locking on Write
151. Chris J. Wright - c.wright@dcs.shef.ac.uk Empirical Study: DBMSs SQLite PostgreSQL Client-Server Model Local Client Model Simultaneous Read/Write Locking on Write Prevents Parallel Approaches
152. Chris J. Wright - c.wright@dcs.shef.ac.uk Results
153. Chris J. Wright - c.wright@dcs.shef.ac.uk Results • Median of repetitions
154. Chris J. Wright - c.wright@dcs.shef.ac.uk Results • Median of repetitions • Lower-is-better metric
155. Chris J. Wright - c.wright@dcs.shef.ac.uk Results • Median of repetitions • Lower-is-better metric • Split by...
156. Chris J. Wright - c.wright@dcs.shef.ac.uk Results • Median of repetitions • Lower-is-better metric • Split by... ...case study
157. Chris J. Wright - c.wright@dcs.shef.ac.uk Results • Median of repetitions • Lower-is-better metric • Split by... ...case study ...DBMS
158. Chris J. Wright - c.wright@dcs.shef.ac.uk Results • Median of repetitions • Lower-is-better metric • Split by... ...case study ...DBMS • Full details in paper (including statistics)
159. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 1000 2000 3000 Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Postgres – Cloc Original Full Minimal Up-Front Just-in-Time
160. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 1000 2000 3000 Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Postgres – Cloc ~3.27s Original Full Minimal Up-Front Just-in-Time
161. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 1000 2000 3000 Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Postgres – Cloc ~3.27s ~2.08s Original Full Minimal Up-Front Just-in-Time
162. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 50000 100000 150000 200000 250000 Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Postgres – RiskIt Original Full Minimal Up-Front Just-in-Time
163. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 50000 100000 150000 200000 250000 Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Postgres – RiskIt ~238s Original Full Minimal Up-Front Just-in-Time
164. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 50000 100000 150000 200000 250000 Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Postgres – RiskIt ~238s ~225s Original Full Minimal Up-Front Just-in-Time
165. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 50000 100000 150000 200000 250000 Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata Up−Front Schemata Just−in−Time Schemata Postgres – RiskIt ~238s ~225s ~23s Original Full Minimal Up-Front Just-in-Time
166. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – Postgres
167. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – Postgres ‘Full Schemata’
168. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – Postgres ‘Full Schemata’ Improvement decreases with larger schemas
169. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – Postgres ‘Minimal Schemata’ ‘Full Schemata’ Improvement decreases with larger schemas
170. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – Postgres ‘Minimal Schemata’ ‘Full Schemata’ Improvement decreases with larger schemas Consistently faster, scales very well
171. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – Postgres ‘Just-in-Time Schemata’ ‘Minimal Schemata’ ‘Full Schemata’ Improvement decreases with larger schemas Consistently faster, scales very well
172. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – Postgres ‘Just-in-Time Schemata’ Consistently faster, scales very well ‘Minimal Schemata’ ‘Full Schemata’ Improvement decreases with larger schemas Consistently faster, scales very well
173. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – Postgres ‘Up-Front Schemata’ ‘Just-in-Time Schemata’ Consistently faster, scales very well ‘Minimal Schemata’ ‘Full Schemata’ Improvement decreases with larger schemas Consistently faster, scales very well
174. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – Postgres ‘Up-Front Schemata’ ‘Just-in-Time Schemata’ Consistently faster, scales very well Improvement decreases with larger schemas ‘Minimal Schemata’ ‘Full Schemata’ Improvement decreases with larger schemas Consistently faster, scales very well
175. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – Postgres ‘Up-Front Schemata’ ‘Just-in-Time Schemata’ Consistently faster, scales very well Improvement decreases with larger schemas ‘Minimal Schemata’ ‘Full Schemata’ Improvement decreases with larger schemas Consistently faster, scales very well
176. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 5000 10000 15000 Original Full Schemata Minimal Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata SQLite – Cloc Original Full Minimal
177. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 5000 10000 15000 Original Full Schemata Minimal Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata SQLite – Cloc ~18.0s Original Full Minimal
178. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 5000 10000 15000 Original Full Schemata Minimal Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata SQLite – Cloc ~18.0s ~18.7s Original Full Minimal
179. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 5000 10000 15000 Original Full Schemata Minimal Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata SQLite – Cloc ~18.0s ~18.7s ~8.28s Original Full Minimal
180. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 500000 1000000 1500000 2000000 Original Full Schemata Minimal Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata SQLite – RiskIt Original Full Minimal
181. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 500000 1000000 1500000 2000000 Original Full Schemata Minimal Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata SQLite – RiskIt ~20.2 min Original Full Minimal
182. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 500000 1000000 1500000 2000000 Original Full Schemata Minimal Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata SQLite – RiskIt ~20.2 min ~31.8 min Original Full Minimal
183. Chris J. Wright - c.wright@dcs.shef.ac.uk 0 500000 1000000 1500000 2000000 Original Full Schemata Minimal Schemata Mutation Analysis Technique Mutation Analysis Time (ms) Original Full Schemata Minimal Schemata SQLite – RiskIt ~20.2 min ~31.8 min ~2.5 min Original Full Minimal
184. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – SQLite
185. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – SQLite ‘Full Schemata’
186. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – SQLite ‘Full Schemata’ Increasingly worsened with larger schemas
187. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – SQLite ‘Minimal Schemata’ ‘Full Schemata’ Increasingly worsened with larger schemas
188. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – SQLite ‘Minimal Schemata’ ‘Full Schemata’ Increasingly worsened with larger schemas Consistently faster, scales very well
189. Chris J. Wright - c.wright@dcs.shef.ac.uk Results – SQLite ‘Minimal Schemata’ ‘Full Schemata’ Increasingly worsened with larger schemas Consistently faster, scales very well
190. Chris J. Wright - c.wright@dcs.shef.ac.uk Future Work & Limitations
191. Chris J. Wright - c.wright@dcs.shef.ac.uk Future Work & Limitations Case Studies
192. Chris J. Wright - c.wright@dcs.shef.ac.uk Future Work & Limitations Case Studies DBMSs
193. Chris J. Wright - c.wright@dcs.shef.ac.uk Future Work & Limitations Detailed Timing Case Studies DBMSs
194. Chris J. Wright - c.wright@dcs.shef.ac.uk Future Work & Limitations Approaches Detailed Timing Case Studies DBMSs
195. Chris J. Wright - c.wright@dcs.shef.ac.uk Future Work & Limitations Approaches Detailed Timing Case Studies DBMSs Parallel Configurations
196. Chris J. Wright - c.wright@dcs.shef.ac.uk Future Work & Limitations Test Suite Approaches Detailed Timing Case Studies DBMSs Parallel Configurations
197. Chris J. Wright - c.wright@dcs.shef.ac.uk Test Suite Generation
198. Chris J. Wright - c.wright@dcs.shef.ac.uk Test Suite Generation • SchemaAnalyst tool
199. Chris J. Wright - c.wright@dcs.shef.ac.uk Test Suite Generation • SchemaAnalyst tool • Gregory Kapfhammer
200. Chris J. Wright - c.wright@dcs.shef.ac.uk Test Suite Generation • SchemaAnalyst tool • Gregory Kapfhammer • Tuesday 11:00am, Research & Industrial Track
201. Chris J. Wright - c.wright@dcs.shef.ac.uk Conclusions
202. Chris J. Wright - c.wright@dcs.shef.ac.uk Conclusions • Mutant Schemata and Parallelisation can both reduce the cost of mutation analysis
203. Chris J. Wright - c.wright@dcs.shef.ac.uk Conclusions • Mutant Schemata and Parallelisation can both reduce the cost of mutation analysis • The ‘Minimal Schemata’ approach...
204. Chris J. Wright - c.wright@dcs.shef.ac.uk Conclusions • Mutant Schemata and Parallelisation can both reduce the cost of mutation analysis • The ‘Minimal Schemata’ approach... ...consistently faster than original
205. Chris J. Wright - c.wright@dcs.shef.ac.uk Conclusions • Mutant Schemata and Parallelisation can both reduce the cost of mutation analysis • The ‘Minimal Schemata’ approach... ...consistently faster than original ...gives a reduction of up to 10x
206. Chris J. Wright - c.wright@dcs.shef.ac.uk Conclusions • Mutant Schemata and Parallelisation can both reduce the cost of mutation analysis • The ‘Minimal Schemata’ approach... ...consistently faster than original ...gives a reduction of up to 10x ...scales very well (for our case studies)
207. Chris J. Wright - c.wright@dcs.shef.ac.uk Conclusions • Mutant Schemata and Parallelisation can both reduce the cost of mutation analysis • The ‘Minimal Schemata’ approach... ...consistently faster than original ...gives a reduction of up to 10x ...scales very well (for our case studies) ...doesn’t require parallel DBMS access
208. Chris J. Wright - c.wright@dcs.shef.ac.uk Conclusions • Mutant Schemata and Parallelisation can both reduce the cost of mutation analysis • The ‘Minimal Schemata’ approach... ...consistently faster than original ...gives a reduction of up to 10x ...scales very well (for our case studies) ...doesn’t require parallel DBMS access • Website: http://schemaanalyst.org/
209. Chris J. Wright - c.wright@dcs.shef.ac.uk Conclusions • Mutant Schemata and Parallelisation can both reduce the cost of mutation analysis • The ‘Minimal Schemata’ approach... ...consistently faster than original ...gives a reduction of up to 10x ...scales very well (for our case studies) ...doesn’t require parallel DBMS access • Website: http://schemaanalyst.org/