Automatically evaluating the efficiency of search-based test data generation for relational database schemas

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Kinneer2015/

Transcript

1. Automatically Evaluating the Efficiency of Search-Based Test Data Generation (for Relational Database Schemas) Cody Kinneer SEKE 2015 July 7, 2015
2. Random Testing
3. Random Testing Easy to implement — and yet not always very e ective!
4. Search-Based Testing 0 0.2 0.4 0.6 0.8 1 0 0.5 1 0 0.5 1
5. Search-Based Testing 0 0.2 0.4 0.6 0.8 1 0 0.5 1 0 0.5 1 Often much more e ective than random testing
6. Performance of SBST
7. Performance of SBST Fitness Function
8. Performance of SBST Fitness Function Data Generator
9. Performance of SBST Fitness Function Data Generator Restart Rule
10. Performance of SBST Fitness Function Data Generator Restart Rule Stop Rule
11. Performance of SBST Fitness Function Data Generator Restart Rule Stop Rule Search Budget
12. Performance of SBST Fitness Function Data Generator Restart Rule Stop Rule Search Budget How do parameter values in uence the e ciency of SBST?
13. Performance of SBST O( )
14. Performance of SBST O(?)
15. Performance of SBST O(?) Analytical
16. Performance of SBST O(?) Analytical
17. Performance of SBST O(?) Analytical Empirical
18. Performance of SBST O(?) Analytical Empirical
19. Doubling Experiment Input
20. Doubling Experiment Input
21. Doubling Experiment Input Time = 14.98
22. Doubling Experiment Input Time = 14.98 Input
23. Doubling Experiment Input Time = 14.98 Input
24. Doubling Experiment Input Time = 14.98 Input Time = 31.45
25. Doubling Experiment Input Time = 14.98 Input Time = 31.45
26. Doubling Experiment Input Time = 14.98 Input Time = 31.45 Ratio ≈ 2
27. Doubling Experiment Input Time = 14.98 Input Time = 31.45 Ratio ≈ 2 Linear — O(n)
28. Doubling Experiment Input Input
29. Doubling Experiment Input Time = 12.63 Input
30. Doubling Experiment Input Time = 12.63 Input Time = 51.48
31. Doubling Experiment Input Time = 12.63 Input Time = 51.48 Ratio ≈ 4
32. Doubling Experiment Input Time = 12.63 Input Time = 51.48 Ratio ≈ 4 Quadratic — O(n2)
33. Doubling Experiment Input Input
34. Doubling Experiment Input Time = 11.23 Input
35. Doubling Experiment Input Time = 11.23 Input Time = 89.72
36. Doubling Experiment Input Time = 11.23 Input Time = 89.72 Ratio ≈ 8
37. Doubling Experiment Input Time = 11.23 Input Time = 89.72 Ratio ≈ 8 Cubic — O(n3)
38. Relational Databases Deployment Locations for Databases
39. Relational Databases Deployment Locations for Databases Database Application Server
40. Relational Databases Deployment Locations for Databases Database Application Server Mobile Phone or Tablet
41. Relational Databases Deployment Locations for Databases Database Application Server Mobile Phone or Tablet O ce and Productivity Software
42. Relational Databases Deployment Locations for Databases Database Application Server Mobile Phone or Tablet O ce and Productivity Software Government
43. Relational Databases Deployment Locations for Databases Database Application Server Mobile Phone or Tablet O ce and Productivity Software Government Astrophysics
44. Database Schemas Relational Database Management System
45. Database Schemas Relational Database Management System E-commerce
46. Database Schemas Relational Database Management System E-commerce Schema
47. Database Schemas Relational Database Management System E-commerce Schema State
48. Database Schemas Relational Database Management System E-commerce Schema State Schema Integrity Constraints
49. Database Schemas Relational Database Management System E-commerce Schema State Schema Integrity Constraints PRIMARY KEY
50. Database Schemas Relational Database Management System E-commerce Schema State Schema Integrity Constraints PRIMARY KEY FOREIGN KEY
51. Database Schemas Relational Database Management System E-commerce Schema State Schema Integrity Constraints PRIMARY KEY FOREIGN KEY Arbitrary CHECK
52. Database Schemas Relational Database Management System E-commerce Schema State State Relational Components
53. Database Schemas Relational Database Management System E-commerce Schema State State Relational Components Tables
54. Database Schemas Relational Database Management System E-commerce Schema State State Relational Components Tables Rows
55. Database Schemas Relational Database Management System E-commerce Schema State State Relational Components Tables Rows Columns
56. Database Testing The Data Warehouse Institute reports that North American organizations experience a $611 billion annual loss due to poor data quality
57. Database Testing The Data Warehouse Institute reports that North American organizations experience a $611 billion annual loss due to poor data quality Scott W. Ambler argues that the “virtual absence” of database testing — the validation of the contents, schema, and functionality of the database — is the primary cause of this loss
58. Database Testing The Data Warehouse Institute reports that North American organizations experience a $611 billion annual loss due to poor data quality Scott W. Ambler argues that the “virtual absence” of database testing — the validation of the contents, schema, and functionality of the database — is the primary cause of this loss Past papers presented SchemaAnalyst, a search-based system for testing the complex integrity constraints in relational schemas
59. Method of Approach SchemaAnalyst Execution
60. Method of Approach SchemaAnalyst Execution Coverage Criterion
61. Method of Approach SchemaAnalyst Execution Coverage Criterion Data Generator
62. Method of Approach SchemaAnalyst Execution Coverage Criterion Data Generator Database Schema
63. Method of Approach SchemaAnalyst Execution Coverage Criterion Data Generator Database Schema Test Suite
64. Method of Approach SchemaAnalyst Execution Coverage Criterion Data Generator Database Schema Runtime
65. Method of Approach SchemaAnalyst Execution Coverage Criterion Data Generator Runtime Schema Doubler Provides Schema Database Schema
66. Method of Approach SchemaAnalyst Execution Coverage Criterion Data Generator Runtime Schema Doubler Provides Schema Database Schema Doubler Choice
67. Method of Approach SchemaAnalyst Execution Coverage Criterion Data Generator Runtime Schema Doubler Provides Schema Database Schema Doubler Choice Convergence Algorithm Continue?
68. Doubling Schemas Table Column 1 Column 2 . . . Column n
69. Doubling Schemas Table Column 1 Column 2 . . . Column n NOT NULL
70. Doubling Schemas Table Column 1 Column 2 . . . Column n NOT NULL PRIMARY KEY
71. Doubling Schemas Table Column 1 Column 2 . . . Column n NOT NULL PRIMARY KEY UNIQUE
72. Doubling Schemas Table Column 1 Column 2 . . . Column n NOT NULL PRIMARY KEY UNIQUE CHECK
73. Doubling Schemas Table Column 1 Column 2 . . . Column n NOT NULL PRIMARY KEY UNIQUE CHECK FORIEGN KEY
74. Doubling Schemas Table Column 1 Column 2 . . . Column n NOT NULL PRIMARY KEY UNIQUE CHECK FORIEGN KEY NOT NULL
75. Experiments Experimental Parameters
76. Experiments Experimental Parameters Coverage Criterion
77. Experiments Experimental Parameters Coverage Criterion Data Generator
78. Experiments Experimental Parameters Coverage Criterion Data Generator Doubling Technique
79. Experiments Experimental Parameters Coverage Criterion Data Generator Doubling Technique Database Schema
80. Experiments Experimental Parameters Coverage Criterion Data Generator Doubling Technique Database Schema Over 2,000 unique combinations of parameters!
81. Experiments Experimental Parameters Coverage Criterion Data Generator Doubling Technique Database Schema Over 2,000 unique combinations of parameters! Experiments ran on HPC cluster with 3,440 cores
82. Relational Schemas Schema Tables Columns Constraints BioSQL 28 129 186 Cloc 2 10 0 iTrust 42 309 134 JWhoisServer 6 49 50 NistWeather 2 9 13 NistXTS7 1 3 3 NistXTS749 1 3 3 RiskIt 13 57 36 UnixUsage 8 32 24
83. Relational Schemas Schema Tables Columns Constraints BioSQL 28 129 186 Cloc 2 10 0 iTrust 42 309 134 JWhoisServer 6 49 50 NistWeather 2 9 13 NistXTS7 1 3 3 NistXTS749 1 3 3 RiskIt 13 57 36 UnixUsage 8 32 24
84. Relational Schemas Schema Tables Columns Constraints BioSQL 28 129 186 Cloc 2 10 0 iTrust 42 309 134 JWhoisServer 6 49 50 NistWeather 2 9 13 NistXTS7 1 3 3 NistXTS749 1 3 3 RiskIt 13 57 36 UnixUsage 8 32 24
85. Relational Schemas Schema Tables Columns Constraints BioSQL 28 129 186 Cloc 2 10 0 iTrust 42 309 134 JWhoisServer 6 49 50 NistWeather 2 9 13 NistXTS7 1 3 3 NistXTS749 1 3 3 RiskIt 13 57 36 UnixUsage 8 32 24
86. Empirical Results Doubled UNIQUEs NOT NULLs CHECKs
87. Empirical Results Doubled UNIQUEs NOT NULLs CHECKs 699 Experiments
88. Empirical Results Doubled UNIQUEs NOT NULLs CHECKs 699 Experiments 8% Stopped
89. Empirical Results Doubled UNIQUEs NOT NULLs CHECKs 699 Experiments 8% Stopped 20% O(1) or O(log)
90. Empirical Results Doubled UNIQUEs NOT NULLs CHECKs 699 Experiments 8% Stopped 20% O(1) or O(log) 72% O(n) or O(n log n)
91. Empirical Results Doubled UNIQUEs NOT NULLs CHECKs 699 Experiments 8% Stopped 20% O(1) or O(log) 72% O(n) or O(n log n) SchemaAnalyst ∈ O(n) for constraints studied
92. Empirical Results Doubled Tables
93. Empirical Results Doubled Tables 467 Experiments
94. Empirical Results Doubled Tables 467 Experiments 56% Stopped
95. Empirical Results Doubled Tables 467 Experiments 56% Stopped 72 O(n2)
96. Empirical Results Doubled Tables 467 Experiments 56% Stopped 72 O(n2) 10 O(n3)
97. Empirical Results Doubled Tables 467 Experiments 56% Stopped 72 O(n2) 10 O(n3) SchemaAnalyst ∈ O(n3) or worse for tables
98. Empirical Results Doubled Columns
99. Empirical Results Doubled Columns 467 Experiments
100. Empirical Results Doubled Columns 467 Experiments 203 Stopped
101. Empirical Results Doubled Columns 467 Experiments 203 Stopped 208 O(n) or O(n log n)
102. Empirical Results Doubled Columns 467 Experiments 203 Stopped 208 O(n) or O(n log n) 28 O(n2) and 2 O(n3)
103. Empirical Results Doubled Columns 467 Experiments 203 Stopped 208 O(n) or O(n log n) 28 O(n2) and 2 O(n3) SchemaAnalyst ∈ O(n3) or worse for columns
104. Adequacy Criteria
105. Adequacy Criteria More e ective criteria require additional runtime
106. Data Generator
107. Data Generator More e ective generators can also be more e cient
108. Key Contributions Search-based test data generation is often highly e ective, but worst-case time complexity unknown
109. Key Contributions Search-based test data generation is often highly e ective, but worst-case time complexity unknown A technique for automated doubling experiments
110. Key Contributions Search-based test data generation is often highly e ective, but worst-case time complexity unknown A technique for automated doubling experiments Emprical suggestions for worst-case time complexity
111. Key Contributions Search-based test data generation is often highly e ective, but worst-case time complexity unknown A technique for automated doubling experiments Emprical suggestions for worst-case time complexity Tradeo s in search-based test data generation
112. Key Contributions Search-based test data generation is often highly e ective, but worst-case time complexity unknown A technique for automated doubling experiments Emprical suggestions for worst-case time complexity Tradeo s in search-based test data generation https://github.com/kinneerc/ExpOse