Searching for the best tests

1.

“Searching” for the Best Tests An Introduction to Automated Software
Testing with Search-Based Techniques Gregory M. Kapfhammer Department of Computer Science Allegheny College March 31, 2015

2.

Software is Everywhere Software is pervasive — and so it
must be thoroughly tested! Computer Server Program

3.

must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program

4.

must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program

5.

must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program

6.

must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scienti c Device Program Household Appliance Program

7.

must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scienti c Device Program Household Appliance Program Network Router Program Scienti c Device Program

8.

Software is Complex Even simple programs are intricate — and
difficult to test! Computer Software

9.

difficult to test! Computer Software Lines of Code

10.

difficult to test! Computer Software Lines of Code Numerous Features

11.

difficult to test! Computer Software Lines of Code Numerous Features Feature Interactions

12.

difficult to test! Computer Software Lines of Code Numerous Features Feature Interactions Runtime Environs

13.

difficult to test! Computer Software Lines of Code Numerous Features Feature Interactions Runtime Environs “Software entities are more complex for their size than perhaps any other human construct” — Frederick P. Brooks, Jr.

14.

Software is Evolving Software is continuously updated — making testing
critical! Execution Environment Program

15.

critical! Execution Environment Program Execution Environment Program

16.

critical! Execution Environment Program Execution Environment Program Program Changed because of the addition of a new feature or the correction of a defect

17.

critical! Execution Environment Program Execution Environment Program

18.

critical! Execution Environment Program Execution Environment Program Execution Environment Program

19.

critical! Execution Environment Program Execution Environment Program Execution Environment Program Execution Environment Changed due to an up- grade in a kernel, device driver, or virtual machine

20.

critical! Execution Environment Program Execution Environment Program Execution Environment Program Execution Environment Changed due to an up- grade in a kernel, device driver, or virtual machine “Release early, release often” means that programs are regularly updated

21.

Motivating Example The computation of an object’s velocity presents challenges!
K = 1 2 m × v2

22.

K = 1 2 m × v2

23.

K = 1 2 m × v2

24.

K = 1 2 m × v2

25.

Computing Velocity public static String computeVelocity(int kinetic, int mass) {
int velocitySquared = 0; int velocity = 0; StringBuffer finalVelocity = new StringBuffer(); if( mass != 0 ) { velocitySquared = 3 * (kinetic / mass); velocity = (int)Math.sqrt(velocitySquared); finalVelocity.append(velocity); } else { finalVelocity.append("Undefined"); } return finalVelocity.toString(); }

26.

Important Questions Finding software defects is a challenging and rewarding
task Can you nd the defect in this program?

27.

Important Questions Finding software defects is a challenging and rewarding
task Are there general purpose strategies for defect isolation?

28.

The Challenges of Software Development Pervasiveness of Software Complexity of
Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

29.


30.


31.


32.

What is a Test Case? A test case calls a
method and checks its output with an oracle Method Un- der Test

33.

method and checks its output with an oracle Method Un- der Test Input

34.

method and checks its output with an oracle Method Un- der Test Input Output

35.

method and checks its output with an oracle Method Un- der Test Test Set Up

36.

method and checks its output with an oracle Method Un- der Test Test Set Up Input

37.

method and checks its output with an oracle Method Un- der Test Test Set Up Input Output

38.

method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up

39.

method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle

40.

method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output

41.

method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict

42.

method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output

43.

method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case passes and the code is correct!

44.

method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output

45.

method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case fails and a defect is found!

46.

What is a Test Suite? A test suite is an
organized collection of test cases T1

47.

organized collection of test cases T1 T2

48.

organized collection of test cases T1 T2 T3

49.

organized collection of test cases T1 T2 T3 T4

50.

organized collection of test cases T1 T2 T3 T4 . . .

51.

organized collection of test cases T1 T2 T3 T4 . . . Tn

52.

organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite

53.

organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing?

54.

organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit

55.

organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant

56.

organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant Eclipse

57.

A JUnit Test Case @Test public void testOne() { String
expected = new String("Undefined"); String actual = Kinetic. computeVelocity(5,0); assertEquals(expected, actual); }

58.

Another JUnit Test @Test public void testTwo() { String expected
= new String("0"); String actual = Kinetic. computeVelocity(0,5); assertEquals(expected, actual); }

59.

Important Questions Not all tests have the same fault detection
effectiveness! Will these test cases nd the fault in the example program?

60.

effectiveness! T1 assigns K = 5, m = 0 — Pass or fail?

61.

effectiveness! T2 assigns K = 0, m = 5 — Pass or fail?

62.

effectiveness! How do we study the e ectiveness of di erent test cases?

63.

The PIE Model There are necessary and sufficient conditions for
fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?

64.


65.


66.


67.


68.

fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using the PIE model, will the test cases nd the defect in the program?

69.

A JUnit Test Case — T1 @Test public void testOne()
{ String expected = new String("Undefined"); String actual = Kinetic. computeVelocity(5,0); assertEquals(expected, actual); } E I P

70.

A JUnit Test Case — T1 @Test public void testOne()
{ String expected = new String("Undefined"); String actual = Kinetic. computeVelocity(5,0); assertEquals(expected, actual); } E I P

71.

A JUnit Test Case — T2 @Test public void testTwo()
{ String expected = new String("0"); String actual = Kinetic. computeVelocity(0,5); assertEquals(expected, actual); } E I P

72.

A JUnit Test Case — T2 @Test public void testTwo()

73.

A JUnit Test Case — T3 @Test public void testThree()

74.

A JUnit Test Case — T3 @Test public void testThree()

75.

A JUnit Test Case — T4 @Test public void testFour()

76.

A JUnit Test Case — T4 @Test public void testFour()

77.

Test Suite Summary A test case must create specific inputs
in order to cause failure! Test Case Status T1 Pass T2 Pass T3 Pass T4 Fail

78.

Important Insights Software testing is fundamentally challenging — is there
help? I shall not deny that the construction of these testing programs has been a major intellectual e ort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. Edsger W. Dijkstra, Communications of the ACM, 1968

79.


80.


81.


82.


83.

Manual Testing While it has benefits, this industry standard may
be limited! Manual Testing

84.

be limited! Manual Testing Laborious

85.

be limited! Manual Testing Laborious Time Consuming

86.

be limited! Manual Testing Laborious Time Consuming Very Tedious

87.

be limited! Manual Testing Laborious Time Consuming Very Tedious Di cult

88.

be limited! Manual Testing Laborious Time Consuming Very Tedious Di cult Can we develop and employ methods that will automatically generate high- quality test cases for real-world software?

89.

Automated Testing Automatically generating tests is amazing — but does
it work? Automated Testing

90.

it work? Automated Testing Laborious

91.

it work? Automated Testing Laborious Time Consuming

92.

it work? Automated Testing Laborious Time Consuming Very Tedious

93.

it work? Automated Testing Laborious Time Consuming Very Tedious Di cult

94.

it work? Automated Testing Laborious Time Consuming Very Tedious Di cult Laborious Time Consuming Very Tedious Testing is less laborious and tedious because an algorithm generates the tests. While computational time is needed, a human can be less involved!

95.

it work? Automated Testing Laborious Time Consuming Very Tedious Di cult Laborious Time Consuming Very Tedious Laborious Time Consuming Very Tedious Di cult Automated testing is less di cult since a good tness function can guide the algorithm to inputs that nd the faults

96.

Random Testing It is easy to randomly generate tests —
but how good are they? −6 −4 −2 0 2 4 6 −1 −0.5 0 0.5 1 0 2 4 6 8 10 12 14

97.

Search-Based Testing Use a fitness function to guide the search
to “good” values 0 0.2 0.4 0.6 0.8 1 0 0.5 1 0 0.5 1

98.

Mutation Testing Let’s purposefully insert faults into the program under
test! T1 T2

99.

test! T1 T2 T3 T4

100.

test! T1 T2 T3 T4 T5 T6

101.

test! T1 T2 T3 T4 T5 T6 T7 T8

102.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

103.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10

104.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2

105.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4

106.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6

107.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8

108.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10

109.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12

110.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12}

111.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12

112.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12

113.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12

114.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4

115.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4 T1

116.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4 T1

117.

test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4 T1 M8

118.

Testing with EvoSuite This prototype can automatically generate real JUnit
test suites! Evolutionary Testing

119.

test suites! Evolutionary Testing Representation

120.

test suites! Evolutionary Testing Representation Fitness Function

121.

test suites! Evolutionary Testing Representation Fitness Function Modify Program

122.

test suites! Evolutionary Testing Representation Fitness Function Modify Program Operators

123.

test suites! Evolutionary Testing Representation Fitness Function Modify Program Operators “1600 Faults in 100 Projects: Automatically Finding Faults While Achieving High Coverage with EvoSuite”, Empirical Software Engineering

124.

Configuring EvoSuite This tool has many unique configurations — which
are best? EvoSuite’s Con gurations

125.

are best? EvoSuite’s Con gurations Random

126.

are best? EvoSuite’s Con gurations Random Fixed-Random

127.

are best? EvoSuite’s Con gurations Random Fixed-Random Genetic

128.

are best? EvoSuite’s Con gurations Random Fixed-Random Genetic Regression

129.

are best? EvoSuite’s Con gurations Random Fixed-Random Genetic Regression The tness function computes branch coverage, weak mutation score, or strong mutation score to guide the search to the “best” test cases

130.

A Test from EvoSuite @Test public void test001() throws Throwable
{ int int0 = 0; String string0 = Kinetic.computeVelocity(int0, int0); assertEquals("Undefined", string0); assertNotNull(string0); int int1 = 1262; String string1 = Kinetic.computeVelocity(int0, int0); int int2 = 5349; String string2 = Kinetic.computeVelocity(int1, int2); int int3 = 0; int int4 = 3; String string3 = Kinetic.computeVelocity(int3, int4); Kinetic kinetic0 = new Kinetic(); }

131.

Important Questions EvoSuite is an advanced, yet sometimes limited, testing
tool Will EvoSuite’s test cases nd the fault in the example program?

132.

tool What is missing from the test cases that EvoSuite generates?

133.

tool How does the oracle problem in uence the e ectiveness of EvoSuite?

134.

tool What are the fundamental limitations of automated testing?

135.

No Silver Bullet Software tools are fundamentally limited — what
is our hope? There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity. Frederick P. Brooks, Jr., Proceedings of the IFIP Tenth World Computing Conference, 1986

136.


137.


138.


139.


140.

The Software Crisis Solutions are available — but not always
obvious or popular What are the solutions to the software crisis?

141.

obvious or popular Great Designers

142.

obvious or popular Unremitting Care

143.

obvious or popular Incremental Advances

144.

Final Admonishment An epilogue called “Fifty Years of Wonder, Excitement,
and Joy” Too many interests, too many exciting opportunities for learning, research, and thought. What a marvelous predicament! Not only is the end not in sight, but the pace is not slackening. We have many future joys. Frederick P. Brooks, Jr., The Mythical Man-Month, 1995

Searching for the best tests

Searching for the best tests

Want more?

Transcript