Jeremias Rößler (ReTest), Tested by Monkeys — The end of banana software!, CodeFest 2017

https://2017.codefest.ru/lecture/1141

Various automation tools have been available for quite some time now, but due to high creation and maintenance efforts of tests, GUI-Testing is still mainly a manual task. Meanwhile overall testing effort has risen to make up 30% of a typical software budget. Is crowd-testing the answer? What if we could have automated test cases be created automatically?

Monkey Testing is not a new idea. But combined with a manually trainable AI and an innovative new testing approach (dubbed "difference testing"), we can now not only have the monkey search for technical bugs (i.e. crashes) but generate functional test cases that are optimized towards several goals and are even better than manually created ones.

Visit the future of testing and see how AI can help us create better software!

Transcript

1. Tested by Monkeys The end of banana software! Dr. Jeremias Rößler Founder ReTest
3. 3 Hello!
4. 4 Bug
5. 5 Change Bug
6. 6 Change Bug Test
7. 7 Change Bug Test
8. 8 Change Bug Test
9. 9 Change Bug Test
10. 10 Change Bug Test
11. 11 Change Bug Test
12. 12 Change Bug Test
13. 13 Change Bug Test
14. 14 Change Bug Test
15. 15 Change Bug Test
16. 16 Change Bug Test
17. 17 Change Bug Test
18. 18 Change Bug Test
19. 19 Change Bug Test
20. 20 SURVEY
21. 21 Who has ever tested manually? SURVEY
22. 22 Who has ever regression-tested manually? Who has ever tested manually? SURVEY
23. 23 Who has ever regression-tested manually? Who has ever tested manually? Who liked it? SURVEY
24. 24 SURVEY Why didn’t you automate regression testing?
25. 25 Mehr-Aufwand Zeit Einsparung Time Additional Effort Saved Effort
26. 26 AUTOMATION Maintenance effort Risk Test Coverage
27. 27 Problem: - high testing effort - high test automation effort
28. 28 Infinite-Monkey-Theorem: A monkey hitting keys at random on a typewriter for a long enough time will type the complete works of William Shakespeare.
29. 29 We replace the typewriter with a computer…
30. Infinite Monkey 30 public static void main(String... args) throws Exception { Robot robot = new Robot(); while (true) { robot.mouseMove(random.nextInt(maxX), random.nextInt(maxY)); robot.mousePress(InputEvent.BUTTON1_DOWN_MASK); robot.mouseRelease(InputEvent.BUTTON1_DOWN_MASK); robot.delay(200); for (char inputChar : randomString().toCharArray()) { robot.keyPress((int) inputChar); robot.keyRelease((int) inputChar); robot.delay(10); } robot.keyPress(KeyEvent.VK_ENTER); robot.keyRelease(KeyEvent.VK_ENTER); } } 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16.
31. Infinite Monkey 31 DEMO
35. Intelligent Monkey 35 public static void main(String... args) throws Exception { WebDriver driver = new FirefoxDriver(); driver.get("http://www.retest.de"); while (true) { List<WebElement> links = driver.findElements(By.tagName("a")); links.get(random.nextInt(links.size())).click(); Thread.sleep(500); List<WebElement> fields = driver.findElements(By.xpath("//input[@type='text']")); WebElement field = fields.get(random.nextInt(fields.size())); field.sendKeys(randomString()); Thread.sleep(500); } } 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.
36. 36 Intelligent Monkey DEMO
38. Intelligent Monkey 38
39. 39 - gremlins.js - NetFlix Chaos Monkey - UI/Application Exerciser Monkey Open Source Products
40. 40 - quick and easy to implement - finds bugs Traditional monkey testing but: - relevant bugs? - not intelligent - not trainable - needs sandboxing
41. 41 Intelligent Monkey ?
44. 44 Problem space Problem space
48. 48 - bases on realistic use case - covers corner cases - what about coverage? Trainable monkey testing
50. 50 Several searches in parallel
51. 51 Source: https://www.youtube.com/watch?v=yJpLUP93pRo
52. 52 Source: http://cnx.org/content/m47304/latest/ © Robert Bear and David Rintoul
53. 53 Gen Exon Intron Exon
54. Tower of Hanoi 54 © André Karwath aka Aka Source: http://en.wikipedia.org/wiki/File:Tower_of_Hanoi_4.gif
55. 55 Tower of Hanoi Possibilities 63 >2
56. 56 9.223.372.036.854.775.808 Tower of Hanoi Possibilities
57. 57 Tower of Hanoi Possibilities 292.471.208 Years
58. 58 Genetic Algorithm efficiency baseline: 150 mio. years with GA: 66 seconds
60. 60 - bases on realistic use case - optimises for coverage - very effective intelligent monkey testing
61. 61 How do you bring AI into testing?
62. 62 AI in Testautomation is a hype. Wolfgang Platz, Founder & CPO of Tricentis “
63. 63 AI in Testing? – so far I do not see anyone doing it in a useful way. Antony Edwards, CTO of TestPlant “
64. 64 Which bugs can the monkey find?
66. 66 Oracle Problem
67. 67 When is a bug a bug? When it’s not a feature!
68. 68 What is a bug? Without specification, there are no bugs — only surprises. Brian Kernighan “
69. 69 Why do we test? After implementation? 1 + 1 = 3 2 + 2 = 4
73. 73 Why do we test? After implementation? 1 + 1 = 3 2 + 2 = 4 After change? 1 + 1 = 3 1 + 1 = 5
74. 74 regression-test != test
75. 75 regression-test == version control
80. 80 - oracle problems hinders AI - regression testing needs no oracle - VCS for the GUI functional monkey testing
81. 81 Intelligenter Affe Summary Monkey-Testing: cheaper faster better multiplies manual tests Difference Testing little maintenance no wasteful testing yet more complete
83. 83 Coming 2017 Mobile Web
84. 84 https://retest.rocks @ReTest_en
85. @roesslerj Dr. Jeremias Rößler Founder ReTest Questions? https://www.facebook.com/retest.de https://retest.rocks roessler@retest.de
86. 86 1. if (name[0] == '.') continue; Is it a bug? . .. text.c other.files
87. 87 Is it a bug?
88. 88 Is it a bug?