CodeFest 2019. Карл Кроуфорд (Badoo) — Visual Regression Testing

Visual regression testing (VRT) is a weapon-like tool used by many QAs to protect against the inevitable and unexpected (but expected) dangers of development. In recent years the area has grown in popularity, with many more commercial and custom solutions becoming available — but what is it? How can we use it? And how hard is it to implement? In this presentation, I aim to share the principles of VRT while incrementally building a VRT solution from scratch, using Puppeteer and Headless Chrome. The plan is for you to come away with both knowledge and examples to help you bring VRT to your workplace.

Transcript

1. Visual Regression Testing (VRT) Carl Crawford · 30th of March
2. Carl Crawford Frontend Engineer
4. > 418 000 000 people all over the world use our apps
5. 1. What is Visual Regression Testing? 2. Why use it? 3. When to use it? 4. Available tools? 5. Building VRT 6. Pros and Cons 7. Real Failures 5
6. So what is visual regression testing? 6
7. visual /ˈvɪʒ(j)ʊəl,ˈvɪzjʊəl/ adjective 1. relating to seeing or sight.
 
 regression /rɪˈɡrɛʃ(ə)n/ noun 1. a return to a former or less developed state. 7
8. “Testing - that something visual - did not return to a former or less developed state” visual /ˈvɪʒ(j)ʊəl,ˈvɪzjʊəl/ adjective 1. relating to seeing or sight.
 
 regression /rɪˈɡrɛʃ(ə)n/ noun 1. a return to a former or less developed state. 8
9. “Testing - that something visual - did not return to a former or less developed state” visual /ˈvɪʒ(j)ʊəl,ˈvɪzjʊəl/ adjective 1. relating to seeing or sight.
 
 regression /rɪˈɡrɛʃ(ə)n/ noun 1. a return to a former or less developed state. УХУДШЕ́ НИЕ Средний род 1. Изменение, ухудшающее что-н. 9
10. Why VRT? Detecting visual changes is hard It’s another form of automated testing 10
11. Why VRT? Detecting visual changes is hard It’s another form of automated testing Automation is easy and reliable Tests are descriptive, defined, and reproducible 11
12. Why VRT? Detecting visual changes is hard It’s another form of automated testing Automation is easy and reliable Tests are descriptive, defined, and reproducible Avoid “change blindness” Magic word 12
13. Change blindness is a perceptual phenomenon that occurs when a change in a visual stimulus is introduced and the observer does not notice it. 13
14. Selective Attention Test 14
15. https://youtu.be/vJG698U2Mvo
16. Lets play a game of “spot the difference” 16
18. How many differences did you spot? Lets make it easier 18
21. Easier? Lets see a computer do it… 21
23. How does it work? The simple version 23
24. 1 Generate a baseline How does VRT work? 24
25. 1 Generate a baseline 2 Make some changes How does VRT work? 25
26. 1 Generate a baseline 2 Make some changes 3 Take another screen capture How does VRT work? 26
27. 1 Generate a baseline 2 Make some changes 3 Take another screen capture 4 Compare the two How does VRT work? 27
28. Image courtesy of Nikhil Verma Where does it fit with traditional testing? 28
29. What is it good for? 29
33. What’s available? An awesome source of information for all VRT related topics provided by Alexander Wunshik https://github.com/mojoaxel/awesome-regression-testing 33
34. 1. Browsers https://github.com/mojoaxel/awesome-regression-testing 34
35. 1. Browsers 2. Frameworks https://github.com/mojoaxel/awesome-regression-testing 35
36. 1. Browsers 2. Frameworks 3. Online Services https://github.com/mojoaxel/awesome-regression-testing 36
37. 1. Browsers 2. Frameworks 3. Online Services 4. Blog posts https://github.com/mojoaxel/awesome-regression-testing 37
38. 1. Browsers 2. Frameworks 3. Online Services 4. Blog posts 5. Talks/Videos from the community https://github.com/mojoaxel/awesome-regression-testing 38
39. Alternatively, build your own! 39
40. 1 Browser and/or App Automation General Architecture 40
41. 1 Browser and/or App Automation 2 Tool or framework for captures General Architecture 41
42. 1 Browser and/or App Automation 2 Tool or framework for captures 3 Something to compare captures General Architecture 42
43. A brief history of VRT in Badoo PhantomJS Load and control our site 43
44. A brief history of VRT in Badoo PhantomJS Load and control our site Casper and PhantomCSS Provided us tools for scripting and comparing screenshots as tests 44
45. A brief history of VRT in Badoo PhantomJS Load and control our site Casper and PhantomCSS Provided us tools for scripting and comparing screenshots as tests Custom interface Used to present the results to QA and developers in a user friendly fashion 45
46. A brief history of VRT in Badoo Then along came Headless Chrome With puppeteer, it replaced PhantomJS and PhantomCSS for us 46
47. A brief example please forgive my markup 47
48. <html> <head> <link href=“styles.css” … /> </head> <body> <ul class="navbar"> <li>About Me</li> <li>Kittens</li> </ul> <hr> <div class="content"> <div class="avatar"> <img src="me.png"> </div> <div> <p>Hey! my name is Carl</p> <p>I like cats!</p> <p>also, things</p> </div> </div> </body> </html> 48
49. <html> <head> <link href=“styles.css” … /> </head> <body> <ul class="navbar"> <li>About Me</li> <li>Kittens</li> </ul> <hr> <div class="content"> <div class="kitty"> <img src="mlem.png"> </div> </div> </body> </html> 49
50. # prepare a screen capture utility $ yarn add puppeteer puppeteer-core 50
51. # prepare a screen capture utility $ yarn add puppeteer puppeteer-core $ vim screenshotter.js 51
52. // screenshotter.js 52
53. // screenshotter.js const puppeteer = require('puppeteer'); 53
54. // screenshotter.js const puppeteer = require('puppeteer'); // read arguments from command line const args = process.argv.slice(2); 54
55. // screenshotter.js const puppeteer = require('puppeteer'); // read arguments from command line const args = process.argv.slice(2); // async IIFE - because we hate ourselves (async () => { })(); 55
56. // screenshotter.js const puppeteer = require('puppeteer'); // read arguments from command line const args = process.argv.slice(2); // async IIFE - because we hate ourselves (async () => { // launch a browser and open a page (we provide on command line) const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto(args[0]); })(); 56
57. // screenshotter.js const puppeteer = require('puppeteer'); // read arguments from command line const args = process.argv.slice(2); // async IIFE - because we hate ourselves (async () => { // launch a browser and open a page (we provide on command line) const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto(args[0]); // take a screenshot await page.screenshot({path: `${args[1]}.png`}); })(); 57
58. // screenshotter.js const puppeteer = require('puppeteer'); // read arguments from command line const args = process.argv.slice(2); // async IIFE - because we hate ourselves (async () => { // launch a browser and open a page (we provide on command line) const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto(args[0]); // take a screenshot await page.screenshot({path: `${args[1]}.png`}); // close the browser await browser.close(); })(); 58
59. # prepare a screen capture utility $ yarn add puppeteer puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline 59
61. # prepare a screen capture utility $ yarn add puppeteer puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility 61
62. # prepare a screen capture utility $ yarn add puppeteer puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility $ yarn add resemblejs 62
63. # prepare a screen capture utility $ yarn add puppeteer puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility $ yarn add resemblejs $ vim comparer.js 63
64. // comparer.js 64
65. // comparer.js // we import filesystem to read our images const fs = require(‘fs'); 65
66. // comparer.js // we import filesystem to read our images const fs = require(‘fs'); // and then compareImages utility from resembleJS const compareImages = require(‘resemblejs/compareImages'); 66
67. // comparer.js // we import filesystem to read our images const fs = require(‘fs'); // and then compareImages utility from resembleJS const compareImages = require(‘resemblejs/compareImages'); // again, we also need to get our arguments from the command line const args = process.argv.slice(2); 67
68. // comparer.js // we import filesystem to read our images const fs = require(‘fs'); // and then compareImages utility from resembleJS const compareImages = require(‘resemblejs/compareImages'); // again, we also need to get our arguments from the command line const args = process.argv.slice(2); // read our images const baseline = fs.readFileSync(args[0]); const check = fs.readFileSync(args[1]); 68
69. // comparer.js // we import filesystem to read our images const fs = require(‘fs'); // and then compareImages utility from resembleJS const compareImages = require(‘resemblejs/compareImages'); // again, we also need to get our arguments from the command line const args = process.argv.slice(2); // read our images const baseline = fs.readFileSync(args[0]); const check = fs.readFileSync(args[1]); // and then compare, printing the comparison data to console compareImages(baseline, check) .then(data => console.log(data)); 69
70. # prepare a screen capture utility $ yarn add puppeteer puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility $ yarn add resemblejs $ vim comparer.js # check that it works 70
71. # prepare a screen capture utility $ yarn add puppeteer puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility $ yarn add resemblejs $ vim comparer.js # check that it works $ node screenshotter.js url check 71
72. # prepare a screen capture utility $ yarn add puppeteer puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility $ yarn add resemblejs $ vim comparer.js # check that it works $ node screenshotter.js url check $ node comparer.js baseline.png check.png 72
73. { isSameDimensions: true, dimensionDifference: { width: 0, height: 0 }, rawMisMatchPercentage: 0, misMatchPercentage: '0.00', diffBounds: { top: 600, left: 800, bottom: 0, right: 0 }, analysisTime: 39, getImageDataUrl: [Function], getBuffer: [Function] } 73
74. # 0% difference is expected # lets make a change 74
75. # 0% difference is expected # lets make a change # styles.css img { border-radius: 150px; } 75
76. # 0% difference is expected # lets make a change # styles.css img { border-radius: 150px; } # let’s try again $ node screenshotter.js url check $ node comparer.js baseline.png check.png 76
77. { isSameDimensions: true, dimensionDifference: { width: 0, height: 0 }, rawMisMatchPercentage: 3.31145834, misMatchPercentage: '3.31', diffBounds: { top: 106, left: 264, bottom: 405, right: 535 }, analysisTime: 47, getImageDataUrl: [Function], getBuffer: [Function] } 77
78. // the output text came from here compareImages(baseline, check) .then(data => console.log(data)); 78
79. // the output text came from here compareImages(baseline, check) .then(data => console.log(data)); // but you may have noticed this line in the output // getBuffer: [Function] // this gives us a "diff" image of the two input images which we // can then save to the file system 79
80. // the output text came from here compareImages(baseline, check) .then(data => console.log(data)); // but you may have noticed this line in the output // getBuffer: [Function] // this gives us a "diff" image of the two input images which we // can then save to the file system // we can write a new function to do this: async function handleCompareData(data) { const suffix = args[1].substring(args[1].lastIndexOf('.')); const diff = args[1].replace(suffix, `.diff${suffix}`); await fs.writeFile(diff, data.getBuffer()); } 80
81. // the output text came from here compareImages(baseline, check) .then(data => console.log(data)); // but you may have noticed this line in the output // getBuffer: [Function] // this gives us a "diff" image of the two input images which we // can then save to the file system // we can write a new function to do this: async function handleCompareData(data) { const suffix = args[1].substring(args[1].lastIndexOf('.')); const diff = args[1].replace(suffix, `.diff${suffix}`); await fs.writeFile(diff, data.getBuffer()); } // and then replace our earlier code with: compareImages(baseline, check) .then(handleCompareData); 81
82. # so running it once again $ node comparer.js baseline.png check.png 82
83. # so running it once again $ node comparer.js baseline.png check.png # we can see the diff clearly 83
84. # so running it once again $ node comparer.js baseline.png check.png # we can see the diff clearly # and because it’s expected # we can recreate our baseline! 84
85. # but wait! # we almost forget the kitty! 85
86. # but wait! # we almost forget the kitty! # let’s check our change didn’t break anything $ node comparer.js kitty- baseline.png kitty-check.png 86
88. Lots of components Icons can be renamed or changed Text can change without client even knowing Components can have many states Buttons are a goldmine for changes More complex sites have more parts and more places to break - so VRT can become a much needed safety net Complex sites 88
89. How can our simple system be improved for large scale developments? 89
90. Hook it up to CI Automatically generate baseline and check any new branches/changes More automation! 90
91. Hook it up to CI Automatically generate baseline and check any new branches/changes Notify DEV and QA Slack, JIRA, Mattermost, Email - tell them if something regressed! More automation! 91
92. Hook it up to CI Automatically generate baseline and check any new branches/changes Notify DEV and QA Slack, JIRA, Mattermost, Email - tell them if something regressed! Improved interface Make it easier for developers and QA to spot problems More automation! 92
94. The downsides 94
95. Mutable content Automatically generate baseline and check any new branches/changes Downsides 95
96. Mutable content Automatically generate baseline and check any new branches/changes Maintenance New features, ABTests and general refactoring Downsides 96
97. Mutable content Automatically generate baseline and check any new branches/changes Maintenance New features, ABTests and general refactoring Not pixel perfect Antialiasing and animations Downsides 97
98. Mutable content Automatically generate baseline and check any new branches/changes Maintenance New features, ABTests and general refactoring Not pixel perfect Antialiasing and animations Downsides Centralised Generally used for local comparison - slow to compare local captures to remote 98
99. Sometimes an upside 99
100. Another level of safety Can catch issues missed by manual or functional testing Upsides 100
101. Another level of safety Can catch issues missed by manual or functional testing Documented and reproducible Tests are self documenting and generally reliable Upsides 101
102. Another level of safety Can catch issues missed by manual or functional testing Documented and reproducible Tests are self documenting and generally reliable Wide range Can automatically check visual changes on a variety of devices, resolutions, themes, brands, etc Upsides 102
103. A few real failures 103
104. Real failure 104
105. Real failure 105
106. Real failure 106
107. github.com/crawfordcarl/to-be-changed badootech.badoo.com github.com/crawfordcarl @stickycarl CHEERS! © 2019