Pro Yearly is on sale from $80 to $50! »

CodeFest 2019. Карл Кроуфорд (Badoo) — Visual Regression Testing

16b6c87229eaf58768d25ed7b2bbbf52?s=47 CodeFest
April 05, 2019

CodeFest 2019. Карл Кроуфорд (Badoo) — Visual Regression Testing

Visual regression testing (VRT) is a weapon-like tool used by many QAs to protect against the inevitable and unexpected (but expected) dangers of development. In recent years the area has grown in popularity, with many more commercial and custom solutions becoming available — but what is it? How can we use it? And how hard is it to implement? In this presentation, I aim to share the principles of VRT while incrementally building a VRT solution from scratch, using Puppeteer and Headless Chrome. The plan is for you to come away with both knowledge and examples to help you bring VRT to your workplace.

16b6c87229eaf58768d25ed7b2bbbf52?s=128

CodeFest

April 05, 2019
Tweet

Transcript

  1. Visual Regression Testing (VRT) Carl Crawford · 30th of March

  2. Carl Crawford Frontend Engineer

  3. None
  4. > 418 000 000 people all over the world use

    our apps
  5. 1. What is Visual Regression Testing? 2. Why use it?

    3. When to use it? 4. Available tools? 5. Building VRT 6. Pros and Cons 7. Real Failures 5
  6. So what is visual regression testing? 6

  7. visual /ˈvɪʒ(j)ʊəl,ˈvɪzjʊəl/ adjective 1. relating to seeing or sight.
 


    regression /rɪˈɡrɛʃ(ə)n/ noun 1. a return to a former or less developed state. 7
  8. “Testing - that something visual - did not return to

    a former or less developed state” visual /ˈvɪʒ(j)ʊəl,ˈvɪzjʊəl/ adjective 1. relating to seeing or sight.
 
 regression /rɪˈɡrɛʃ(ə)n/ noun 1. a return to a former or less developed state. 8
  9. “Testing - that something visual - did not return to

    a former or less developed state” visual /ˈvɪʒ(j)ʊəl,ˈvɪzjʊəl/ adjective 1. relating to seeing or sight.
 
 regression /rɪˈɡrɛʃ(ə)n/ noun 1. a return to a former or less developed state. УХУДШЕ́ НИЕ Средний род 1. Изменение, ухудшающее что-н. 9
  10. Why VRT? Detecting visual changes is hard It’s another form

    of automated testing 10
  11. Why VRT? Detecting visual changes is hard It’s another form

    of automated testing Automation is easy and reliable Tests are descriptive, defined, and reproducible 11
  12. Why VRT? Detecting visual changes is hard It’s another form

    of automated testing Automation is easy and reliable Tests are descriptive, defined, and reproducible Avoid “change blindness” Magic word 12
  13. Change blindness is a perceptual phenomenon that occurs when a

    change in a visual stimulus is introduced and the observer does not notice it. 13
  14. Selective Attention Test 14

  15. https://youtu.be/vJG698U2Mvo

  16. Lets play a game of “spot the difference” 16

  17. 17

  18. How many differences did you spot? Lets make it easier

    18
  19. 19

  20. 20

  21. Easier? Lets see a computer do it… 21

  22. 22

  23. How does it work? The simple version 23

  24. 1 Generate a baseline How does VRT work? 24

  25. 1 Generate a baseline 2 Make some changes How does

    VRT work? 25
  26. 1 Generate a baseline 2 Make some changes 3 Take

    another screen capture How does VRT work? 26
  27. 1 Generate a baseline 2 Make some changes 3 Take

    another screen capture 4 Compare the two How does VRT work? 27
  28. Image courtesy of Nikhil Verma Where does it fit with

    traditional testing? 28
  29. What is it good for? 29

  30. 30

  31. 31

  32. 32

  33. What’s available? An awesome source of information for all VRT

    related topics provided by Alexander Wunshik https://github.com/mojoaxel/awesome-regression-testing 33
  34. 1. Browsers https://github.com/mojoaxel/awesome-regression-testing 34

  35. 1. Browsers 2. Frameworks https://github.com/mojoaxel/awesome-regression-testing 35

  36. 1. Browsers 2. Frameworks 3. Online Services https://github.com/mojoaxel/awesome-regression-testing 36

  37. 1. Browsers 2. Frameworks 3. Online Services 4. Blog posts

    https://github.com/mojoaxel/awesome-regression-testing 37
  38. 1. Browsers 2. Frameworks 3. Online Services 4. Blog posts

    5. Talks/Videos from the community https://github.com/mojoaxel/awesome-regression-testing 38
  39. Alternatively, build your own! 39

  40. 1 Browser and/or App Automation General Architecture 40

  41. 1 Browser and/or App Automation 2 Tool or framework for

    captures General Architecture 41
  42. 1 Browser and/or App Automation 2 Tool or framework for

    captures 3 Something to compare captures General Architecture 42
  43. A brief history of VRT in Badoo PhantomJS Load and

    control our site 43
  44. A brief history of VRT in Badoo PhantomJS Load and

    control our site Casper and PhantomCSS Provided us tools for scripting and comparing screenshots as tests 44
  45. A brief history of VRT in Badoo PhantomJS Load and

    control our site Casper and PhantomCSS Provided us tools for scripting and comparing screenshots as tests Custom interface Used to present the results to QA and developers in a user friendly fashion 45
  46. A brief history of VRT in Badoo Then along came

    Headless Chrome With puppeteer, it replaced PhantomJS and PhantomCSS for us 46
  47. A brief example please forgive my markup 47

  48. <html> <head> <link href=“styles.css” … /> </head> <body> <ul class="navbar">

    <li>About Me</li> <li>Kittens</li> </ul> <hr> <div class="content"> <div class="avatar"> <img src="me.png"> </div> <div> <p>Hey! my name is Carl</p> <p>I like cats!</p> <p>also, things</p> </div> </div> </body> </html> 48
  49. <html> <head> <link href=“styles.css” … /> </head> <body> <ul class="navbar">

    <li>About Me</li> <li>Kittens</li> </ul> <hr> <div class="content"> <div class="kitty"> <img src="mlem.png"> </div> </div> </body> </html> 49
  50. # prepare a screen capture utility $ yarn add puppeteer

    puppeteer-core 50
  51. # prepare a screen capture utility $ yarn add puppeteer

    puppeteer-core $ vim screenshotter.js 51
  52. // screenshotter.js 52

  53. // screenshotter.js const puppeteer = require('puppeteer'); 53

  54. // screenshotter.js const puppeteer = require('puppeteer'); // read arguments from

    command line const args = process.argv.slice(2); 54
  55. // screenshotter.js const puppeteer = require('puppeteer'); // read arguments from

    command line const args = process.argv.slice(2); // async IIFE - because we hate ourselves (async () => { })(); 55
  56. // screenshotter.js const puppeteer = require('puppeteer'); // read arguments from

    command line const args = process.argv.slice(2); // async IIFE - because we hate ourselves (async () => { // launch a browser and open a page (we provide on command line) const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto(args[0]); })(); 56
  57. // screenshotter.js const puppeteer = require('puppeteer'); // read arguments from

    command line const args = process.argv.slice(2); // async IIFE - because we hate ourselves (async () => { // launch a browser and open a page (we provide on command line) const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto(args[0]); // take a screenshot await page.screenshot({path: `${args[1]}.png`}); })(); 57
  58. // screenshotter.js const puppeteer = require('puppeteer'); // read arguments from

    command line const args = process.argv.slice(2); // async IIFE - because we hate ourselves (async () => { // launch a browser and open a page (we provide on command line) const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto(args[0]); // take a screenshot await page.screenshot({path: `${args[1]}.png`}); // close the browser await browser.close(); })(); 58
  59. # prepare a screen capture utility $ yarn add puppeteer

    puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline 59
  60. 60

  61. # prepare a screen capture utility $ yarn add puppeteer

    puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility 61
  62. # prepare a screen capture utility $ yarn add puppeteer

    puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility $ yarn add resemblejs 62
  63. # prepare a screen capture utility $ yarn add puppeteer

    puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility $ yarn add resemblejs $ vim comparer.js 63
  64. // comparer.js 64

  65. // comparer.js // we import filesystem to read our images

    const fs = require(‘fs'); 65
  66. // comparer.js // we import filesystem to read our images

    const fs = require(‘fs'); // and then compareImages utility from resembleJS const compareImages = require(‘resemblejs/compareImages'); 66
  67. // comparer.js // we import filesystem to read our images

    const fs = require(‘fs'); // and then compareImages utility from resembleJS const compareImages = require(‘resemblejs/compareImages'); // again, we also need to get our arguments from the command line const args = process.argv.slice(2); 67
  68. // comparer.js // we import filesystem to read our images

    const fs = require(‘fs'); // and then compareImages utility from resembleJS const compareImages = require(‘resemblejs/compareImages'); // again, we also need to get our arguments from the command line const args = process.argv.slice(2); // read our images const baseline = fs.readFileSync(args[0]); const check = fs.readFileSync(args[1]); 68
  69. // comparer.js // we import filesystem to read our images

    const fs = require(‘fs'); // and then compareImages utility from resembleJS const compareImages = require(‘resemblejs/compareImages'); // again, we also need to get our arguments from the command line const args = process.argv.slice(2); // read our images const baseline = fs.readFileSync(args[0]); const check = fs.readFileSync(args[1]); // and then compare, printing the comparison data to console compareImages(baseline, check) .then(data => console.log(data)); 69
  70. # prepare a screen capture utility $ yarn add puppeteer

    puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility $ yarn add resemblejs $ vim comparer.js # check that it works 70
  71. # prepare a screen capture utility $ yarn add puppeteer

    puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility $ yarn add resemblejs $ vim comparer.js # check that it works $ node screenshotter.js url check 71
  72. # prepare a screen capture utility $ yarn add puppeteer

    puppeteer-core $ vim screenshotter.js # test it by creating baseline images $ node screenshotter.js url baseline # prepare a comparison utility $ yarn add resemblejs $ vim comparer.js # check that it works $ node screenshotter.js url check $ node comparer.js baseline.png check.png 72
  73. { isSameDimensions: true, dimensionDifference: { width: 0, height: 0 },

    rawMisMatchPercentage: 0, misMatchPercentage: '0.00', diffBounds: { top: 600, left: 800, bottom: 0, right: 0 }, analysisTime: 39, getImageDataUrl: [Function], getBuffer: [Function] } 73
  74. # 0% difference is expected # lets make a change

    74
  75. # 0% difference is expected # lets make a change

    # styles.css img { border-radius: 150px; } 75
  76. # 0% difference is expected # lets make a change

    # styles.css img { border-radius: 150px; } # let’s try again $ node screenshotter.js url check $ node comparer.js baseline.png check.png 76
  77. { isSameDimensions: true, dimensionDifference: { width: 0, height: 0 },

    rawMisMatchPercentage: 3.31145834, misMatchPercentage: '3.31', diffBounds: { top: 106, left: 264, bottom: 405, right: 535 }, analysisTime: 47, getImageDataUrl: [Function], getBuffer: [Function] } 77
  78. // the output text came from here compareImages(baseline, check) .then(data

    => console.log(data)); 78
  79. // the output text came from here compareImages(baseline, check) .then(data

    => console.log(data)); // but you may have noticed this line in the output // getBuffer: [Function] // this gives us a "diff" image of the two input images which we // can then save to the file system 79
  80. // the output text came from here compareImages(baseline, check) .then(data

    => console.log(data)); // but you may have noticed this line in the output // getBuffer: [Function] // this gives us a "diff" image of the two input images which we // can then save to the file system // we can write a new function to do this: async function handleCompareData(data) { const suffix = args[1].substring(args[1].lastIndexOf('.')); const diff = args[1].replace(suffix, `.diff${suffix}`); await fs.writeFile(diff, data.getBuffer()); } 80
  81. // the output text came from here compareImages(baseline, check) .then(data

    => console.log(data)); // but you may have noticed this line in the output // getBuffer: [Function] // this gives us a "diff" image of the two input images which we // can then save to the file system // we can write a new function to do this: async function handleCompareData(data) { const suffix = args[1].substring(args[1].lastIndexOf('.')); const diff = args[1].replace(suffix, `.diff${suffix}`); await fs.writeFile(diff, data.getBuffer()); } // and then replace our earlier code with: compareImages(baseline, check) .then(handleCompareData); 81
  82. # so running it once again $ node comparer.js baseline.png

    check.png 82
  83. # so running it once again $ node comparer.js baseline.png

    check.png # we can see the diff clearly 83
  84. # so running it once again $ node comparer.js baseline.png

    check.png # we can see the diff clearly # and because it’s expected # we can recreate our baseline! 84
  85. # but wait! # we almost forget the kitty! 85

  86. # but wait! # we almost forget the kitty! #

    let’s check our change didn’t break anything $ node comparer.js kitty- baseline.png kitty-check.png 86
  87. 87

  88. Lots of components Icons can be renamed or changed Text

    can change without client even knowing Components can have many states Buttons are a goldmine for changes More complex sites have more parts and more places to break - so VRT can become a much needed safety net Complex sites 88
  89. How can our simple system be improved for large scale

    developments? 89
  90. Hook it up to CI Automatically generate baseline and check

    any new branches/changes More automation! 90
  91. Hook it up to CI Automatically generate baseline and check

    any new branches/changes Notify DEV and QA Slack, JIRA, Mattermost, Email - tell them if something regressed! More automation! 91
  92. Hook it up to CI Automatically generate baseline and check

    any new branches/changes Notify DEV and QA Slack, JIRA, Mattermost, Email - tell them if something regressed! Improved interface Make it easier for developers and QA to spot problems More automation! 92
  93. 93

  94. The downsides 94

  95. Mutable content Automatically generate baseline and check any new branches/changes

    Downsides 95
  96. Mutable content Automatically generate baseline and check any new branches/changes

    Maintenance New features, ABTests and general refactoring Downsides 96
  97. Mutable content Automatically generate baseline and check any new branches/changes

    Maintenance New features, ABTests and general refactoring Not pixel perfect Antialiasing and animations Downsides 97
  98. Mutable content Automatically generate baseline and check any new branches/changes

    Maintenance New features, ABTests and general refactoring Not pixel perfect Antialiasing and animations Downsides Centralised Generally used for local comparison - slow to compare local captures to remote 98
  99. Sometimes an upside 99

  100. Another level of safety Can catch issues missed by manual

    or functional testing Upsides 100
  101. Another level of safety Can catch issues missed by manual

    or functional testing Documented and reproducible Tests are self documenting and generally reliable Upsides 101
  102. Another level of safety Can catch issues missed by manual

    or functional testing Documented and reproducible Tests are self documenting and generally reliable Wide range Can automatically check visual changes on a variety of devices, resolutions, themes, brands, etc Upsides 102
  103. A few real failures 103

  104. Real failure 104

  105. Real failure 105

  106. Real failure 106

  107. github.com/crawfordcarl/to-be-changed badootech.badoo.com github.com/crawfordcarl @stickycarl CHEERS! © 2019