Automating Visual Regression Testing with BackstopJS

Automating Visual Regression Testing with BackstopJS

Slides from my PNSQC 2018 talk on how my team and I used automated testing to help us catch content loss and style regressions.

3bbdbcf5326e7a6df8b2d6276208e70d?s=128

Angela Riggs

October 10, 2018
Tweet

Transcript

  1. Automating Visual Regression Testing with BackstopJS PNSQC | October 2018

  2. How It’s Useful and How We Used It @angelariggs_ PNSQC

    | October 2018
  3. Hi, I’m Angela! - QA Engineer at Vacasa - Teacher

    > Dev Intern > QA - Pronouns: She / Her - Twitter: @angelariggs_
  4. @angelariggs_ PNSQC | 2018 Front-end testing is hard

  5. @angelariggs_ PNSQC | 2018 Mobile Desktop Tablet

  6. @angelariggs_ PNSQC | 2018 Fonts Images Layout

  7. @angelariggs_ PNSQC | 2018 Fragile CSS ecosystems

  8. @angelariggs_ PNSQC | 2018

  9. @angelariggs_ PNSQC | 2018 Front-end testing is hard

  10. @angelariggs_ PNSQC | 2018 The Problem: Disappearing content on deploys

  11. @angelariggs_ PNSQC | 2018 Manual testing was tedious & ineffective

  12. @angelariggs_ PNSQC | 2018 BackstopJS

  13. @angelariggs_ PNSQC | 2018 - Open-source package

  14. @angelariggs_ PNSQC | 2018 - Open-source package - Before &

    after screenshots
  15. @angelariggs_ PNSQC | 2018 - Open-source package - Before &

    after screenshots - Chrome Headless, PhantomJS, or Slimer
  16. @angelariggs_ PNSQC | 2018 - Open-source package - Before &

    after screenshots - Chrome Headless, PhantomJS, or Slimer - UI simulations
  17. @angelariggs_ PNSQC | 2018 - Open-source package - Before &

    after screenshots - Chrome Headless, PhantomJS, or Slimer - UI simulations - HTML report for comparison
  18. @angelariggs_ PNSQC | 2018

  19. @angelariggs_ PNSQC | 2018

  20. @angelariggs_ PNSQC | 2018

  21. @angelariggs_ PNSQC | 2018

  22. @angelariggs_ PNSQC | 2018

  23. @angelariggs_ PNSQC | 2018

  24. @angelariggs_ PNSQC | 2018

  25. @angelariggs_ PNSQC | 2018

  26. @angelariggs_ PNSQC | 2018

  27. @angelariggs_ PNSQC | 2018

  28. @angelariggs_ PNSQC | 2018

  29. @angelariggs_ PNSQC | 2018

  30. @angelariggs_ PNSQC | 2018

  31. @angelariggs_ PNSQC | 2018

  32. @angelariggs_ PNSQC | 2018

  33. @angelariggs_ PNSQC | 2018

  34. @angelariggs_ PNSQC | 2018 Success: Automated visual regression testing helped

    us find and fix regressions before it became the client’s problem
  35. @angelariggs_ PNSQC | 2018 But…

  36. @angelariggs_ PNSQC | 2018

  37. @angelariggs_ PNSQC | 2018

  38. @angelariggs_ PNSQC | 2018

  39. @angelariggs_ PNSQC | 2018

  40. @angelariggs_ PNSQC | 2018

  41. @angelariggs_ PNSQC | 2018

  42. @angelariggs_ PNSQC | 2018 70 URLs

  43. @angelariggs_ PNSQC | 2018 70 URLs x 3 CSS selectors

  44. @angelariggs_ PNSQC | 2018 70 URLs x 3 CSS selectors

    x 2 viewports
  45. @angelariggs_ PNSQC | 2018 70 URLs x 3 CSS selectors

    x 2 viewports 420 screenshots
  46. @angelariggs_ PNSQC | 2018

  47. @angelariggs_ PNSQC | 2018 1. Run screenshot references

  48. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go

    play Towerfall for 35 minutes
  49. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go

    play Towerfall for 35 minutes 3. Developer deploys
  50. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go

    play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests
  51. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go

    play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch
  52. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go

    play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report
  53. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go

    play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report 7. Feedback to developer
  54. @angelariggs_ PNSQC | 2018 1. Run screenshot references* 2. Go

    play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests* 5. Go have lunch 6. Review report 7. Feedback to developer
  55. @angelariggs_ PNSQC | 2018 * Remember to include “no-timeout” parameter

  56. @angelariggs_ PNSQC | 2018

  57. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go

    play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report* 7. Feedback to developer
  58. @angelariggs_ PNSQC | 2018 “Automated”

  59. @angelariggs_ PNSQC | 2018 Narrowing the Scope

  60. @angelariggs_ PNSQC | 2018

  61. @angelariggs_ PNSQC | 2018

  62. @angelariggs_ PNSQC | 2018 - Increase viewport height to grab

    entire page - One CSS selector per URL - Remove URLs that use the same template
  63. @angelariggs_ PNSQC | 2018 420 screenshots

  64. @angelariggs_ PNSQC | 2018 420 screenshots ➡ 40 screenshots

  65. @angelariggs_ PNSQC | 2018

  66. @angelariggs_ PNSQC | 2018 1. Run screenshot references* 2. Go

    play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests* 5. Go have lunch 6. Review report* 7. Feedback to developer
  67. @angelariggs_ PNSQC | 2018 1. Tester runs screenshot references 2.

    Developer deploys 3. Tester runs screenshot test 4. Review report 5. Feedback to developer
  68. @angelariggs_ PNSQC | 2018 Smoke test for all staging and

    prod deploys
  69. @angelariggs_ PNSQC | 2018 But…

  70. @angelariggs_ PNSQC | 2018 - Four separate configuration files

  71. @angelariggs_ PNSQC | 2018 - Four separate configuration files -

    Lots of back & forth between developer and tester
  72. @angelariggs_ PNSQC | 2018

  73. @angelariggs_ PNSQC | 2018 More automated Still tedious

  74. @angelariggs_ PNSQC | 2018 - Developers oversee deploys

  75. @angelariggs_ PNSQC | 2018 - Developers oversee deploys - Developers

    should oversee deploy tests
  76. @angelariggs_ PNSQC | 2018 Lower the Barrier of Use

  77. @angelariggs_ PNSQC | 2018 Custom JavaScript configuration

  78. @angelariggs_ PNSQC | 2018

  79. @angelariggs_ PNSQC | 2018

  80. @angelariggs_ PNSQC | 2018

  81. @angelariggs_ PNSQC | 2018

  82. @angelariggs_ PNSQC | 2018

  83. @angelariggs_ PNSQC | 2018

  84. @angelariggs_ PNSQC | 2018

  85. @angelariggs_ PNSQC | 2018

  86. @angelariggs_ PNSQC | 2018

  87. @angelariggs_ PNSQC | 2018

  88. @angelariggs_ PNSQC | 2018

  89. @angelariggs_ PNSQC | 2018

  90. @angelariggs_ PNSQC | 2018 --env=prod

  91. @angelariggs_ PNSQC | 2018

  92. @angelariggs_ PNSQC | 2018

  93. @angelariggs_ PNSQC | 2018

  94. @angelariggs_ PNSQC | 2018

  95. @angelariggs_ PNSQC | 2018

  96. @angelariggs_ PNSQC | 2018

  97. @angelariggs_ PNSQC | 2018

  98. @angelariggs_ PNSQC | 2018

  99. @angelariggs_ PNSQC | 2018

  100. @angelariggs_ PNSQC | 2018

  101. @angelariggs_ PNSQC | 2018

  102. @angelariggs_ PNSQC | 2018

  103. @angelariggs_ PNSQC | 2018

  104. @angelariggs_ PNSQC | 2018

  105. @angelariggs_ PNSQC | 2018

  106. @angelariggs_ PNSQC | 2018

  107. @angelariggs_ PNSQC | 2018

  108. @angelariggs_ PNSQC | 2018

  109. @angelariggs_ PNSQC | 2018 4 config files

  110. @angelariggs_ PNSQC | 2018 4 config files ➡ 1 config

    file
  111. @angelariggs_ PNSQC | 2018 - detailed README for setup and

    use - template repo for quick setup on new projects
  112. @angelariggs_ PNSQC | 2018

  113. @angelariggs_ PNSQC | 2018 But…

  114. @angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod

  115. @angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod

  116. @angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod

  117. @angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod

  118. @angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod --refHost=http://prod-site.com

  119. @angelariggs_ PNSQC | 2018

  120. @angelariggs_ PNSQC | 2018 make prod-reference

  121. @angelariggs_ PNSQC | 2018 make prod-test

  122. @angelariggs_ PNSQC | 2018

  123. @angelariggs_ PNSQC | 2018 - Smoke tests for all deploys

    on all projects
  124. @angelariggs_ PNSQC | 2018 1. Dev runs references 2. Dev

    deploys 3. Dev runs tests 4. Dev reviews the report
  125. @angelariggs_ PNSQC | 2018 - Smoke tests for all deploys

    on all projects - Regression test during feature testing
  126. @angelariggs_ PNSQC | 2018 1. Run references on master branch

    2. Check out feature branch 3. Run tests on feature branch 4. Review the report
  127. @angelariggs_ PNSQC | 2018 - Smoke tests for all deploys

    on all projects - Regression test during feature testing - Local use during feature development
  128. @angelariggs_ PNSQC | 2018 1. Create and check out feature

    branch 2. Run references on fresh branch 3. Do local development 4. Run tests on branch periodically 5. Review report
  129. @angelariggs_ PNSQC | 2018 Front-end testing is hard easy

  130. @angelariggs_ PNSQC | 2018 Problem ➡ Solution

  131. @angelariggs_ PNSQC | 2018 Problem ➡ Idea

  132. @angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate

  133. @angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate ➡

    Iterate
  134. @angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate ➡

    Iterate ➡ Solution
  135. @angelariggs_ PNSQC | 2018 BackstopJS

  136. @angelariggs_ PNSQC | 2018 BackstopJS ➡ Refined configuration

  137. @angelariggs_ PNSQC | 2018 Refined config ➡ Customized config

  138. @angelariggs_ PNSQC | 2018 Automation is useful*

  139. @angelariggs_ PNSQC | 2018 Find the right automation solution for

    your problem
  140. @angelariggs_ PNSQC | 2018 Automated testing requires manual work

  141. @angelariggs_ PNSQC | 2018 Front-end testing is easy easier with

    the right automation
  142. https://github.com/angelariggs/visual-regression @angelariggs_ PNSQC | October 2018

  143. Thank you! @angelariggs_ PNSQC | October 2018