Automating Visual Regression Testing with BackstopJS

Slides from my PNSQC 2018 talk on how my team and I used automated testing to help us catch content loss and style regressions.

Transcript

1. Automating Visual Regression Testing with BackstopJS PNSQC | October 2018
2. How It’s Useful and How We Used It @angelariggs_ PNSQC | October 2018
3. Hi, I’m Angela! - QA Engineer at Vacasa - Teacher > Dev Intern > QA - Pronouns: She / Her - Twitter: @angelariggs_
4. @angelariggs_ PNSQC | 2018 Front-end testing is hard
5. @angelariggs_ PNSQC | 2018 Mobile Desktop Tablet
6. @angelariggs_ PNSQC | 2018 Fonts Images Layout
7. @angelariggs_ PNSQC | 2018 Fragile CSS ecosystems
8. @angelariggs_ PNSQC | 2018
9. @angelariggs_ PNSQC | 2018 Front-end testing is hard
10. @angelariggs_ PNSQC | 2018 The Problem: Disappearing content on deploys
11. @angelariggs_ PNSQC | 2018 Manual testing was tedious & ineffective
12. @angelariggs_ PNSQC | 2018 BackstopJS
13. @angelariggs_ PNSQC | 2018 - Open-source package
14. @angelariggs_ PNSQC | 2018 - Open-source package - Before & after screenshots
15. @angelariggs_ PNSQC | 2018 - Open-source package - Before & after screenshots - Chrome Headless, PhantomJS, or Slimer
16. @angelariggs_ PNSQC | 2018 - Open-source package - Before & after screenshots - Chrome Headless, PhantomJS, or Slimer - UI simulations
17. @angelariggs_ PNSQC | 2018 - Open-source package - Before & after screenshots - Chrome Headless, PhantomJS, or Slimer - UI simulations - HTML report for comparison
18. @angelariggs_ PNSQC | 2018
19. @angelariggs_ PNSQC | 2018
20. @angelariggs_ PNSQC | 2018
21. @angelariggs_ PNSQC | 2018
22. @angelariggs_ PNSQC | 2018
23. @angelariggs_ PNSQC | 2018
24. @angelariggs_ PNSQC | 2018
25. @angelariggs_ PNSQC | 2018
26. @angelariggs_ PNSQC | 2018
27. @angelariggs_ PNSQC | 2018
28. @angelariggs_ PNSQC | 2018
29. @angelariggs_ PNSQC | 2018
30. @angelariggs_ PNSQC | 2018
31. @angelariggs_ PNSQC | 2018
32. @angelariggs_ PNSQC | 2018
33. @angelariggs_ PNSQC | 2018
34. @angelariggs_ PNSQC | 2018 Success: Automated visual regression testing helped us find and fix regressions before it became the client’s problem
35. @angelariggs_ PNSQC | 2018 But…
36. @angelariggs_ PNSQC | 2018
37. @angelariggs_ PNSQC | 2018
38. @angelariggs_ PNSQC | 2018
39. @angelariggs_ PNSQC | 2018
40. @angelariggs_ PNSQC | 2018
41. @angelariggs_ PNSQC | 2018
42. @angelariggs_ PNSQC | 2018 70 URLs
43. @angelariggs_ PNSQC | 2018 70 URLs x 3 CSS selectors
44. @angelariggs_ PNSQC | 2018 70 URLs x 3 CSS selectors x 2 viewports
45. @angelariggs_ PNSQC | 2018 70 URLs x 3 CSS selectors x 2 viewports 420 screenshots
46. @angelariggs_ PNSQC | 2018
47. @angelariggs_ PNSQC | 2018 1. Run screenshot references
48. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes
49. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys
50. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests
51. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch
52. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report
53. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report 7. Feedback to developer
54. @angelariggs_ PNSQC | 2018 1. Run screenshot references* 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests* 5. Go have lunch 6. Review report 7. Feedback to developer
55. @angelariggs_ PNSQC | 2018 * Remember to include “no-timeout” parameter
56. @angelariggs_ PNSQC | 2018
57. @angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report* 7. Feedback to developer
58. @angelariggs_ PNSQC | 2018 “Automated”
59. @angelariggs_ PNSQC | 2018 Narrowing the Scope
60. @angelariggs_ PNSQC | 2018
61. @angelariggs_ PNSQC | 2018
62. @angelariggs_ PNSQC | 2018 - Increase viewport height to grab entire page - One CSS selector per URL - Remove URLs that use the same template
63. @angelariggs_ PNSQC | 2018 420 screenshots
64. @angelariggs_ PNSQC | 2018 420 screenshots ➡ 40 screenshots
65. @angelariggs_ PNSQC | 2018
66. @angelariggs_ PNSQC | 2018 1. Run screenshot references* 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests* 5. Go have lunch 6. Review report* 7. Feedback to developer
67. @angelariggs_ PNSQC | 2018 1. Tester runs screenshot references 2. Developer deploys 3. Tester runs screenshot test 4. Review report 5. Feedback to developer
68. @angelariggs_ PNSQC | 2018 Smoke test for all staging and prod deploys
69. @angelariggs_ PNSQC | 2018 But…
70. @angelariggs_ PNSQC | 2018 - Four separate configuration files
71. @angelariggs_ PNSQC | 2018 - Four separate configuration files - Lots of back & forth between developer and tester
72. @angelariggs_ PNSQC | 2018
73. @angelariggs_ PNSQC | 2018 More automated Still tedious
74. @angelariggs_ PNSQC | 2018 - Developers oversee deploys
75. @angelariggs_ PNSQC | 2018 - Developers oversee deploys - Developers should oversee deploy tests
76. @angelariggs_ PNSQC | 2018 Lower the Barrier of Use
77. @angelariggs_ PNSQC | 2018 Custom JavaScript configuration
78. @angelariggs_ PNSQC | 2018
79. @angelariggs_ PNSQC | 2018
80. @angelariggs_ PNSQC | 2018
81. @angelariggs_ PNSQC | 2018
82. @angelariggs_ PNSQC | 2018
83. @angelariggs_ PNSQC | 2018
84. @angelariggs_ PNSQC | 2018
85. @angelariggs_ PNSQC | 2018
86. @angelariggs_ PNSQC | 2018
87. @angelariggs_ PNSQC | 2018
88. @angelariggs_ PNSQC | 2018
89. @angelariggs_ PNSQC | 2018
90. @angelariggs_ PNSQC | 2018 --env=prod
91. @angelariggs_ PNSQC | 2018
92. @angelariggs_ PNSQC | 2018
93. @angelariggs_ PNSQC | 2018
94. @angelariggs_ PNSQC | 2018
95. @angelariggs_ PNSQC | 2018
96. @angelariggs_ PNSQC | 2018
97. @angelariggs_ PNSQC | 2018
98. @angelariggs_ PNSQC | 2018
99. @angelariggs_ PNSQC | 2018
100. @angelariggs_ PNSQC | 2018
101. @angelariggs_ PNSQC | 2018
102. @angelariggs_ PNSQC | 2018
103. @angelariggs_ PNSQC | 2018
104. @angelariggs_ PNSQC | 2018
105. @angelariggs_ PNSQC | 2018
106. @angelariggs_ PNSQC | 2018
107. @angelariggs_ PNSQC | 2018
108. @angelariggs_ PNSQC | 2018
109. @angelariggs_ PNSQC | 2018 4 config files
110. @angelariggs_ PNSQC | 2018 4 config files ➡ 1 config file
111. @angelariggs_ PNSQC | 2018 - detailed README for setup and use - template repo for quick setup on new projects
112. @angelariggs_ PNSQC | 2018
113. @angelariggs_ PNSQC | 2018 But…
114. @angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod
115. @angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod
116. @angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod
117. @angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod
118. @angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod --refHost=http://prod-site.com
119. @angelariggs_ PNSQC | 2018
120. @angelariggs_ PNSQC | 2018 make prod-reference
121. @angelariggs_ PNSQC | 2018 make prod-test
122. @angelariggs_ PNSQC | 2018
123. @angelariggs_ PNSQC | 2018 - Smoke tests for all deploys on all projects
124. @angelariggs_ PNSQC | 2018 1. Dev runs references 2. Dev deploys 3. Dev runs tests 4. Dev reviews the report
125. @angelariggs_ PNSQC | 2018 - Smoke tests for all deploys on all projects - Regression test during feature testing
126. @angelariggs_ PNSQC | 2018 1. Run references on master branch 2. Check out feature branch 3. Run tests on feature branch 4. Review the report
127. @angelariggs_ PNSQC | 2018 - Smoke tests for all deploys on all projects - Regression test during feature testing - Local use during feature development
128. @angelariggs_ PNSQC | 2018 1. Create and check out feature branch 2. Run references on fresh branch 3. Do local development 4. Run tests on branch periodically 5. Review report
129. @angelariggs_ PNSQC | 2018 Front-end testing is hard easy
130. @angelariggs_ PNSQC | 2018 Problem ➡ Solution
131. @angelariggs_ PNSQC | 2018 Problem ➡ Idea
132. @angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate
133. @angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate ➡ Iterate
134. @angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate ➡ Iterate ➡ Solution
135. @angelariggs_ PNSQC | 2018 BackstopJS
136. @angelariggs_ PNSQC | 2018 BackstopJS ➡ Refined configuration
137. @angelariggs_ PNSQC | 2018 Refined config ➡ Customized config
138. @angelariggs_ PNSQC | 2018 Automation is useful*
139. @angelariggs_ PNSQC | 2018 Find the right automation solution for your problem
140. @angelariggs_ PNSQC | 2018 Automated testing requires manual work
141. @angelariggs_ PNSQC | 2018 Front-end testing is easy easier with the right automation
142. https://github.com/angelariggs/visual-regression @angelariggs_ PNSQC | October 2018
143. Thank you! @angelariggs_ PNSQC | October 2018