Automating Visual Regression Testing with BackstopJS

Automating Visual Regression Testing with BackstopJS PNSQC | October 2018

How It’s Useful and How We Used It @angelariggs_ PNSQC
| October 2018

Hi, I’m Angela! - QA Engineer at Vacasa - Teacher
> Dev Intern > QA - Pronouns: She / Her - Twitter: @angelariggs_

@angelariggs_ PNSQC | 2018 Front-end testing is hard

@angelariggs_ PNSQC | 2018 Mobile Desktop Tablet

@angelariggs_ PNSQC | 2018 Fonts Images Layout

@angelariggs_ PNSQC | 2018 Fragile CSS ecosystems

@angelariggs_ PNSQC | 2018

@angelariggs_ PNSQC | 2018 Front-end testing is hard

@angelariggs_ PNSQC | 2018 The Problem: Disappearing content on deploys

@angelariggs_ PNSQC | 2018 Manual testing was tedious & ineffective

@angelariggs_ PNSQC | 2018 BackstopJS

@angelariggs_ PNSQC | 2018 - Open-source package

@angelariggs_ PNSQC | 2018 - Open-source package - Before &
after screenshots

after screenshots - Chrome Headless, PhantomJS, or Slimer

after screenshots - Chrome Headless, PhantomJS, or Slimer - UI simulations

after screenshots - Chrome Headless, PhantomJS, or Slimer - UI simulations - HTML report for comparison

@angelariggs_ PNSQC | 2018 Success: Automated visual regression testing helped
us ﬁnd and ﬁx regressions before it became the client’s problem

@angelariggs_ PNSQC | 2018 But…

@angelariggs_ PNSQC | 2018 70 URLs

@angelariggs_ PNSQC | 2018 70 URLs x 3 CSS selectors

x 2 viewports

x 2 viewports 420 screenshots

@angelariggs_ PNSQC | 2018 1. Run screenshot references

@angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go
play Towerfall for 35 minutes

play Towerfall for 35 minutes 3. Developer deploys

play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests

play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch

play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report

play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report 7. Feedback to developer

@angelariggs_ PNSQC | 2018 1. Run screenshot references* 2. Go
play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests* 5. Go have lunch 6. Review report 7. Feedback to developer

@angelariggs_ PNSQC | 2018 * Remember to include “no-timeout” parameter

play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report* 7. Feedback to developer

@angelariggs_ PNSQC | 2018 “Automated”

@angelariggs_ PNSQC | 2018 Narrowing the Scope

@angelariggs_ PNSQC | 2018 - Increase viewport height to grab
entire page - One CSS selector per URL - Remove URLs that use the same template

@angelariggs_ PNSQC | 2018 420 screenshots

@angelariggs_ PNSQC | 2018 420 screenshots ➡ 40 screenshots

@angelariggs_ PNSQC | 2018 1. Run screenshot references* 2. Go
play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests* 5. Go have lunch 6. Review report* 7. Feedback to developer

@angelariggs_ PNSQC | 2018 1. Tester runs screenshot references 2.
Developer deploys 3. Tester runs screenshot test 4. Review report 5. Feedback to developer

@angelariggs_ PNSQC | 2018 Smoke test for all staging and
prod deploys

@angelariggs_ PNSQC | 2018 - Four separate conﬁguration ﬁles

@angelariggs_ PNSQC | 2018 - Four separate conﬁguration ﬁles -
Lots of back & forth between developer and tester

@angelariggs_ PNSQC | 2018 More automated Still tedious

@angelariggs_ PNSQC | 2018 - Developers oversee deploys

@angelariggs_ PNSQC | 2018 - Developers oversee deploys - Developers
should oversee deploy tests

@angelariggs_ PNSQC | 2018 Lower the Barrier of Use

@angelariggs_ PNSQC | 2018 Custom JavaScript conﬁguration

@angelariggs_ PNSQC | 2018 --env=prod

@angelariggs_ PNSQC | 2018 4 conﬁg ﬁles

@angelariggs_ PNSQC | 2018 4 config files ➡ 1 config
file

@angelariggs_ PNSQC | 2018 - detailed README for setup and
use - template repo for quick setup on new projects

@angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod

@angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod --refHost=http://prod-site.com

@angelariggs_ PNSQC | 2018 make prod-reference

@angelariggs_ PNSQC | 2018 make prod-test

@angelariggs_ PNSQC | 2018 - Smoke tests for all deploys
on all projects

@angelariggs_ PNSQC | 2018 1. Dev runs references 2. Dev
deploys 3. Dev runs tests 4. Dev reviews the report

on all projects - Regression test during feature testing

@angelariggs_ PNSQC | 2018 1. Run references on master branch
2. Check out feature branch 3. Run tests on feature branch 4. Review the report

on all projects - Regression test during feature testing - Local use during feature development

@angelariggs_ PNSQC | 2018 1. Create and check out feature
branch 2. Run references on fresh branch 3. Do local development 4. Run tests on branch periodically 5. Review report

@angelariggs_ PNSQC | 2018 Front-end testing is hard easy

@angelariggs_ PNSQC | 2018 Problem ➡ Solution

@angelariggs_ PNSQC | 2018 Problem ➡ Idea

@angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate

@angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate ➡
Iterate

@angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate ➡
Iterate ➡ Solution

@angelariggs_ PNSQC | 2018 BackstopJS

@angelariggs_ PNSQC | 2018 BackstopJS ➡ Reﬁned conﬁguration

@angelariggs_ PNSQC | 2018 Refined config ➡ Customized config

@angelariggs_ PNSQC | 2018 Automation is useful*

@angelariggs_ PNSQC | 2018 Find the right automation solution for
your problem

@angelariggs_ PNSQC | 2018 Automated testing requires manual work

@angelariggs_ PNSQC | 2018 Front-end testing is easy easier with
the right automation

https://github.com/angelariggs/visual-regression @angelariggs_ PNSQC | October 2018

Thank you! @angelariggs_ PNSQC | October 2018

Automating Visual Regression Testing with Backs...

Automating Visual Regression Testing with BackstopJS

More Decks by Angela Riggs

Other Decks in Programming

Featured

Transcript