Automating Visual Regression Testing with BackstopJS

Slide 1

Slide 1 text

Automating Visual Regression Testing with BackstopJS PNSQC | October 2018

Slide 2

Slide 2 text

How It’s Useful and How We Used It @angelariggs_ PNSQC | October 2018

Slide 3

Slide 3 text

Hi, I’m Angela! - QA Engineer at Vacasa - Teacher > Dev Intern > QA - Pronouns: She / Her - Twitter: @angelariggs_

Slide 4

Slide 4 text

@angelariggs_ PNSQC | 2018 Front-end testing is hard

Slide 5

Slide 5 text

@angelariggs_ PNSQC | 2018 Mobile Desktop Tablet

Slide 6

Slide 6 text

@angelariggs_ PNSQC | 2018 Fonts Images Layout

Slide 7

Slide 7 text

@angelariggs_ PNSQC | 2018 Fragile CSS ecosystems

Slide 8

Slide 8 text

@angelariggs_ PNSQC | 2018

Slide 9

Slide 9 text

@angelariggs_ PNSQC | 2018 Front-end testing is hard

Slide 10

Slide 10 text

@angelariggs_ PNSQC | 2018 The Problem: Disappearing content on deploys

Slide 11

Slide 11 text

@angelariggs_ PNSQC | 2018 Manual testing was tedious & ineffective

Slide 12

Slide 12 text

@angelariggs_ PNSQC | 2018 BackstopJS

Slide 13

Slide 13 text

@angelariggs_ PNSQC | 2018 - Open-source package

Slide 14

Slide 14 text

@angelariggs_ PNSQC | 2018 - Open-source package - Before & after screenshots

Slide 15

Slide 15 text

@angelariggs_ PNSQC | 2018 - Open-source package - Before & after screenshots - Chrome Headless, PhantomJS, or Slimer

Slide 16

Slide 16 text

@angelariggs_ PNSQC | 2018 - Open-source package - Before & after screenshots - Chrome Headless, PhantomJS, or Slimer - UI simulations

Slide 17

Slide 17 text

@angelariggs_ PNSQC | 2018 - Open-source package - Before & after screenshots - Chrome Headless, PhantomJS, or Slimer - UI simulations - HTML report for comparison

Slide 18

Slide 18 text

@angelariggs_ PNSQC | 2018

Slide 19

Slide 19 text

@angelariggs_ PNSQC | 2018

Slide 20

Slide 20 text

@angelariggs_ PNSQC | 2018

Slide 21

Slide 21 text

@angelariggs_ PNSQC | 2018

Slide 22

Slide 22 text

@angelariggs_ PNSQC | 2018

Slide 23

Slide 23 text

@angelariggs_ PNSQC | 2018

Slide 24

Slide 24 text

@angelariggs_ PNSQC | 2018

Slide 25

Slide 25 text

@angelariggs_ PNSQC | 2018

Slide 26

Slide 26 text

@angelariggs_ PNSQC | 2018

Slide 27

Slide 27 text

@angelariggs_ PNSQC | 2018

Slide 28

Slide 28 text

@angelariggs_ PNSQC | 2018

Slide 29

Slide 29 text

@angelariggs_ PNSQC | 2018

Slide 30

Slide 30 text

@angelariggs_ PNSQC | 2018

Slide 31

Slide 31 text

@angelariggs_ PNSQC | 2018

Slide 32

Slide 32 text

@angelariggs_ PNSQC | 2018

Slide 33

Slide 33 text

@angelariggs_ PNSQC | 2018

Slide 34

Slide 34 text

@angelariggs_ PNSQC | 2018 Success: Automated visual regression testing helped us ﬁnd and ﬁx regressions before it became the client’s problem

Slide 35

Slide 35 text

@angelariggs_ PNSQC | 2018 But…

Slide 36

Slide 36 text

@angelariggs_ PNSQC | 2018

Slide 37

Slide 37 text

@angelariggs_ PNSQC | 2018

Slide 38

Slide 38 text

@angelariggs_ PNSQC | 2018

Slide 39

Slide 39 text

@angelariggs_ PNSQC | 2018

Slide 40

Slide 40 text

@angelariggs_ PNSQC | 2018

Slide 41

Slide 41 text

@angelariggs_ PNSQC | 2018

Slide 42

Slide 42 text

@angelariggs_ PNSQC | 2018 70 URLs

Slide 43

Slide 43 text

@angelariggs_ PNSQC | 2018 70 URLs x 3 CSS selectors

Slide 44

Slide 44 text

@angelariggs_ PNSQC | 2018 70 URLs x 3 CSS selectors x 2 viewports

Slide 45

Slide 45 text

@angelariggs_ PNSQC | 2018 70 URLs x 3 CSS selectors x 2 viewports 420 screenshots

Slide 46

Slide 46 text

@angelariggs_ PNSQC | 2018

Slide 47

Slide 47 text

@angelariggs_ PNSQC | 2018 1. Run screenshot references

Slide 48

Slide 48 text

@angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes

Slide 49

Slide 49 text

@angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys

Slide 50

Slide 50 text

@angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests

Slide 51

Slide 51 text

@angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch

Slide 52

Slide 52 text

@angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report

Slide 53

Slide 53 text

@angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report 7. Feedback to developer

Slide 54

Slide 54 text

@angelariggs_ PNSQC | 2018 1. Run screenshot references* 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests* 5. Go have lunch 6. Review report 7. Feedback to developer

Slide 55

Slide 55 text

@angelariggs_ PNSQC | 2018 * Remember to include “no-timeout” parameter

Slide 56

Slide 56 text

@angelariggs_ PNSQC | 2018

Slide 57

Slide 57 text

@angelariggs_ PNSQC | 2018 1. Run screenshot references 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests 5. Go have lunch 6. Review report* 7. Feedback to developer

Slide 58

Slide 58 text

@angelariggs_ PNSQC | 2018 “Automated”

Slide 59

Slide 59 text

@angelariggs_ PNSQC | 2018 Narrowing the Scope

Slide 60

Slide 60 text

@angelariggs_ PNSQC | 2018

Slide 61

Slide 61 text

@angelariggs_ PNSQC | 2018

Slide 62

Slide 62 text

@angelariggs_ PNSQC | 2018 - Increase viewport height to grab entire page - One CSS selector per URL - Remove URLs that use the same template

Slide 63

Slide 63 text

@angelariggs_ PNSQC | 2018 420 screenshots

Slide 64

Slide 64 text

@angelariggs_ PNSQC | 2018 420 screenshots ➡ 40 screenshots

Slide 65

Slide 65 text

@angelariggs_ PNSQC | 2018

Slide 66

Slide 66 text

@angelariggs_ PNSQC | 2018 1. Run screenshot references* 2. Go play Towerfall for 35 minutes 3. Developer deploys 4. Run screenshot tests* 5. Go have lunch 6. Review report* 7. Feedback to developer

Slide 67

Slide 67 text

@angelariggs_ PNSQC | 2018 1. Tester runs screenshot references 2. Developer deploys 3. Tester runs screenshot test 4. Review report 5. Feedback to developer

Slide 68

Slide 68 text

@angelariggs_ PNSQC | 2018 Smoke test for all staging and prod deploys

Slide 69

Slide 69 text

@angelariggs_ PNSQC | 2018 But…

Slide 70

Slide 70 text

@angelariggs_ PNSQC | 2018 - Four separate conﬁguration ﬁles

Slide 71

Slide 71 text

@angelariggs_ PNSQC | 2018 - Four separate conﬁguration ﬁles - Lots of back & forth between developer and tester

Slide 72

Slide 72 text

@angelariggs_ PNSQC | 2018

Slide 73

Slide 73 text

@angelariggs_ PNSQC | 2018 More automated Still tedious

Slide 74

Slide 74 text

@angelariggs_ PNSQC | 2018 - Developers oversee deploys

Slide 75

Slide 75 text

@angelariggs_ PNSQC | 2018 - Developers oversee deploys - Developers should oversee deploy tests

Slide 76

Slide 76 text

@angelariggs_ PNSQC | 2018 Lower the Barrier of Use

Slide 77

Slide 77 text

@angelariggs_ PNSQC | 2018 Custom JavaScript conﬁguration

Slide 78

Slide 78 text

@angelariggs_ PNSQC | 2018

Slide 79

Slide 79 text

@angelariggs_ PNSQC | 2018

Slide 80

Slide 80 text

@angelariggs_ PNSQC | 2018

Slide 81

Slide 81 text

@angelariggs_ PNSQC | 2018

Slide 82

Slide 82 text

@angelariggs_ PNSQC | 2018

Slide 83

Slide 83 text

@angelariggs_ PNSQC | 2018

Slide 84

Slide 84 text

@angelariggs_ PNSQC | 2018

Slide 85

Slide 85 text

@angelariggs_ PNSQC | 2018

Slide 86

Slide 86 text

@angelariggs_ PNSQC | 2018

Slide 87

Slide 87 text

@angelariggs_ PNSQC | 2018

Slide 88

Slide 88 text

@angelariggs_ PNSQC | 2018

Slide 89

Slide 89 text

@angelariggs_ PNSQC | 2018

Slide 90

Slide 90 text

@angelariggs_ PNSQC | 2018 --env=prod

Slide 91

Slide 91 text

@angelariggs_ PNSQC | 2018

Slide 92

Slide 92 text

@angelariggs_ PNSQC | 2018

Slide 93

Slide 93 text

@angelariggs_ PNSQC | 2018

Slide 94

Slide 94 text

@angelariggs_ PNSQC | 2018

Slide 95

Slide 95 text

@angelariggs_ PNSQC | 2018

Slide 96

Slide 96 text

@angelariggs_ PNSQC | 2018

Slide 97

Slide 97 text

@angelariggs_ PNSQC | 2018

Slide 98

Slide 98 text

@angelariggs_ PNSQC | 2018

Slide 99

Slide 99 text

@angelariggs_ PNSQC | 2018

Slide 100

Slide 100 text

@angelariggs_ PNSQC | 2018

Slide 101

Slide 101 text

@angelariggs_ PNSQC | 2018

Slide 102

Slide 102 text

@angelariggs_ PNSQC | 2018

Slide 103

Slide 103 text

@angelariggs_ PNSQC | 2018

Slide 104

Slide 104 text

@angelariggs_ PNSQC | 2018

Slide 105

Slide 105 text

@angelariggs_ PNSQC | 2018

Slide 106

Slide 106 text

@angelariggs_ PNSQC | 2018

Slide 107

Slide 107 text

@angelariggs_ PNSQC | 2018

Slide 108

Slide 108 text

@angelariggs_ PNSQC | 2018

Slide 109

Slide 109 text

@angelariggs_ PNSQC | 2018 4 conﬁg ﬁles

Slide 110

Slide 110 text

@angelariggs_ PNSQC | 2018 4 config files ➡ 1 config file

Slide 111

Slide 111 text

@angelariggs_ PNSQC | 2018 - detailed README for setup and use - template repo for quick setup on new projects

Slide 112

Slide 112 text

@angelariggs_ PNSQC | 2018

Slide 113

Slide 113 text

@angelariggs_ PNSQC | 2018 But…

Slide 114

Slide 114 text

@angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod

Slide 115

Slide 115 text

@angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod

Slide 116

Slide 116 text

@angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod

Slide 117

Slide 117 text

@angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod

Slide 118

Slide 118 text

@angelariggs_ PNSQC | 2018 backstop reference --configPath=backstop.js --pathFile=paths --env=prod --refHost=http://prod-site.com

Slide 119

Slide 119 text

@angelariggs_ PNSQC | 2018

Slide 120

Slide 120 text

@angelariggs_ PNSQC | 2018 make prod-reference

Slide 121

Slide 121 text

@angelariggs_ PNSQC | 2018 make prod-test

Slide 122

Slide 122 text

@angelariggs_ PNSQC | 2018

Slide 123

Slide 123 text

@angelariggs_ PNSQC | 2018 - Smoke tests for all deploys on all projects

Slide 124

Slide 124 text

@angelariggs_ PNSQC | 2018 1. Dev runs references 2. Dev deploys 3. Dev runs tests 4. Dev reviews the report

Slide 125

Slide 125 text

@angelariggs_ PNSQC | 2018 - Smoke tests for all deploys on all projects - Regression test during feature testing

Slide 126

Slide 126 text

@angelariggs_ PNSQC | 2018 1. Run references on master branch 2. Check out feature branch 3. Run tests on feature branch 4. Review the report

Slide 127

Slide 127 text

@angelariggs_ PNSQC | 2018 - Smoke tests for all deploys on all projects - Regression test during feature testing - Local use during feature development

Slide 128

Slide 128 text

@angelariggs_ PNSQC | 2018 1. Create and check out feature branch 2. Run references on fresh branch 3. Do local development 4. Run tests on branch periodically 5. Review report

Slide 129

Slide 129 text

@angelariggs_ PNSQC | 2018 Front-end testing is hard easy

Slide 130

Slide 130 text

@angelariggs_ PNSQC | 2018 Problem ➡ Solution

Slide 131

Slide 131 text

@angelariggs_ PNSQC | 2018 Problem ➡ Idea

Slide 132

Slide 132 text

@angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate

Slide 133

Slide 133 text

@angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate ➡ Iterate

Slide 134

Slide 134 text

@angelariggs_ PNSQC | 2018 Problem ➡ Idea ➡ Iterate ➡ Iterate ➡ Solution

Slide 135

Slide 135 text

@angelariggs_ PNSQC | 2018 BackstopJS

Slide 136

Slide 136 text

@angelariggs_ PNSQC | 2018 BackstopJS ➡ Reﬁned conﬁguration

Slide 137

Slide 137 text

@angelariggs_ PNSQC | 2018 Refined config ➡ Customized config

Slide 138

Slide 138 text

@angelariggs_ PNSQC | 2018 Automation is useful*

Slide 139

Slide 139 text

@angelariggs_ PNSQC | 2018 Find the right automation solution for your problem

Slide 140

Slide 140 text

@angelariggs_ PNSQC | 2018 Automated testing requires manual work

Slide 141

Slide 141 text

@angelariggs_ PNSQC | 2018 Front-end testing is easy easier with the right automation

Slide 142

Slide 142 text

https://github.com/angelariggs/visual-regression @angelariggs_ PNSQC | October 2018

Slide 143

Slide 143 text

Thank you! @angelariggs_ PNSQC | October 2018