@angelariggs_ TSQA | 2020
A Beginner’s Guide
to the
Test Automation Galaxy
Slide 2
Slide 2 text
@angelariggs_ TSQA | 2020
Slide 3
Slide 3 text
@angelariggs_ TSQA | 2020
Embedded QAE
QAE for multiple teams
Centralized QAE
QA Manager at Instrument
Slide 4
Slide 4 text
@angelariggs_ TSQA | 2020
Embedded QAE
QAE for multiple teams
QA Manager at Instrument
Centralized QAE
Slide 5
Slide 5 text
@angelariggs_ TSQA | 2020
Embedded QAE
QAE for multiple teams
QA Manager at Instrument
Centralized QAE
Slide 6
Slide 6 text
@angelariggs_ TSQA | 2020
Embedded QAE
QAE for multiple teams
QA Manager at Instrument
Centralized QAE
Slide 7
Slide 7 text
@angelariggs_ TSQA | 2020
Embedded QAE
QAE for multiple teams
QA Manager at Instrument
Centralized QAE
Slide 8
Slide 8 text
@angelariggs_ TSQA | 2020
Embedded QAE
QAE for multiple teams
QA Manager at Instrument
Centralized QAE
Slide 9
Slide 9 text
@angelariggs_ TSQA | 2020
Embedded QAE
QAE for multiple teams
QA Manager at Instrument
Centralized QAE
Slide 10
Slide 10 text
@angelariggs_ TSQA | 2020
Embedded QAE
QAE for multiple teams
QA Manager at Instrument
Centralized QAE
Slide 11
Slide 11 text
@angelariggs_ TSQA | 2020
Itinerary
Slide 12
Slide 12 text
@angelariggs_ TSQA | 2020
Which tests to automate
Getting buy-in
Choosing a framework
Writing your tests
Running your tests
Value
Slide 13
Slide 13 text
@angelariggs_ TSQA | 2020
Which tests to automate
Getting buy-in
Choosing a framework
Writing your tests
Running your tests
Value
Slide 14
Slide 14 text
@angelariggs_ TSQA | 2020
Which tests to automate
Getting buy-in
Choosing a framework
Writing your tests
Running your tests
Value
Slide 15
Slide 15 text
@angelariggs_ TSQA | 2020
Which tests to automate
Getting buy-in
Choosing a framework
Writing your tests
Running your tests
Value
Slide 16
Slide 16 text
@angelariggs_ TSQA | 2020
Which tests to automate
Getting buy-in
Choosing a framework
Writing your tests
Running your tests
Value
Slide 17
Slide 17 text
@angelariggs_ TSQA | 2020
Which tests to automate
Getting buy-in
Choosing a framework
Writing your tests
Running your tests
Value
Slide 18
Slide 18 text
@angelariggs_ TSQA | 2020
This talk isn’t intended to give you all
the answers.
It’s to help you understand the
questions you need to ask, so you can
find the answers that work for you.
Slide 19
Slide 19 text
@angelariggs_ TSQA | 2020
Slide 20
Slide 20 text
@angelariggs_ TSQA | 2020
Slide 21
Slide 21 text
@angelariggs_ TSQA | 2020
Slide 22
Slide 22 text
@angelariggs_ TSQA | 2020
How do I choose a framework?
Who should write the tests?
Should the tests run in our pipeline?
What are some best practices?
❓
Slide 23
Slide 23 text
@angelariggs_ TSQA | 2020
Which tests should we automate?
Slide 24
Slide 24 text
@angelariggs_ TSQA | 2020
Prevent defects
Maintain quality
Meet users’ needs
Increase confidence
Understand our product
Why do we test?
Slide 25
Slide 25 text
@angelariggs_ TSQA | 2020
Why do we
automate?
Run tests faster
Better coverage
Increase reliability
Reusable / repeatable
More frequent testing
Slide 26
Slide 26 text
@angelariggs_ TSQA | 2020
Slide 27
Slide 27 text
@angelariggs_ TSQA | 2020
Automation is not the goal.
The goal is to have tests that
enhance your ability to achieve and
maintain quality.
Slide 28
Slide 28 text
@angelariggs_ TSQA | 2020
Which tests should we automate?
Slide 29
Slide 29 text
@angelariggs_ TSQA | 2020
Slide 30
Slide 30 text
@angelariggs_ TSQA | 2020
Low-risk or stable functionality
Slide 31
Slide 31 text
@angelariggs_ TSQA | 2020
Smoke tests
Low-level test to make
sure nothing’s on fire
Slide 32
Slide 32 text
@angelariggs_ TSQA | 2020
UI workflows
Avoids tedium of
manual testing or
“selective attention”
mistakes
Slide 33
Slide 33 text
@angelariggs_ TSQA | 2020
Visual validation
Compares before and
after snapshots across
the site or product
Slide 34
Slide 34 text
@angelariggs_ TSQA | 2020
Use automation for low-risk work
so you can focus on high-risk
or high-priority functionality.
Slide 35
Slide 35 text
@angelariggs_ TSQA | 2020
High-risk items that need
consistent testing
Slide 36
Slide 36 text
@angelariggs_ TSQA | 2020
API tests
Are your APIs handling
data the right way?
Slide 37
Slide 37 text
@angelariggs_ TSQA | 2020
Business-critical
workflows
Validate the UIs that
are high-impact,
or fall under
regulatory compliance
Slide 38
Slide 38 text
@angelariggs_ TSQA | 2020
Performance
testing
Understand your
product’s stability,
speed, and scalability
so you can future-proof
Slide 39
Slide 39 text
@angelariggs_ TSQA | 2020
Slide 40
Slide 40 text
@angelariggs_ TSQA | 2020
Getting Buy-In
Slide 41
Slide 41 text
@angelariggs_ TSQA | 2020
Automation is valid, meaningful work.
Slide 42
Slide 42 text
@angelariggs_ TSQA | 2020
What do you want to accomplish?
Slide 43
Slide 43 text
@angelariggs_ TSQA | 2020
How does this benefit them?
Slide 44
Slide 44 text
@angelariggs_ TSQA | 2020
Catch regressions
Sprint goals
Prevent defects
Team confidence
Stakeholder confidence
Benefits
for buy-in
Slide 45
Slide 45 text
@angelariggs_ TSQA | 2020
Improve your testing skills
Slide 46
Slide 46 text
@angelariggs_ TSQA | 2020
The goal of getting buy-in is to
make sure your automation efforts are
supported by the team and business,
so you can build automation
that supports them.
Slide 47
Slide 47 text
@angelariggs_ TSQA | 2020
Choosing a
Framework
Slide 48
Slide 48 text
@angelariggs_ TSQA | 2020
What is a test automation framework?
Slide 49
Slide 49 text
@angelariggs_ TSQA | 2020
The test automation framework is made
up of the tools and libraries that you use
to write the tests, as well as execute,
verify, and report on those tests.
Angie
Jones!
@angelariggs_ TSQA | 2020
There is no framework to rule them all.
The right tools are the ones that you’ll
actually use.
Slide 52
Slide 52 text
@angelariggs_ TSQA | 2020
Code-based or codeless?
Open-source or paid?
Slide 53
Slide 53 text
@angelariggs_ TSQA | 2020
Who’s writing the tests?
Slide 54
Slide 54 text
@angelariggs_ TSQA | 2020
How are you running your tests?
Slide 55
Slide 55 text
@angelariggs_ TSQA | 2020
Is there a budget?
Slide 56
Slide 56 text
@angelariggs_ TSQA | 2020
Is there a “freemium” level?
Slide 57
Slide 57 text
@angelariggs_ TSQA | 2020
Is there a trial period?
Slide 58
Slide 58 text
@angelariggs_ TSQA | 2020
What’s the documentation like?
Slide 59
Slide 59 text
@angelariggs_ TSQA | 2020
How actively is the framework maintained?
Slide 60
Slide 60 text
@angelariggs_ TSQA | 2020
Slide 61
Slide 61 text
@angelariggs_ TSQA | 2020
Slide 62
Slide 62 text
@angelariggs_ TSQA | 2020
Which tests to automate
Getting buy-in
Choosing a framework
Writing your tests
Running your tests
Recap
You are here.
Slide 63
Slide 63 text
@angelariggs_ TSQA | 2020
Slide 64
Slide 64 text
@angelariggs_ TSQA | 2020
Writing Your Tests
Slide 65
Slide 65 text
@angelariggs_ TSQA | 2020
The goal of automation is reliable,
anti-fragile tests that are simple to
maintain, update, and hand off.
Slide 66
Slide 66 text
@angelariggs_ TSQA | 2020
Test automation is code!
Slide 67
Slide 67 text
@angelariggs_ TSQA | 2020
Security
Slide 68
Slide 68 text
@angelariggs_ TSQA | 2020
Security
Slide 69
Slide 69 text
@angelariggs_ TSQA | 2020
Slide 70
Slide 70 text
@angelariggs_ TSQA | 2020
Clarity
Slide 71
Slide 71 text
@angelariggs_ TSQA | 2020
Write tests for Future You
Slide 72
Slide 72 text
@angelariggs_ TSQA | 2020
sleep(10)
Slide 73
Slide 73 text
@angelariggs_ TSQA | 2020
waitForElement(#footer)
Slide 74
Slide 74 text
@angelariggs_ TSQA | 2020
Independent tests
Slide 75
Slide 75 text
@angelariggs_ TSQA | 2020
Create reservation
✦ Enter location
✦ Select dates
✦ Select unit
✦ Book vacation
Modify reservation
✦ Look up reservation
✦ Select Edit option
✦ Change dates
✦ Confirm new dates
Slide 76
Slide 76 text
@angelariggs_ TSQA | 2020
Create reservation
✦ Enter location
✦ Select dates
✦ Select unit
✦ Book vacation
Modify reservation
✦ Look up reservation
✦ Select Edit option
✦ Change dates
✦ Confirm new dates
Slide 77
Slide 77 text
@angelariggs_ TSQA | 2020
Create reservation
✅ Enter location
✦ Select dates
✦ Select unit
✦ Book vacation
Modify reservation
✦ Look up reservation
✦ Select Edit option
✦ Change dates
✦ Confirm new dates
Slide 78
Slide 78 text
@angelariggs_ TSQA | 2020
Create reservation
✅ Enter location
✅ Select dates
✦ Select unit
✦ Book vacation
Modify reservation
✦ Look up reservation
✦ Select Edit option
✦ Change dates
✦ Confirm new dates
Slide 79
Slide 79 text
@angelariggs_ TSQA | 2020
Create reservation
✅ Enter location
✅ Select dates
✅ Select unit
✦ Book vacation
Modify reservation
✦ Look up reservation
✦ Select Edit option
✦ Change dates
✦ Confirm new dates
Slide 80
Slide 80 text
@angelariggs_ TSQA | 2020
Create reservation
✅ Enter location
✅ Select dates
✅ Select unit
❌ Book vacation
Modify reservation
✦ Look up reservation
✦ Select Edit option
✦ Change dates
✦ Confirm new dates
Slide 81
Slide 81 text
@angelariggs_ TSQA | 2020
Create reservation
✅ Enter location
✅ Select dates
✅ Select unit
❌ Book vacation
Modify reservation
❌ Look up reservation
❌ Select Edit option
❌ Change dates
❌ Confirm new dates
Slide 82
Slide 82 text
@angelariggs_ TSQA | 2020
Slide 83
Slide 83 text
@angelariggs_ TSQA | 2020
Independent tests reduce the fragility of
your test automation.
Slide 84
Slide 84 text
@angelariggs_ TSQA | 2020
Parallel testing
Slide 85
Slide 85 text
@angelariggs_ TSQA | 2020
Parallel testing means
running multiple automated tests
concurrently instead of consecutively.
Slide 86
Slide 86 text
@angelariggs_ TSQA | 2020
Minutes
0
1
2
3
4
5
6
Test 1
Test 2
Test 3
Test 4
Test 5
Test 6
Slide 87
Slide 87 text
@angelariggs_ TSQA | 2020
Minutes
0
1
2
3
4
5
6
Process 1 Process 2
Test 1
Test 3
Test 5
Test 2
Test 4
Test 6
Slide 88
Slide 88 text
@angelariggs_ TSQA | 2020
Parallel testing is faster testing,
and allows you to scale your
automation more effectively.
Slide 89
Slide 89 text
@angelariggs_ TSQA | 2020
Granularity
Slide 90
Slide 90 text
@angelariggs_ TSQA | 2020
Slide 91
Slide 91 text
@angelariggs_ TSQA | 2020
Slide 92
Slide 92 text
@angelariggs_ TSQA | 2020
{ Difficult to refactor }
Lack of granularity → dependent tests
Hard to read & maintain
❌
Slide 93
Slide 93 text
@angelariggs_ TSQA | 2020
Refactored into granular tests
↑ Independence, ↓ flakiness
Outline tests before writing
✅
Slide 94
Slide 94 text
@angelariggs_ TSQA | 2020
Think of your tests as narrating
your product’s functionality.
You want short, declarative
sentences rather than long
stream-of-consciousness narration.
Slide 95
Slide 95 text
@angelariggs_ TSQA | 2020
Failures
Slide 96
Slide 96 text
@angelariggs_ TSQA | 2020
Slide 97
Slide 97 text
@angelariggs_ TSQA | 2020
The goal of automation isn’t
having tests that pass.
The goal of automation is
having tests that
accurately catch failures.
Slide 98
Slide 98 text
@angelariggs_ TSQA | 2020
If a test never fails, is it providing value?
Slide 99
Slide 99 text
@angelariggs_ TSQA | 2020
Planning for failure means writing your tests
in a way that make those failures useful.
Slide 100
Slide 100 text
@angelariggs_ TSQA | 2020
“it 404’ed”
Slide 101
Slide 101 text
@angelariggs_ TSQA | 2020
“Ticket page sends a 404 if support
file does not exist”
Slide 102
Slide 102 text
@angelariggs_ TSQA | 2020
Running Your Tests
Slide 103
Slide 103 text
@angelariggs_ TSQA | 2020
Manual kick-off & reporting
Scheduled runs during the day
Integrated into a CI pipeline
Running
your tests
@angelariggs_ TSQA | 2020
Failures block deploy process
May need cross-team work to update
Cons
Slide 113
Slide 113 text
@angelariggs_ TSQA | 2020
Having tests is better than not having tests.
Pick a method, but be willing to iterate
and adjust if something isn’t working.
Slide 114
Slide 114 text
@angelariggs_ TSQA | 2020
Showing Value
Slide 115
Slide 115 text
@angelariggs_ TSQA | 2020
How can you show that the work
has been worth the effort?
Slide 116
Slide 116 text
@angelariggs_ TSQA | 2020
What benefits were you aiming for?
Slide 117
Slide 117 text
@angelariggs_ TSQA | 2020
Reduced number of regressions
Speed up development time
Team morale
Client confidence
Showing
value
Slide 118
Slide 118 text
@angelariggs_ TSQA | 2020
Test automation helps you mitigate
risk, validate functionality, and
prioritize your testing efforts.
Slide 119
Slide 119 text
@angelariggs_ TSQA | 2020
Slide 120
Slide 120 text
@angelariggs_ TSQA | 2020
Understand what questions to ask,
so you can find the answers that work for you
Slide 121
Slide 121 text
@angelariggs_ TSQA | 2020
Understand the benefits of automation,
so you know when to use it
Slide 122
Slide 122 text
@angelariggs_ TSQA | 2020
Understand how to get buy-in for
prioritizing automation work
Slide 123
Slide 123 text
@angelariggs_ TSQA | 2020
Understand how to research frameworks,
so you can find the one that meets your needs
Slide 124
Slide 124 text
@angelariggs_ TSQA | 2020
Understand your product’s fault lines,
so you can add meaningful tests
Slide 125
Slide 125 text
@angelariggs_ TSQA | 2020
Understand how new work integrates
with existing features
Slide 126
Slide 126 text
@angelariggs_ TSQA | 2020
Understand your stakeholders’ priorities,
so your tests can build confidence
Slide 127
Slide 127 text
@angelariggs_ TSQA | 2020
Understand automation best practices,
so you can create anti-fragile tests
Slide 128
Slide 128 text
@angelariggs_ TSQA | 2020
Understand the benefits and trade-offs
between methods for running your tests
Slide 129
Slide 129 text
@angelariggs_ TSQA | 2020
Understand the value of test automation
Slide 130
Slide 130 text
@angelariggs_ TSQA | 2020
Slide 131
Slide 131 text
@angelariggs_ TSQA | 2020
Thank you!
Slide 132
Slide 132 text
@angelariggs_ TSQA | 2020
Resources
Slide 133
Slide 133 text
@angelariggs_ TSQA | 2020
Lucy Wyman & Zach Reichert
Test Automation University
Are Our Tests Any Good?
http://bit.ly/are-our-tests-good
Online automation courses
http://bit.ly/test-automation-uni