Best Practices for Automated Regression Testing

Slide 1

Slide 1 text

Best Practices for Automated Regression Testing Jace Browning

Slide 2

Slide 2 text

Outline • Terminology • Goals • Example / Demo • Limitations • Recommendations

Slide 3

Slide 3 text

About Me +

Slide 4

Slide 4 text

Terminology

Slide 5

Slide 5 text

Unit Tests

Slide 6

Slide 6 text

Integration Tests

Slide 7

Slide 7 text

System Tests

Slide 8

Slide 8 text

Automated System Tests Automated Integration Tests Automated Unit Tests

Slide 9

Slide 9 text

Exploratory Testing Scripted Testing Automated Testing

Slide 10

Slide 10 text

Goals

Slide 11

Slide 11 text

Production Staging Review Review Review

Slide 12

Slide 12 text

Approval Production Staging Review Review Review

Slide 13

Slide 13 text

Promotion Production Staging Review Review Review

Slide 14

Slide 14 text

Validation Production Staging Review Review Review

Slide 15

Slide 15 text

Detection Production Staging Review Review Review

Slide 16

Slide 16 text

Example

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

WebDriver Basics

Slide 21

Slide 21 text

Finding Elements .payment-details__input cardNumber //*[@id=“app”]/div/div[2]/div/form/div/div/div/div/div[2]/div/label[1]/input

Slide 22

Slide 22 text

Finding Elements .async-button__text “Start Trial” //*[@id="app"]/div/div[2]/div/form/div/footer/div/button/span/span/span

Slide 23

Slide 23 text

Language Bindings from selenium import webdriver driver = webdriver.Firefox() driver.get("http://zapier.com/app/team-account-setup/billing") assert "Start Your 14-Day Free Trial" in driver.html element = driver.find_element_by_name("cardNumber") element.send_keys("4242424242424242") element = driver.find_element_by_link_text("Start Trial") element.click() assert "Invite Your Teammates" in driver.html

Slide 24

Slide 24 text

Abstraction Layers from splinter import Browser browser = Browser("firefox") browser.visit("http://zapier.com/app/team-account-setup/billing") assert "Start Your 14-Day Free Trial" in browser.html browser.fill("cardNumber", "4242424242424242") browser.find_by_text("Start Trial").click() assert "Invite Your Teammates" in browser.html

Slide 25

Slide 25 text

Page Object Model class TeamSetupBilling: URL = "http://zapier.com/app/team-account-setup/billing" def __init__(self): self.browser = Browser("firefox") self.browser.visit(self.URL) def __contains__(self, value): return value in self.browser.html def fill_credit_card_number(self, value): self.browser.fill("cardNumber", value) def click_start_trial(self): self.browser.find_by_text("Start Trial").click()

Slide 26

Slide 26 text

Page Object Model page = TeamSetupBilling() assert "Start Your 14-Day Free Trial" in page page.fill_credit_card_number("4242424242424242") page.click_start_trial() assert "Invite Your Teammates" in page

Slide 27

Slide 27 text

Helper Functions def create_team(team_name, team=None, credit_card=None): if team is None: team = get_team(team_name) if credit_card is None: credit_card = get_credit_card() page = views.TeamSetup() return page.complete_wizard(team, credit_card)

Slide 28

Slide 28 text

Helper Functions helpers.create_team(team_name="Beer City") page = TeamSettings() assert "Beer City" in page page.fill_team_members("[email protected]") page.click_invite() assert "Pending Invite" in page

Slide 29

Slide 29 text

Demo

Slide 30

Slide 30 text

Limitations

Slide 31

Slide 31 text

Dependencies • Internal Data • External Services • Browsers

Slide 32

Slide 32 text

Authentication • OAuth endpoints • Two-factor authentication • Administrative features

Slide 33

Slide 33 text

User Experience • Designs • Animations / Rendering / Responsiveness • Experiments

Slide 34

Slide 34 text

Maintenance • Upkeep • Monitoring • Coverage

Slide 35

Slide 35 text

Recommendations

Slide 36

Slide 36 text

Testability • Named inputs • Scoped CSS • IDs where necessary

Slide 37

Slide 37 text

Cheats • Security bypasses • Test credit cards • Exposed internals

Slide 38

Slide 38 text

Cleanup • Identiﬁable test users • Filter from metrics • Delete old accounts

Slide 39

Slide 39 text

Roadmap 1. Sign up / Log in 2. Run after every deploy 3. Low-coverge, High-impact 4. Business critical 5. Postmortems / Retrospectives

Slide 40

Slide 40 text

Questions? @jacebrowning zapier.com