Best Practices for Automated Regression Testing

So you wrote a bunch of great unit tests but your site is still broken? Maybe it’s time to start investing in a suite of end-to-end regression tests. In this talk you’ll learn best practices for incorporating Selenium into your QA process while avoiding some common pitfalls.

This talk was given at Beer City Code in Grand Rapids, MI.

Transcript

1. Best Practices
   for Automated
   Regression Testing
   Jace Browning
   

   View Slide

2. Outline
   • Terminology
   
   • Goals
   
   • Example / Demo
   
   • Limitations
   
   • Recommendations
   

   View Slide

3. About Me
   +
   

   View Slide

4. Terminology
   

   View Slide

5. Unit Tests
   

   View Slide

6. Integration Tests
   

   View Slide

7. System Tests
   

   View Slide

8. Automated
   System Tests
   Automated
   Integration Tests
   Automated
   Unit Tests
   

   View Slide

9. Exploratory Testing
   Scripted Testing
   Automated
   Testing
   

   View Slide

10. Goals
    

    View Slide

11. Production
    Staging
    Review
    Review
    Review
    

    View Slide

12. Approval
    Production
    Staging
    Review
    Review
    Review
    

    View Slide

13. Promotion
    Production
    Staging
    Review
    Review
    Review
    

    View Slide

14. Validation
    Production
    Staging
    Review
    Review
    Review
    

    View Slide

15. Detection
    Production
    Staging
    Review
    Review
    Review
    

    View Slide

16. Example
    

    View Slide

17. View Slide

18. View Slide

19. View Slide

20. WebDriver Basics
    

    View Slide

21. Finding Elements
    .payment-details__input
    cardNumber
    //*[@id=“app”]/div/div[2]/div/form/div/div/div/div/div[2]/div/label[1]/input
    

    View Slide

22. Finding Elements
    .async-button__text
    “Start Trial”
    //*[@id="app"]/div/div[2]/div/form/div/footer/div/button/span/span/span
    

    View Slide

23. Language Bindings
    from selenium import webdriver
    driver = webdriver.Firefox()
    driver.get("http://zapier.com/app/team-account-setup/billing")
    assert "Start Your 14-Day Free Trial" in driver.html
    element = driver.find_element_by_name("cardNumber")
    element.send_keys("4242424242424242")
    element = driver.find_element_by_link_text("Start Trial")
    element.click()
    assert "Invite Your Teammates" in driver.html
    

    View Slide

24. Abstraction Layers
    from splinter import Browser
    browser = Browser("firefox")
    browser.visit("http://zapier.com/app/team-account-setup/billing")
    assert "Start Your 14-Day Free Trial" in browser.html
    browser.fill("cardNumber", "4242424242424242")
    browser.find_by_text("Start Trial").click()
    assert "Invite Your Teammates" in browser.html
    

    View Slide

25. Page Object Model
    class TeamSetupBilling:
    URL = "http://zapier.com/app/team-account-setup/billing"
    def __init__(self):
    self.browser = Browser("firefox")
    self.browser.visit(self.URL)
    def __contains__(self, value):
    return value in self.browser.html
    def fill_credit_card_number(self, value):
    self.browser.fill("cardNumber", value)
    def click_start_trial(self):
    self.browser.find_by_text("Start Trial").click()
    

    View Slide

26. Page Object Model
    page = TeamSetupBilling()
    assert "Start Your 14-Day Free Trial" in page
    page.fill_credit_card_number("4242424242424242")
    page.click_start_trial()
    assert "Invite Your Teammates" in page
    

    View Slide

27. Helper Functions
    def create_team(team_name, team=None, credit_card=None):
    if team is None:
    team = get_team(team_name)
    if credit_card is None:
    credit_card = get_credit_card()
    page = views.TeamSetup()
    return page.complete_wizard(team, credit_card)
    

    View Slide

28. Helper Functions
    helpers.create_team(team_name="Beer City")
    page = TeamSettings()
    assert "Beer City" in page
    page.fill_team_members("jane@beercitycode.com")
    page.click_invite()
    assert "Pending Invite" in page
    

    View Slide

29. Demo
    

    View Slide

30. Limitations
    

    View Slide

31. Dependencies
    • Internal Data
    
    • External Services
    
    • Browsers
    

    View Slide

32. Authentication
    • OAuth endpoints
    
    • Two-factor authentication
    
    • Administrative features
    

    View Slide

33. User Experience
    • Designs
    
    • Animations / Rendering / Responsiveness
    
    • Experiments
    

    View Slide

34. Maintenance
    • Upkeep
    
    • Monitoring
    
    • Coverage
    

    View Slide

35. Recommendations
    

    View Slide

36. Testability
    • Named inputs
    
    • Scoped CSS
    
    • IDs where necessary
    

    View Slide

37. Cheats
    • Security bypasses
    
    • Test credit cards
    
    • Exposed internals
    

    View Slide

38. Cleanup
    • Identifiable test users
    
    • Filter from metrics
    
    • Delete old accounts
    

    View Slide

39. Roadmap
    1. Sign up / Log in
    
    2. Run after every deploy
    
    3. Low-coverge, High-impact
    
    4. Business critical
    
    5. Postmortems / Retrospectives
    

    View Slide

40. Questions?
    @jacebrowning
    zapier.com
    

    View Slide