Jay Goel - Better Integration Testing with Cucumber

One of the hardest questions to answer is "does my program help the user accomplish their goals?" Whether that person is using our website or a programmer using our library, this talk will describe how to write automated tests which map to tasks our users are trying to accomplish. We will demonstrate specific Python testing libraries and evaluate the pros and cons of this approach to testing.

https://us.pycon.org/2016/schedule/presentation/2181/

Transcript

1. Better Integration Testing
   with Cucumber
   Jay Goel
   @poundifdef
   Rent the Runway
   

   View Slide

2. Motivation
   

   View Slide

3. We want to accomplish three things with
   Behavior-Driven Development.
   

   View Slide

4. Increase quality of integration tests
   

   View Slide

5. Reduce regressions
   

   View Slide

6. Focus on users’ behavior rather than
   code’s behavior
   

   View Slide

7. So how does this work?
   

   View Slide

8. In any test, we have...
   1. Preconditions
   2. The thing we’re testing
   3. Verification
   

   View Slide

9. class TestStringMethods(unittest.TestCase):
   def setUp(self):
   # Precondition
   self.s = 'hello world'
   def test_split():
   # Thing we’re testing
   tokens = self.s.split(' ')
   # Verify
   assertEqual(tokens, ['hello', 'world'])
   

   View Slide

10. In unit testing...
    1. Precondition
    2. The thing we’re testing
    3. Verify
    1. setUp()
    2. test_method()
    3. assert()
    

    View Slide

11. Cucumber is a BDD framework. It has its
    own domain-specific language.
    

    View Slide

12. In Cucumber...
    1. Precondition
    2. The thing we’re testing
    3. Verify
    1. Given…
    2. When…
    3. Then...
    

    View Slide

13. Given I have a string 'hello world'
    When I split it based on the ' ' character
    Then I expect 2 tokens
    And I expect these tokens to be present:
    | token |
    | hello |
    | world |
    

    View Slide

14. But where is the actual test? (We're
    getting there!)
    

    View Slide

15. We get better descriptions of test cases
    

    View Slide

16. We're already doing this in code!
    ● test_route_decorator_custom_endpoint_with_dots (Flask)
    ● test_request_cookie_overrides_session_cookie (Requests)
    

    View Slide

17. It forces non-technical users (eg product
    managers) to better define the behavior
    of software
    

    View Slide

18. More examples
    

    View Slide

19. An e-commerce website
    Given I am on the home page
    When I view products
    Then I expect to see a list of items
    And I expect to see pictures of the items
    

    View Slide

20. An authentication system
    Given I am a logged in user
    When I view my account info
    Then I should see my account number
    Given I am not a logged in user
    When I view my account info
    Then I should be shown an "unauthorized" error
    

    View Slide

21. The 'requests' library
    Given I make a request to example.com/API.json
    When I serialize the response to JSON
    Then I expect a valid Python dictionary
    

    View Slide

22. Thus we have a framework for asking
    "What is our user actually trying to do?"
    

    View Slide

23. Code
    

    View Slide

24. Python libraries
    ● Behave (http://pythonhosted.org/behave/)
    ● Lettuce (http://lettuce.it/)
    ● I will use Behave as the example here
    ○ `pip install behave`
    

    View Slide

25. strings.feature
    Feature: Test string features
    In order to play with Behave
    As beginners
    We'll implement tests for Python string features
    Scenario: Splitting a string into tokens
    Given I have a string 'hello world'
    When I split it based on the ' ' character
    Then I expect 2 tokens
    And I expect these tokens to be present:
    | token |
    | hello |
    | world |
    

    View Slide

26. $ behave
    Feature: Test string features # strings.feature:1
    In order to play with Behave
    As beginners
    We'll implement tests for Python string features
    Scenario: Splitting a string into tokens # strings.feature:7
    Given I have a string 'hello world' # None
    When I split it based on the ' ' character # None
    Then I expect two tokens # None
    And I expect these tokens to be present # None
    | token |
    | hello |
    | world |
    Failing scenarios:
    strings.feature:7 Splitting a string into tokens
    0 features passed, 1 failed, 0 skipped
    0 scenarios passed, 1 failed, 0 skipped
    0 steps passed, 0 failed, 0 skipped, 4 undefined
    Took 0m0.000s
    

    View Slide

27. You can implement step definitions for undefined steps with these snippets:
    @given(u'I have a string \'hello world\'')
    def step_impl(context):
    raise NotImplementedError(u'STEP: Given I have a string \'hello world\'')
    @when(u'I split it based on the \' \' character')
    def step_impl(context):
    raise NotImplementedError(u'STEP: When I split it based on the \' \' character')
    @then(u'I expect two tokens')
    def step_impl(context):
    raise NotImplementedError(u'STEP: Then I expect two tokens')
    @then(u'I expect these tokens to be present')
    def step_impl(context):
    raise NotImplementedError(u'STEP: Then I expect these tokens to be present')
    

    View Slide

28. Making tests pass
    

    View Slide

29. steps/strings.py
    from behave import *
    @given(u'I have a string \'{s}\'')
    def step_impl(context, s):
    context.s = s
    @when(u'I split it based on the \'{separator}\' character')
    def step_impl(context, separator):
    context.tokens = context.s.split(' ')
    

    View Slide

30. steps/strings.py (cont)
    @then(u'I expect {num} tokens')
    def step_impl(context, num):
    assert len(context.tokens) == int(num)
    @then(u'I expect these tokens to be present')
    def step_impl(context):
    for i, row in enumerate(context.table):
    assert context.tokens[i] == row['token']
    

    View Slide

31. $ behave
    Feature: Test string features # strings.feature:1
    In order to play with Behave
    As beginners
    We'll implement tests for Python string features
    Scenario: Splitting a string into tokens # strings.feature:7
    Given I have a string 'hello world' # steps/strings.py:3 0.000s
    When I split it based on the ' ' character # steps/strings.py:7 0.000s
    Then I expect 2 tokens # steps/strings.py:11 0.000s
    And I expect these tokens to be present # steps/strings.py:15 0.000s
    | token |
    | hello |
    | world |
    1 feature passed, 0 failed, 0 skipped
    1 scenario passed, 0 failed, 0 skipped
    4 steps passed, 0 failed, 0 skipped, 0 undefined
    Took 0m0.000s
    

    View Slide

32. What if tests fail?
    

    View Slide

33. $ behave
    Feature: Test string features # strings.feature:1
    In order to play with Behave
    As beginners
    We'll implement tests for Python string features
    Scenario: Splitting a string into tokens # strings.feature:7
    Given I have a string 'hello world' # steps/strings.py:3 0.000s
    When I split it based on the ' ' character # steps/strings.py:7 0.000s
    Then I expect 1 tokens # steps/strings.py:11 0.000s
    ... Traceback ...
    And I expect these tokens to be present # None
    | token |
    | hello |
    | world |
    Failing scenarios:
    strings.feature:7 Splitting a string into tokens
    0 features passed, 1 failed, 0 skipped
    0 scenarios passed, 1 failed, 0 skipped
    2 steps passed, 1 failed, 1 skipped, 0 undefined
    Took 0m0.000s
    

    View Slide

34. These steps are analogous to HTTP
    "routes"
    

    View Slide

35. Cucumber as routing
    Code
    @app.route('/user/')
    def show_user_profile(username):
    # ..fetch user from DB..
    # ..display on screen..
    Human-Readable Interface
    GET example.com/user/
    Code
    @when('I view my account info')
    def step_impl(context):
    # ..create selenium instance..
    # ..navigate to user page..
    Human-Readable Interface
    When I view my account info
    

    View Slide

36. Granularity
    

    View Slide

37. Bad - too granular. Looks like code.
    Given I am on the home page
    And I click 'login'
    When I give the 'username' box the focus
    And I enter 'poundifdef' in that box
    And I give the 'password' box the focus
    And I enter 'py7h0n' in that box
    And I click the submit button
    Then I expect a "200" response code
    And the page should contain my username
    

    View Slide

38. Good - Describes user behavior.
    Given I am on the home page
    When I log in as
    username 'poundifdef'
    password 'py7h0n'
    Then I should see my account information
    

    View Slide

39. Can non-technical people write tests?
    

    View Slide

40. I say not really.
    

    View Slide

41. It is useful to use similar language.
    Language, after all, shapes the way we
    think about things.
    

    View Slide

42. A real example from RTR!
    

    View Slide

43. Feature: Creating an account
    Background:
    Given I go to the homepage
    Scenario: Successfully creating an account
    Given I see a login form
    When I fill out the form correctly
    And click the 'join now' button
    And I close the invite friends modal
    Then I should be logged in
    

    View Slide

44. Given(/^I go to the homepage$/) do
    visit '/'
    wait_for_page_load
    expect(current_path).to eq('/')
    end
    Given(/^I see a login form$/) do
    unless page.has_css?('#auth-form')
    puts "Didn't see login form, clicking sign in."
    click_link('Sign in')
    step "I click 'Don't have an account? Join now.'"
    end
    end
    

    View Slide

45. View Slide

46. Key Takeaways
    ● Focus on users' needs and behavior, not the behavior of code
    ● "Given... When... Then" idiom
    ● Our cucumbers can be re-usable and human-understandable
    ● Akin to "routes" as a separation of concerns
    ● Be careful of being too granular
    ● Using language similar to users is helpful for meeting their needs
    

    View Slide

47. Questions? Thank you!
    Jay Goel
    @poundifdef
    Rent the Runway
    

    View Slide