Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Jay Goel - Better Integration Testing with Cucumber

Jay Goel - Better Integration Testing with Cucumber

One of the hardest questions to answer is "does my program help the user accomplish their goals?" Whether that person is using our website or a programmer using our library, this talk will describe how to write automated tests which map to tasks our users are trying to accomplish. We will demonstrate specific Python testing libraries and evaluate the pros and cons of this approach to testing.

https://us.pycon.org/2016/schedule/presentation/2181/

Eec9d25835717f1f1f12a354faf68d87?s=128

PyCon 2016

May 29, 2016
Tweet

More Decks by PyCon 2016

Other Decks in Programming

Transcript

  1. Better Integration Testing with Cucumber Jay Goel @poundifdef Rent the

    Runway
  2. Motivation

  3. We want to accomplish three things with Behavior-Driven Development.

  4. Increase quality of integration tests

  5. Reduce regressions

  6. Focus on users’ behavior rather than code’s behavior

  7. So how does this work?

  8. In any test, we have... 1. Preconditions 2. The thing

    we’re testing 3. Verification
  9. class TestStringMethods(unittest.TestCase): def setUp(self): # Precondition self.s = 'hello world'

    def test_split(): # Thing we’re testing tokens = self.s.split(' ') # Verify assertEqual(tokens, ['hello', 'world'])
  10. In unit testing... 1. Precondition 2. The thing we’re testing

    3. Verify 1. setUp() 2. test_method() 3. assert()
  11. Cucumber is a BDD framework. It has its own domain-specific

    language.
  12. In Cucumber... 1. Precondition 2. The thing we’re testing 3.

    Verify 1. Given… 2. When… 3. Then...
  13. Given I have a string 'hello world' When I split

    it based on the ' ' character Then I expect 2 tokens And I expect these tokens to be present: | token | | hello | | world |
  14. But where is the actual test? (We're getting there!)

  15. We get better descriptions of test cases

  16. We're already doing this in code! • test_route_decorator_custom_endpoint_with_dots (Flask) •

    test_request_cookie_overrides_session_cookie (Requests)
  17. It forces non-technical users (eg product managers) to better define

    the behavior of software
  18. More examples

  19. An e-commerce website Given I am on the home page

    When I view products Then I expect to see a list of items And I expect to see pictures of the items
  20. An authentication system Given I am a logged in user

    When I view my account info Then I should see my account number Given I am not a logged in user When I view my account info Then I should be shown an "unauthorized" error
  21. The 'requests' library Given I make a request to example.com/API.json

    When I serialize the response to JSON Then I expect a valid Python dictionary
  22. Thus we have a framework for asking "What is our

    user actually trying to do?"
  23. Code

  24. Python libraries • Behave (http://pythonhosted.org/behave/) • Lettuce (http://lettuce.it/) • I

    will use Behave as the example here ◦ `pip install behave`
  25. strings.feature Feature: Test string features In order to play with

    Behave As beginners We'll implement tests for Python string features Scenario: Splitting a string into tokens Given I have a string 'hello world' When I split it based on the ' ' character Then I expect 2 tokens And I expect these tokens to be present: | token | | hello | | world |
  26. $ behave Feature: Test string features # strings.feature:1 In order

    to play with Behave As beginners We'll implement tests for Python string features Scenario: Splitting a string into tokens # strings.feature:7 Given I have a string 'hello world' # None When I split it based on the ' ' character # None Then I expect two tokens # None And I expect these tokens to be present # None | token | | hello | | world | Failing scenarios: strings.feature:7 Splitting a string into tokens 0 features passed, 1 failed, 0 skipped 0 scenarios passed, 1 failed, 0 skipped 0 steps passed, 0 failed, 0 skipped, 4 undefined Took 0m0.000s
  27. You can implement step definitions for undefined steps with these

    snippets: @given(u'I have a string \'hello world\'') def step_impl(context): raise NotImplementedError(u'STEP: Given I have a string \'hello world\'') @when(u'I split it based on the \' \' character') def step_impl(context): raise NotImplementedError(u'STEP: When I split it based on the \' \' character') @then(u'I expect two tokens') def step_impl(context): raise NotImplementedError(u'STEP: Then I expect two tokens') @then(u'I expect these tokens to be present') def step_impl(context): raise NotImplementedError(u'STEP: Then I expect these tokens to be present')
  28. Making tests pass

  29. steps/strings.py from behave import * @given(u'I have a string \'{s}\'')

    def step_impl(context, s): context.s = s @when(u'I split it based on the \'{separator}\' character') def step_impl(context, separator): context.tokens = context.s.split(' ')
  30. steps/strings.py (cont) @then(u'I expect {num} tokens') def step_impl(context, num): assert

    len(context.tokens) == int(num) @then(u'I expect these tokens to be present') def step_impl(context): for i, row in enumerate(context.table): assert context.tokens[i] == row['token']
  31. $ behave Feature: Test string features # strings.feature:1 In order

    to play with Behave As beginners We'll implement tests for Python string features Scenario: Splitting a string into tokens # strings.feature:7 Given I have a string 'hello world' # steps/strings.py:3 0.000s When I split it based on the ' ' character # steps/strings.py:7 0.000s Then I expect 2 tokens # steps/strings.py:11 0.000s And I expect these tokens to be present # steps/strings.py:15 0.000s | token | | hello | | world | 1 feature passed, 0 failed, 0 skipped 1 scenario passed, 0 failed, 0 skipped 4 steps passed, 0 failed, 0 skipped, 0 undefined Took 0m0.000s
  32. What if tests fail?

  33. $ behave Feature: Test string features # strings.feature:1 In order

    to play with Behave As beginners We'll implement tests for Python string features Scenario: Splitting a string into tokens # strings.feature:7 Given I have a string 'hello world' # steps/strings.py:3 0.000s When I split it based on the ' ' character # steps/strings.py:7 0.000s Then I expect 1 tokens # steps/strings.py:11 0.000s ... Traceback ... And I expect these tokens to be present # None | token | | hello | | world | Failing scenarios: strings.feature:7 Splitting a string into tokens 0 features passed, 1 failed, 0 skipped 0 scenarios passed, 1 failed, 0 skipped 2 steps passed, 1 failed, 1 skipped, 0 undefined Took 0m0.000s
  34. These steps are analogous to HTTP "routes"

  35. Cucumber as routing Code @app.route('/user/<username>') def show_user_profile(username): # ..fetch user

    from DB.. # ..display on screen.. Human-Readable Interface GET example.com/user/<username> Code @when('I view my account info') def step_impl(context): # ..create selenium instance.. # ..navigate to user page.. Human-Readable Interface When I view my account info
  36. Granularity

  37. Bad - too granular. Looks like code. Given I am

    on the home page And I click 'login' When I give the 'username' box the focus And I enter 'poundifdef' in that box And I give the 'password' box the focus And I enter 'py7h0n' in that box And I click the submit button Then I expect a "200" response code And the page should contain my username
  38. Good - Describes user behavior. Given I am on the

    home page When I log in as username 'poundifdef' password 'py7h0n' Then I should see my account information
  39. Can non-technical people write tests?

  40. I say not really.

  41. It is useful to use similar language. Language, after all,

    shapes the way we think about things.
  42. A real example from RTR!

  43. Feature: Creating an account Background: Given I go to the

    homepage Scenario: Successfully creating an account Given I see a login form When I fill out the form correctly And click the 'join now' button And I close the invite friends modal Then I should be logged in
  44. Given(/^I go to the homepage$/) do visit '/' wait_for_page_load expect(current_path).to

    eq('/') end Given(/^I see a login form$/) do unless page.has_css?('#auth-form') puts "Didn't see login form, clicking sign in." click_link('Sign in') step "I click 'Don't have an account? Join now.'" end end
  45. None
  46. Key Takeaways • Focus on users' needs and behavior, not

    the behavior of code • "Given... When... Then" idiom • Our cucumbers can be re-usable and human-understandable • Akin to "routes" as a separation of concerns • Be careful of being too granular • Using language similar to users is helpful for meeting their needs
  47. Questions? Thank you! Jay Goel @poundifdef Rent the Runway