Jay Goel - Better Integration Testing with Cucumber

One of the hardest questions to answer is "does my program help the user accomplish their goals?" Whether that person is using our website or a programmer using our library, this talk will describe how to write automated tests which map to tasks our users are trying to accomplish. We will demonstrate specific Python testing libraries and evaluate the pros and cons of this approach to testing.

https://us.pycon.org/2016/schedule/presentation/2181/

Transcript

1. Better Integration Testing with Cucumber Jay Goel @poundifdef Rent the Runway
2. Motivation
3. We want to accomplish three things with Behavior-Driven Development.
4. Increase quality of integration tests
5. Reduce regressions
6. Focus on users’ behavior rather than code’s behavior
7. So how does this work?
8. In any test, we have... 1. Preconditions 2. The thing we’re testing 3. Verification
9. class TestStringMethods(unittest.TestCase): def setUp(self): # Precondition self.s = 'hello world' def test_split(): # Thing we’re testing tokens = self.s.split(' ') # Verify assertEqual(tokens, ['hello', 'world'])
10. In unit testing... 1. Precondition 2. The thing we’re testing 3. Verify 1. setUp() 2. test_method() 3. assert()
11. Cucumber is a BDD framework. It has its own domain-specific language.
12. In Cucumber... 1. Precondition 2. The thing we’re testing 3. Verify 1. Given… 2. When… 3. Then...
13. Given I have a string 'hello world' When I split it based on the ' ' character Then I expect 2 tokens And I expect these tokens to be present: | token | | hello | | world |
14. But where is the actual test? (We're getting there!)
15. We get better descriptions of test cases
16. We're already doing this in code! • test_route_decorator_custom_endpoint_with_dots (Flask) • test_request_cookie_overrides_session_cookie (Requests)
17. It forces non-technical users (eg product managers) to better define the behavior of software
18. More examples
19. An e-commerce website Given I am on the home page When I view products Then I expect to see a list of items And I expect to see pictures of the items
20. An authentication system Given I am a logged in user When I view my account info Then I should see my account number Given I am not a logged in user When I view my account info Then I should be shown an "unauthorized" error
21. The 'requests' library Given I make a request to example.com/API.json When I serialize the response to JSON Then I expect a valid Python dictionary
22. Thus we have a framework for asking "What is our user actually trying to do?"
23. Code
24. Python libraries • Behave (http://pythonhosted.org/behave/) • Lettuce (http://lettuce.it/) • I will use Behave as the example here ◦ `pip install behave`
25. strings.feature Feature: Test string features In order to play with Behave As beginners We'll implement tests for Python string features Scenario: Splitting a string into tokens Given I have a string 'hello world' When I split it based on the ' ' character Then I expect 2 tokens And I expect these tokens to be present: | token | | hello | | world |
26. $ behave Feature: Test string features # strings.feature:1 In order to play with Behave As beginners We'll implement tests for Python string features Scenario: Splitting a string into tokens # strings.feature:7 Given I have a string 'hello world' # None When I split it based on the ' ' character # None Then I expect two tokens # None And I expect these tokens to be present # None | token | | hello | | world | Failing scenarios: strings.feature:7 Splitting a string into tokens 0 features passed, 1 failed, 0 skipped 0 scenarios passed, 1 failed, 0 skipped 0 steps passed, 0 failed, 0 skipped, 4 undefined Took 0m0.000s
27. You can implement step definitions for undefined steps with these snippets: @given(u'I have a string \'hello world\'') def step_impl(context): raise NotImplementedError(u'STEP: Given I have a string \'hello world\'') @when(u'I split it based on the \' \' character') def step_impl(context): raise NotImplementedError(u'STEP: When I split it based on the \' \' character') @then(u'I expect two tokens') def step_impl(context): raise NotImplementedError(u'STEP: Then I expect two tokens') @then(u'I expect these tokens to be present') def step_impl(context): raise NotImplementedError(u'STEP: Then I expect these tokens to be present')
28. Making tests pass
29. steps/strings.py from behave import * @given(u'I have a string \'{s}\'') def step_impl(context, s): context.s = s @when(u'I split it based on the \'{separator}\' character') def step_impl(context, separator): context.tokens = context.s.split(' ')
30. steps/strings.py (cont) @then(u'I expect {num} tokens') def step_impl(context, num): assert len(context.tokens) == int(num) @then(u'I expect these tokens to be present') def step_impl(context): for i, row in enumerate(context.table): assert context.tokens[i] == row['token']
31. $ behave Feature: Test string features # strings.feature:1 In order to play with Behave As beginners We'll implement tests for Python string features Scenario: Splitting a string into tokens # strings.feature:7 Given I have a string 'hello world' # steps/strings.py:3 0.000s When I split it based on the ' ' character # steps/strings.py:7 0.000s Then I expect 2 tokens # steps/strings.py:11 0.000s And I expect these tokens to be present # steps/strings.py:15 0.000s | token | | hello | | world | 1 feature passed, 0 failed, 0 skipped 1 scenario passed, 0 failed, 0 skipped 4 steps passed, 0 failed, 0 skipped, 0 undefined Took 0m0.000s
32. What if tests fail?
33. $ behave Feature: Test string features # strings.feature:1 In order to play with Behave As beginners We'll implement tests for Python string features Scenario: Splitting a string into tokens # strings.feature:7 Given I have a string 'hello world' # steps/strings.py:3 0.000s When I split it based on the ' ' character # steps/strings.py:7 0.000s Then I expect 1 tokens # steps/strings.py:11 0.000s ... Traceback ... And I expect these tokens to be present # None | token | | hello | | world | Failing scenarios: strings.feature:7 Splitting a string into tokens 0 features passed, 1 failed, 0 skipped 0 scenarios passed, 1 failed, 0 skipped 2 steps passed, 1 failed, 1 skipped, 0 undefined Took 0m0.000s
34. These steps are analogous to HTTP "routes"
35. Cucumber as routing Code @app.route('/user/<username>') def show_user_profile(username): # ..fetch user from DB.. # ..display on screen.. Human-Readable Interface GET example.com/user/<username> Code @when('I view my account info') def step_impl(context): # ..create selenium instance.. # ..navigate to user page.. Human-Readable Interface When I view my account info
36. Granularity
37. Bad - too granular. Looks like code. Given I am on the home page And I click 'login' When I give the 'username' box the focus And I enter 'poundifdef' in that box And I give the 'password' box the focus And I enter 'py7h0n' in that box And I click the submit button Then I expect a "200" response code And the page should contain my username
38. Good - Describes user behavior. Given I am on the home page When I log in as username 'poundifdef' password 'py7h0n' Then I should see my account information
39. Can non-technical people write tests?
40. I say not really.
41. It is useful to use similar language. Language, after all, shapes the way we think about things.
42. A real example from RTR!
43. Feature: Creating an account Background: Given I go to the homepage Scenario: Successfully creating an account Given I see a login form When I fill out the form correctly And click the 'join now' button And I close the invite friends modal Then I should be logged in
44. Given(/^I go to the homepage$/) do visit '/' wait_for_page_load expect(current_path).to eq('/') end Given(/^I see a login form$/) do unless page.has_css?('#auth-form') puts "Didn't see login form, clicking sign in." click_link('Sign in') step "I click 'Don't have an account? Join now.'" end end
46. Key Takeaways • Focus on users' needs and behavior, not the behavior of code • "Given... When... Then" idiom • Our cucumbers can be re-usable and human-understandable • Akin to "routes" as a separation of concerns • Be careful of being too granular • Using language similar to users is helpful for meeting their needs
47. Questions? Thank you! Jay Goel @poundifdef Rent the Runway