Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automated Testing: Getting it Right

Automated Testing: Getting it Right

Stefan Kanev

March 28, 2018
Tweet

More Decks by Stefan Kanev

Other Decks in Programming

Transcript

  1. Automated Testing Stefan Kanev http://skanev.com/ @skanev Slovenia Ruby User Group

    28 March 2018 Ljubljana Getting it right
  2. Automated testing by developers & QA: Getting it right

  3. clickbait!

  4. It started trying to answer a question: 
 Should developers

    and QA share a suite? 
 It got a bit bigger and more dev-focused 
 But since it’s clickbait: 
 The answer might surprise you
  5. Where I come from? 
 I wrote my first unit

    test on my first job, 13 years ago 
 The only reason I moved into Rails was good testing 
 One of my career goals has been to be super good at automated testing 
 I’ve read most of the books, watched most of the talks and read most of the blogs 
 I’ve written a s*!tload of tests
  6. DISCLAIMER

  7. I will be mixing facts, opinions and personal preferences freely

    
 Take everything with a grain of salt and your mileage might vary (YMMV)
  8. More Q&A than talk: please ask questions!

  9. 1 2 3 4 Types of tests The Testing Pyramid

    (kinda) Integration vs. isolation Overall strategy
  10. 1 Types of tests

  11. System Under Test (SUT) each test (or type of test)

    has a specific SUT
 which conceptual parts (“layers”) of the system are tested 
 by extension, which ones aren’t tested 
 some things outside the SUT are replaced by doubles 
 it’s useful to be thinking about it
  12. doesn’t need a double
 (upper layer) has to be mocked

    or stubbed
  13. stubs vs. mocks replaces a collaborator for the test 


    canned responses 
 sometimes partial EmailSender.stub send_email: true replaces a collaborator for the test 
 asserts interactions 
 always whole object user = double name: 'John' expect(user).to receive(:update). with(name: 'Джон') * this is the Martin Fowler/xUnit Test Patterns terminology. it’s in no way universal (even in RSpec)
  14. Model View Controller Service Routing JavaScript Database Mailer

  15. Model View Controller Service Routing JavaScript Database Mailer unit test

  16. Model View Controller Service Routing JavaScript Database Mailer unit test

  17. Model View Controller Service Routing JavaScript Database Mailer system test

  18. Model View Controller Service Routing JavaScript Database Mailer system test

  19. Model View Controller Service Routing JavaScript Database Mailer system test

  20. Model View Controller Service Routing JavaScript Database Mailer old-style controller

    spec
  21. Model View Controller Service Routing JavaScript Database Mailer with nulldb

    * I am not recommending NullDB
  22. My Terminology unit tests have relatively small or isolated SUTs

    examples: model specs, some service specs, classic controller specs integration tests include a few components examples: Rails’ system, functional & integration tests, some service specs end-to-end involve the browser (simulated or real) examples: functional specs, system tests, anything with capybara This is not how I organise the spec/ directory. It’s just how I think about tests in general.
  23. 2 The Testing Pyramid (kinda)

  24. E2E Integration Unit more of those less of those

  25. Why the Testing Pyramid? Tests in integration are more expensive


    (maintenance, running time, writing time) 
 Developers run tests often
 (personally I run a few times a minute in some projects) 
 Tests in integration are more erratic
 (tend to fail randomly, especially when JavaScript is involved) Tests in integration are more brittle
 (tend to fail due to unrelated changes) Overarching tests don’t provide defect localisation (a simple change can make a big % of your tests to fail)
  26. Erratic tests are ones that fail occasionally in a non-deterministic

    way (they make you distrust your test failures) 
 Brittle tests are ones that fail due to unrelated changes in functionality (they make changes/refactoring unnecessarily harder) 
 Both should be avoided carefully
  27. None
  28. Cases • User registers successfully • Email blank • Email

    invalid • Email already taken • Spaces around email should be trimmed • First name is blank • Last name is blank • Phone number is empty • Phone number is invalid • … 2 end-to-end tests happy path unhappy path 9+ unit tests (assuming there is a Registration object)
  29. QA will have a different take on values

  30. Dev runs tests many times during development gets blocked by

    long running suites can mock and stub aggressively cares about regression, but will be pragmatic after a point tests should help do changes faster, not slower QA runs tests when the work is done wouldn’t mind waiting longer periods (saves manual work) can’t tweak system internals cares only about correctness and regression extra time is well spent if it catches regressions
  31. It started trying to answer a question: Should developers and

    QA share a suite? 
 It got a bit bigger and more dev-focused 
 But since it’s clickbait: The answer might surprise you No, they shouldn’t
  32. 3 Integration vs. isolation

  33. When should we test in integration and when in isolation?

    
 As much as we can in isolation*, but also some in integration. 
 * mocking is hard; often we have to be pragmatic
  34. So why is mocking hard? Tests are sensitive to the

    interface they depend on 
 Whey you mock/test in isolation, the tests depend on the interface between the objects 
 Refactorings that change the interface get harder 
 You should avoid mocking things that don’t
  35. You should avoid mocking interfaces that are not stable 


    You should avoid mocking interfaces that are not as simple as possible
  36. None
  37. In integration

  38. In isolation

  39. Nowhere this is more evident than
 old-style controller specs

  40. Model View Controller Service Routing JavaScript Database Mailer old-style controller

    spec
  41. Controversy alert!
 So should you write them? 
 It depends

    
 The benefits are different – they are not about regressions 
 If you do, both the mocks and the tests should be super simple 
 No, seriously, SUPER SIMPLE
  42. describe ExpenseReportsController do let(:report) { double } let(:receipt_batch_update) { double

    } let(:integration) { double :integration } let(:catalog) { double } before do log_in_as :not_bookkeeper controller.stub report: report report.stub :report_receipt current_user.stub allowed_to_edit_receipt?: true viewed_account.stub :effortless_report_rule, get_integration: integration permit.stub expense_reports_access?: true request.env['HTTP_REFERER'] = root_url end describe 'GET index' do before do current_user.stub is_admin_for_account: true current_user.stub_chain(:expense_report_searches, :new, :filters) end it 'initializes expense report catalog with parameters' do current_user.stub_chain(:expense_report_searches, :new, filters: {total_amount: 100}) ExpenseReportCatalog. should_receive(:new). with(account: viewed_account, user: current_user, access_to_account: true, report_sort: nil, filters: {total_amount: 100}, pagination: { page: nil, per_page: nil, }) get :index end end end Bad example
  43. describe AccountantInvitationsController do let(:request_invitation) { double :request_invitation, email: '[email protected]' }

    before do log_in_as :not_bookkeeper ChangeAccountant::RequestInvitation.stub new: request_invitation end describe 'GET new' do it 'renders the new template' do get :new response.should render_template :new end end describe 'POST create' do it 'responds with reset_content on success' do request_invitation.stub execute: true get :create response.should have_http_status :reset_content end it 'renders new template on failure' do request_invitation.stub execute: false get :new response.should render_template :new end end end “Good” example
  44. The more you mock, the more TDD becomes “Test-Driven Design”

    not “Development” 
 Debatable whether it’s the optimal way to design 
 However, test’s do provide some feedback on the design – simpler mocks and simpler tests usually mean nicer abstractions 
 They also provide some design pressure on the next person who changes the code
  45. Do I write them? 
 Yes, I do. I even

    find them valuable. 
 It took me years to learn how to do them effectively. 
 It’s super hard to teach new people. 
 It’s easy to forget that they are not really about regression.
  46. 4 Overall strategy

  47. Let QA and Dev maintain separate suits with their own

    values 
 As a Dev, don’t abdicate testing only because you have a QA team 
 Tests have a cost, be careful how you spend it 
 Brittle and erratic tests only increase that cost 
 Be careful with mocks – they are useful, but only when done right (stubs are fine)
  48. ? What I left out?

  49. 1000+ pages, you only need the first ~200

  50. Test-Driven Development It’s nuanced and it depends a lot on

    context 
 RSpec? Doesn’t really matter, not a big deal 
 Cucumber Is this still a thing? I suggest you don’t go there 
 100% Test Coverage It’s cheaply achievable in new things if you’re very good in TDD Anything else? Well, ask a question!
  51. None
  52. ?