Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automated Testing: Getting it Right

Automated Testing: Getting it Right

Stefan Kanev

March 28, 2018

More Decks by Stefan Kanev

Other Decks in Programming


  1. It started trying to answer a question: 
 Should developers

    and QA share a suite? 
 It got a bit bigger and more dev-focused 
 But since it’s clickbait: 
 The answer might surprise you
  2. Where I come from? 
 I wrote my first unit

    test on my first job, 13 years ago 
 The only reason I moved into Rails was good testing 
 One of my career goals has been to be super good at automated testing 
 I’ve read most of the books, watched most of the talks and read most of the blogs 
 I’ve written a s*!tload of tests
  3. I will be mixing facts, opinions and personal preferences freely

 Take everything with a grain of salt and your mileage might vary (YMMV)
  4. 1 2 3 4 Types of tests The Testing Pyramid

    (kinda) Integration vs. isolation Overall strategy
  5. System Under Test (SUT) each test (or type of test)

    has a specific SUT
 which conceptual parts (“layers”) of the system are tested 
 by extension, which ones aren’t tested 
 some things outside the SUT are replaced by doubles 
 it’s useful to be thinking about it
  6. stubs vs. mocks replaces a collaborator for the test 

    canned responses 
 sometimes partial EmailSender.stub send_email: true replaces a collaborator for the test 
 asserts interactions 
 always whole object user = double name: 'John' expect(user).to receive(:update). with(name: 'Джон') * this is the Martin Fowler/xUnit Test Patterns terminology. it’s in no way universal (even in RSpec)
  7. My Terminology unit tests have relatively small or isolated SUTs

    examples: model specs, some service specs, classic controller specs integration tests include a few components examples: Rails’ system, functional & integration tests, some service specs end-to-end involve the browser (simulated or real) examples: functional specs, system tests, anything with capybara This is not how I organise the spec/ directory. It’s just how I think about tests in general.
  8. Why the Testing Pyramid? Tests in integration are more expensive

    (maintenance, running time, writing time) 
 Developers run tests often
 (personally I run a few times a minute in some projects) 
 Tests in integration are more erratic
 (tend to fail randomly, especially when JavaScript is involved) Tests in integration are more brittle
 (tend to fail due to unrelated changes) Overarching tests don’t provide defect localisation (a simple change can make a big % of your tests to fail)
  9. Erratic tests are ones that fail occasionally in a non-deterministic

    way (they make you distrust your test failures) 
 Brittle tests are ones that fail due to unrelated changes in functionality (they make changes/refactoring unnecessarily harder) 
 Both should be avoided carefully
  10. Cases • User registers successfully • Email blank • Email

    invalid • Email already taken • Spaces around email should be trimmed • First name is blank • Last name is blank • Phone number is empty • Phone number is invalid • … 2 end-to-end tests happy path unhappy path 9+ unit tests (assuming there is a Registration object)
  11. Dev runs tests many times during development gets blocked by

    long running suites can mock and stub aggressively cares about regression, but will be pragmatic after a point tests should help do changes faster, not slower QA runs tests when the work is done wouldn’t mind waiting longer periods (saves manual work) can’t tweak system internals cares only about correctness and regression extra time is well spent if it catches regressions
  12. It started trying to answer a question: Should developers and

    QA share a suite? 
 It got a bit bigger and more dev-focused 
 But since it’s clickbait: The answer might surprise you No, they shouldn’t
  13. When should we test in integration and when in isolation?

 As much as we can in isolation*, but also some in integration. 
 * mocking is hard; often we have to be pragmatic
  14. So why is mocking hard? Tests are sensitive to the

    interface they depend on 
 Whey you mock/test in isolation, the tests depend on the interface between the objects 
 Refactorings that change the interface get harder 
 You should avoid mocking things that don’t
  15. You should avoid mocking interfaces that are not stable 

    You should avoid mocking interfaces that are not as simple as possible
  16. Controversy alert!
 So should you write them? 
 It depends

 The benefits are different – they are not about regressions 
 If you do, both the mocks and the tests should be super simple 
 No, seriously, SUPER SIMPLE
  17. describe ExpenseReportsController do let(:report) { double } let(:receipt_batch_update) { double

    } let(:integration) { double :integration } let(:catalog) { double } before do log_in_as :not_bookkeeper controller.stub report: report report.stub :report_receipt current_user.stub allowed_to_edit_receipt?: true viewed_account.stub :effortless_report_rule, get_integration: integration permit.stub expense_reports_access?: true request.env['HTTP_REFERER'] = root_url end describe 'GET index' do before do current_user.stub is_admin_for_account: true current_user.stub_chain(:expense_report_searches, :new, :filters) end it 'initializes expense report catalog with parameters' do current_user.stub_chain(:expense_report_searches, :new, filters: {total_amount: 100}) ExpenseReportCatalog. should_receive(:new). with(account: viewed_account, user: current_user, access_to_account: true, report_sort: nil, filters: {total_amount: 100}, pagination: { page: nil, per_page: nil, }) get :index end end end Bad example
  18. describe AccountantInvitationsController do let(:request_invitation) { double :request_invitation, email: '[email protected]' }

    before do log_in_as :not_bookkeeper ChangeAccountant::RequestInvitation.stub new: request_invitation end describe 'GET new' do it 'renders the new template' do get :new response.should render_template :new end end describe 'POST create' do it 'responds with reset_content on success' do request_invitation.stub execute: true get :create response.should have_http_status :reset_content end it 'renders new template on failure' do request_invitation.stub execute: false get :new response.should render_template :new end end end “Good” example
  19. The more you mock, the more TDD becomes “Test-Driven Design”

    not “Development” 
 Debatable whether it’s the optimal way to design 
 However, test’s do provide some feedback on the design – simpler mocks and simpler tests usually mean nicer abstractions 
 They also provide some design pressure on the next person who changes the code
  20. Do I write them? 
 Yes, I do. I even

    find them valuable. 
 It took me years to learn how to do them effectively. 
 It’s super hard to teach new people. 
 It’s easy to forget that they are not really about regression.
  21. Let QA and Dev maintain separate suits with their own

 As a Dev, don’t abdicate testing only because you have a QA team 
 Tests have a cost, be careful how you spend it 
 Brittle and erratic tests only increase that cost 
 Be careful with mocks – they are useful, but only when done right (stubs are fine)
  22. Test-Driven Development It’s nuanced and it depends a lot on

 RSpec? Doesn’t really matter, not a big deal 
 Cucumber Is this still a thing? I suggest you don’t go there 
 100% Test Coverage It’s cheaply achievable in new things if you’re very good in TDD Anything else? Well, ask a question!
  23. ?