Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automated Testing: Getting it Right

Automated Testing: Getting it Right

Stefan Kanev

March 28, 2018
Tweet

More Decks by Stefan Kanev

Other Decks in Programming

Transcript

  1. Automated Testing
    Stefan Kanev
    http://skanev.com/
    @skanev
    Slovenia Ruby User Group
    28 March 2018
    Ljubljana
    Getting it right

    View full-size slide

  2. Automated testing
    by developers & QA:
    Getting it right

    View full-size slide

  3. It started trying to answer a question:

    Should developers and QA share a suite?

    It got a bit bigger and more dev-focused

    But since it’s clickbait:

    The answer might surprise you

    View full-size slide

  4. Where I come from?

    I wrote my first unit test on my first job, 13 years
    ago

    The only reason I moved into Rails was good testing

    One of my career goals has been to be super good
    at automated testing

    I’ve read most of the books, watched most of the
    talks and read most of the blogs

    I’ve written a s*!tload of tests

    View full-size slide

  5. I will be mixing facts, opinions and
    personal preferences freely

    Take everything with a grain of salt and
    your mileage might vary (YMMV)

    View full-size slide

  6. More Q&A than talk: please ask questions!

    View full-size slide

  7. 1
    2
    3
    4
    Types of tests
    The Testing Pyramid (kinda)
    Integration vs. isolation
    Overall strategy

    View full-size slide

  8. 1
    Types of tests

    View full-size slide

  9. System Under Test (SUT)
    each test (or type of test) has a specific SUT

    which conceptual parts (“layers”) of the system are tested

    by extension, which ones aren’t tested

    some things outside the SUT are replaced by doubles

    it’s useful to be thinking about it

    View full-size slide

  10. doesn’t need a
    double

    (upper layer)
    has to be mocked
    or stubbed

    View full-size slide

  11. stubs vs. mocks
    replaces a collaborator for
    the test

    canned responses

    sometimes partial
    EmailSender.stub send_email: true
    replaces a collaborator for
    the test

    asserts interactions

    always whole object
    user = double name: 'John'
    expect(user).to receive(:update).
    with(name: 'Джон')
    * this is the Martin Fowler/xUnit Test Patterns terminology. it’s in no way universal (even in RSpec)

    View full-size slide

  12. Model
    View
    Controller
    Service
    Routing
    JavaScript
    Database
    Mailer

    View full-size slide

  13. Model
    View
    Controller
    Service
    Routing
    JavaScript
    Database
    Mailer
    unit test

    View full-size slide

  14. Model
    View
    Controller
    Service
    Routing
    JavaScript
    Database
    Mailer
    unit test

    View full-size slide

  15. Model
    View
    Controller
    Service
    Routing
    JavaScript
    Database
    Mailer
    system test

    View full-size slide

  16. Model
    View
    Controller
    Service
    Routing
    JavaScript
    Database
    Mailer
    system test

    View full-size slide

  17. Model
    View
    Controller
    Service
    Routing
    JavaScript
    Database
    Mailer
    system test

    View full-size slide

  18. Model
    View
    Controller
    Service
    Routing
    JavaScript
    Database
    Mailer
    old-style controller spec

    View full-size slide

  19. Model
    View
    Controller
    Service
    Routing
    JavaScript
    Database
    Mailer
    with nulldb
    * I am not recommending NullDB

    View full-size slide

  20. My Terminology
    unit tests have relatively small or isolated SUTs
    examples: model specs, some service specs, classic controller specs
    integration tests include a few components
    examples: Rails’ system, functional & integration tests, some service specs
    end-to-end involve the browser (simulated or real)
    examples: functional specs, system tests, anything with capybara
    This is not how I organise the spec/ directory. It’s just how I think
    about tests in general.

    View full-size slide

  21. 2
    The Testing Pyramid (kinda)

    View full-size slide

  22. E2E
    Integration
    Unit
    more of those
    less of those

    View full-size slide

  23. Why the Testing Pyramid?
    Tests in integration are more expensive

    (maintenance, running time, writing time)

    Developers run tests often

    (personally I run a few times a minute in some projects)

    Tests in integration are more erratic

    (tend to fail randomly, especially when JavaScript is involved)
    Tests in integration are more brittle

    (tend to fail due to unrelated changes)
    Overarching tests don’t provide defect localisation
    (a simple change can make a big % of your tests to fail)

    View full-size slide

  24. Erratic tests are ones that fail occasionally in
    a non-deterministic way
    (they make you distrust your test failures)

    Brittle tests are ones that fail due to
    unrelated changes in functionality
    (they make changes/refactoring unnecessarily harder)

    Both should be avoided carefully

    View full-size slide

  25. Cases
    • User registers successfully
    • Email blank
    • Email invalid
    • Email already taken
    • Spaces around email should be trimmed
    • First name is blank
    • Last name is blank
    • Phone number is empty
    • Phone number is invalid
    • …
    2 end-to-end tests
    happy path
    unhappy path
    9+ unit tests
    (assuming there is a Registration object)

    View full-size slide

  26. QA will have a different take on
    values

    View full-size slide

  27. Dev
    runs tests many times during
    development
    gets blocked by long running
    suites
    can mock and stub aggressively
    cares about regression, but will
    be pragmatic after a point
    tests should help do changes
    faster, not slower
    QA
    runs tests when the work is
    done
    wouldn’t mind waiting longer
    periods (saves manual work)
    can’t tweak system internals
    cares only about correctness
    and regression
    extra time is well spent if it
    catches regressions

    View full-size slide

  28. It started trying to answer a question:
    Should developers and QA share a suite?

    It got a bit bigger and more dev-focused

    But since it’s clickbait:
    The answer might surprise you
    No, they shouldn’t

    View full-size slide

  29. 3
    Integration vs. isolation

    View full-size slide

  30. When should we test in integration
    and when in isolation?

    As much as we can in isolation*, but
    also some in integration.

    * mocking is hard; often we have to be
    pragmatic

    View full-size slide

  31. So why is mocking hard?
    Tests are sensitive to the interface they depend on

    Whey you mock/test in isolation, the tests depend
    on the interface between the objects

    Refactorings that change the interface get harder

    You should avoid mocking things that don’t

    View full-size slide

  32. You should avoid mocking interfaces that
    are not stable

    You should avoid mocking interfaces that
    are not as simple as possible

    View full-size slide

  33. In integration

    View full-size slide

  34. In isolation

    View full-size slide

  35. Nowhere this is more evident than

    old-style controller specs

    View full-size slide

  36. Model
    View
    Controller
    Service
    Routing
    JavaScript
    Database
    Mailer
    old-style controller spec

    View full-size slide

  37. Controversy alert!

    So should you write them?

    It depends

    The benefits are different – they are not
    about regressions

    If you do, both the mocks and the tests
    should be super simple

    No, seriously, SUPER SIMPLE

    View full-size slide

  38. describe ExpenseReportsController do
    let(:report) { double }
    let(:receipt_batch_update) { double }
    let(:integration) { double :integration }
    let(:catalog) { double }
    before do
    log_in_as :not_bookkeeper
    controller.stub report: report
    report.stub :report_receipt
    current_user.stub allowed_to_edit_receipt?: true
    viewed_account.stub :effortless_report_rule, get_integration: integration
    permit.stub expense_reports_access?: true
    request.env['HTTP_REFERER'] = root_url
    end
    describe 'GET index' do
    before do
    current_user.stub is_admin_for_account: true
    current_user.stub_chain(:expense_report_searches, :new, :filters)
    end
    it 'initializes expense report catalog with parameters' do
    current_user.stub_chain(:expense_report_searches, :new, filters: {total_amount: 100})
    ExpenseReportCatalog.
    should_receive(:new).
    with(account: viewed_account,
    user: current_user,
    access_to_account: true,
    report_sort: nil,
    filters: {total_amount: 100},
    pagination: {
    page: nil,
    per_page: nil,
    })
    get :index
    end
    end
    end
    Bad example

    View full-size slide

  39. describe AccountantInvitationsController do
    let(:request_invitation) { double :request_invitation, email: '[email protected]' }
    before do
    log_in_as :not_bookkeeper
    ChangeAccountant::RequestInvitation.stub new: request_invitation
    end
    describe 'GET new' do
    it 'renders the new template' do
    get :new
    response.should render_template :new
    end
    end
    describe 'POST create' do
    it 'responds with reset_content on success' do
    request_invitation.stub execute: true
    get :create
    response.should have_http_status :reset_content
    end
    it 'renders new template on failure' do
    request_invitation.stub execute: false
    get :new
    response.should render_template :new
    end
    end
    end
    “Good” example

    View full-size slide

  40. The more you mock, the more TDD becomes
    “Test-Driven Design” not “Development”

    Debatable whether it’s the optimal way to design

    However, test’s do provide some feedback on the
    design – simpler mocks and simpler tests usually
    mean nicer abstractions

    They also provide some design pressure on the
    next person who changes the code

    View full-size slide

  41. Do I write them?

    Yes, I do. I even find them valuable.

    It took me years to learn how to do them
    effectively.

    It’s super hard to teach new people.

    It’s easy to forget that they are not really
    about regression.

    View full-size slide

  42. 4
    Overall strategy

    View full-size slide

  43. Let QA and Dev maintain separate suits with their
    own values

    As a Dev, don’t abdicate testing only because you
    have a QA team

    Tests have a cost, be careful how you spend it

    Brittle and erratic tests only increase that cost

    Be careful with mocks – they are useful, but only
    when done right (stubs are fine)

    View full-size slide

  44. ?
    What I left out?

    View full-size slide

  45. 1000+ pages, you only need the first ~200

    View full-size slide

  46. Test-Driven Development
    It’s nuanced and it depends a lot on context

    RSpec?
    Doesn’t really matter, not a big deal

    Cucumber
    Is this still a thing? I suggest you don’t go there

    100% Test Coverage
    It’s cheaply achievable in new things if you’re very good in TDD
    Anything else?
    Well, ask a question!

    View full-size slide