Automated Testing: Getting it Right

Transcript

1. Automated Testing Stefan Kanev http://skanev.com/ @skanev Slovenia Ruby User Group 28 March 2018 Ljubljana Getting it right
2. Automated testing by developers & QA: Getting it right
3. clickbait!
4. It started trying to answer a question: 
 Should developers and QA share a suite? 
 It got a bit bigger and more dev-focused 
 But since it’s clickbait: 
 The answer might surprise you
5. Where I come from? 
 I wrote my first unit test on my first job, 13 years ago 
 The only reason I moved into Rails was good testing 
 One of my career goals has been to be super good at automated testing 
 I’ve read most of the books, watched most of the talks and read most of the blogs 
 I’ve written a s*!tload of tests
6. DISCLAIMER
7. I will be mixing facts, opinions and personal preferences freely 
 Take everything with a grain of salt and your mileage might vary (YMMV)
8. More Q&A than talk: please ask questions!
9. 1 2 3 4 Types of tests The Testing Pyramid (kinda) Integration vs. isolation Overall strategy
10. 1 Types of tests
11. System Under Test (SUT) each test (or type of test) has a specific SUT
 which conceptual parts (“layers”) of the system are tested 
 by extension, which ones aren’t tested 
 some things outside the SUT are replaced by doubles 
 it’s useful to be thinking about it
12. doesn’t need a double
 (upper layer) has to be mocked or stubbed
13. stubs vs. mocks replaces a collaborator for the test 
 canned responses 
 sometimes partial EmailSender.stub send_email: true replaces a collaborator for the test 
 asserts interactions 
 always whole object user = double name: 'John' expect(user).to receive(:update). with(name: 'Джон') * this is the Martin Fowler/xUnit Test Patterns terminology. it’s in no way universal (even in RSpec)
14. Model View Controller Service Routing JavaScript Database Mailer
15. Model View Controller Service Routing JavaScript Database Mailer unit test
16. Model View Controller Service Routing JavaScript Database Mailer unit test
17. Model View Controller Service Routing JavaScript Database Mailer system test
18. Model View Controller Service Routing JavaScript Database Mailer system test
19. Model View Controller Service Routing JavaScript Database Mailer system test
20. Model View Controller Service Routing JavaScript Database Mailer old-style controller spec
21. Model View Controller Service Routing JavaScript Database Mailer with nulldb * I am not recommending NullDB
22. My Terminology unit tests have relatively small or isolated SUTs examples: model specs, some service specs, classic controller specs integration tests include a few components examples: Rails’ system, functional & integration tests, some service specs end-to-end involve the browser (simulated or real) examples: functional specs, system tests, anything with capybara This is not how I organise the spec/ directory. It’s just how I think about tests in general.
23. 2 The Testing Pyramid (kinda)
24. E2E Integration Unit more of those less of those
25. Why the Testing Pyramid? Tests in integration are more expensive
 (maintenance, running time, writing time) 
 Developers run tests often
 (personally I run a few times a minute in some projects) 
 Tests in integration are more erratic
 (tend to fail randomly, especially when JavaScript is involved) Tests in integration are more brittle
 (tend to fail due to unrelated changes) Overarching tests don’t provide defect localisation (a simple change can make a big % of your tests to fail)
26. Erratic tests are ones that fail occasionally in a non-deterministic way (they make you distrust your test failures) 
 Brittle tests are ones that fail due to unrelated changes in functionality (they make changes/refactoring unnecessarily harder) 
 Both should be avoided carefully
28. Cases • User registers successfully • Email blank • Email invalid • Email already taken • Spaces around email should be trimmed • First name is blank • Last name is blank • Phone number is empty • Phone number is invalid • … 2 end-to-end tests happy path unhappy path 9+ unit tests (assuming there is a Registration object)
29. QA will have a different take on values
30. Dev runs tests many times during development gets blocked by long running suites can mock and stub aggressively cares about regression, but will be pragmatic after a point tests should help do changes faster, not slower QA runs tests when the work is done wouldn’t mind waiting longer periods (saves manual work) can’t tweak system internals cares only about correctness and regression extra time is well spent if it catches regressions
31. It started trying to answer a question: Should developers and QA share a suite? 
 It got a bit bigger and more dev-focused 
 But since it’s clickbait: The answer might surprise you No, they shouldn’t
32. 3 Integration vs. isolation
33. When should we test in integration and when in isolation? 
 As much as we can in isolation*, but also some in integration. 
 * mocking is hard; often we have to be pragmatic
34. So why is mocking hard? Tests are sensitive to the interface they depend on 
 Whey you mock/test in isolation, the tests depend on the interface between the objects 
 Refactorings that change the interface get harder 
 You should avoid mocking things that don’t
35. You should avoid mocking interfaces that are not stable 
 You should avoid mocking interfaces that are not as simple as possible
37. In integration
38. In isolation
39. Nowhere this is more evident than
 old-style controller specs
40. Model View Controller Service Routing JavaScript Database Mailer old-style controller spec
41. Controversy alert!
 So should you write them? 
 It depends 
 The benefits are different – they are not about regressions 
 If you do, both the mocks and the tests should be super simple 
 No, seriously, SUPER SIMPLE
42. describe ExpenseReportsController do let(:report) { double } let(:receipt_batch_update) { double } let(:integration) { double :integration } let(:catalog) { double } before do log_in_as :not_bookkeeper controller.stub report: report report.stub :report_receipt current_user.stub allowed_to_edit_receipt?: true viewed_account.stub :effortless_report_rule, get_integration: integration permit.stub expense_reports_access?: true request.env['HTTP_REFERER'] = root_url end describe 'GET index' do before do current_user.stub is_admin_for_account: true current_user.stub_chain(:expense_report_searches, :new, :filters) end it 'initializes expense report catalog with parameters' do current_user.stub_chain(:expense_report_searches, :new, filters: {total_amount: 100}) ExpenseReportCatalog. should_receive(:new). with(account: viewed_account, user: current_user, access_to_account: true, report_sort: nil, filters: {total_amount: 100}, pagination: { page: nil, per_page: nil, }) get :index end end end Bad example
43. describe AccountantInvitationsController do let(:request_invitation) { double :request_invitation, email: 'email@example.com' } before do log_in_as :not_bookkeeper ChangeAccountant::RequestInvitation.stub new: request_invitation end describe 'GET new' do it 'renders the new template' do get :new response.should render_template :new end end describe 'POST create' do it 'responds with reset_content on success' do request_invitation.stub execute: true get :create response.should have_http_status :reset_content end it 'renders new template on failure' do request_invitation.stub execute: false get :new response.should render_template :new end end end “Good” example
44. The more you mock, the more TDD becomes “Test-Driven Design” not “Development” 
 Debatable whether it’s the optimal way to design 
 However, test’s do provide some feedback on the design – simpler mocks and simpler tests usually mean nicer abstractions 
 They also provide some design pressure on the next person who changes the code
45. Do I write them? 
 Yes, I do. I even find them valuable. 
 It took me years to learn how to do them effectively. 
 It’s super hard to teach new people. 
 It’s easy to forget that they are not really about regression.
46. 4 Overall strategy
47. Let QA and Dev maintain separate suits with their own values 
 As a Dev, don’t abdicate testing only because you have a QA team 
 Tests have a cost, be careful how you spend it 
 Brittle and erratic tests only increase that cost 
 Be careful with mocks – they are useful, but only when done right (stubs are fine)
48. ? What I left out?
49. 1000+ pages, you only need the first ~200
50. Test-Driven Development It’s nuanced and it depends a lot on context 
 RSpec? Doesn’t really matter, not a big deal 
 Cucumber Is this still a thing? I suggest you don’t go there 
 100% Test Coverage It’s cheaply achievable in new things if you’re very good in TDD Anything else? Well, ask a question!
52. ?