RRUG - Efficient Testing

I tried to summarise years of working with TestProf and help those who have yet to start. The presentation mostly touches the let_it_be and before_all helpers and problems developers could face when start using that

Transcript

1. Ef f icient Testing Makar Ermokhin
2. Dzień dobry! • Work in Chattermill since 2023 • Live in Warsaw since 2021 • Work with Ruby since 2015 • Love tests since 2015 Wake up, samurai, we have some tests to optimise
3. Why speed up tests?
4. Why speed up tests? • Feedback loop • Developer experience • Reduce CI costs 4
5. 5 https://youtu.be/WNluKNQ1OFU
6. 6 https://youtu.be/WNluKNQ1OFU
7. 7 https://youtu.be/VD51AkG8EZw
8. 8 https://youtu.be/VD51AkG8EZw
9. What drives tests slow?
10. Pro f iling: Test Prof https://test-prof.evilmartians.io/ 10
11. Pro f iling: Test Prof https://test-prof.evilmartians.io/ 11
12. Classic RSpec 12
17. RSpec encourages developers to use factories often
18. What can we do?
19. Build, don’t create
20. build_stubbed 20
21. build_stubbed • Doesn’t save the records 👍 • Doesn’t allow records to access database 👍 • Doesn’t help with tests like requests 😕 21
22. Reduce, reuse, recycle
23. Reduce, reuse, recycle
24. before(:all) 24
25. before(:all) 25
26. before(:all) 26 ✅ ❌
27. Even if the database changes are reverted, the objects might be spoiled
28. 28 before(:all) + f ind
29. 29 before(:all) + f ind
30. 30 before(:all) + f ind ✅ ✅ ❌
31. 31 before(:all) + f ind + after(:all)
32. 32 before(:all) + f ind + after(:all)
33. 33 before(:all) + f ind + after(:all) ✅ ✅ ❌
34. 34 before(:all) + f ind + after(:all) + mental overhead
35. before(:all) • Helps with amount of records created 👍 • Provokes state leaks between f iles 😕 • Destroying records is also slow 😕 • Advised against in RSpec docs 😕 • Has a dedicated RuboCop https://docs.rubocop.org/rubocop-rspec/ cops_rspec.html#rspecbeforeafterall 😕 35
36. Transactions!
37. Transactions • Begin a transaction • Create records • Run tests (each in an individual transaction) • Rollback the transaction (much faster than destroying) • … and that’s pretty much what Test Prof’s before_all & let_it_be does. 37
38. before_all • Starts a transaction • Executes the block • Rollbacks the transaction after tests • No state leak outside of f ile, reloading prevents the leaks within the f ile 38
39. let_it_be • Does the same in one line • reload modi f ier might be moved to the rails_helper 39
40. Is it really worth it? Without let_it_be With some let_it_be 40
41. Is it really worth it? 41
42. Caveats
43. Read the docs • https://test-prof.evilmartians.io/recipes/before_all?id=caveats • https://test-prof.evilmartians.io/recipes/let_it_be?id=caveats • Mentions the state leak and how to avoid it using the reload option 43
44. It’s not that simple • Simple let - > let_it_be replacement might not work • Sometimes your tests might need serious rearrangement 44
45. It’s not that simple ❌ undef i ned local variable or method `user' • Simple let - > let_it_be replacement might not work • Sometimes your tests might need serious rearrangement 45
46. It’s not that simple ❌ Validation failed: User can't be blank • Simple let - > let_it_be replacement might not work • let_it_be is run in before(:all) hook • Sometimes your tests might need serious rearrangement 46
47. It’s not that simple • Simple let - > let_it_be replacement might not work • let_it_be is run in before(:all) hook • let_it_be is order dependent • Sometimes your tests might need serious rearrangement 47
48. It’s not that simple ❌ • Simple let - > let_it_be replacement might not work • let_it_be is run in before(:all) hook • let_it_be is order dependent • Sometimes your tests might need serious rearrangement 48
49. It’s not that simple • Simple let - > let_it_be replacement might not work • let_it_be is run in before(:all) hook • let_it_be is order dependent • let_it_be doesn’t update associations automatically • Sometimes your tests might need serious rearrangement ✅ ✅ 49
50. State leaks are possible • Even with reload modi f ier, we still might have some state leaks • It is because reload doesn’t work within before_all / before(:all) hooks 50
51. State leaks are possible • Even with reload modi f ier, we still might have some state leaks • It is because reload doesn’t work within before_all / before(:all) hooks • Possible solutions: reload manually, freeze objects with freeze modi f ier to prevent updates 51
52. When to use let_it_be?
54. https://youtu.be/WNluKNQ1OFU 54
55. Me, every time let let_it_be 55
56. Be reasonable 56
57. let vs let_it_be • let_it_be is a good default • Suite consisting of one test doesn’t require let_it_be (though it won’t hurt) • Non-factory declarations could bene f it from let • Some suites might bene f it from preferring let to let_it_be 57
58. Non-factory declarations 58
59. Non-factory declarations • Don’t create anything and therefore don’t slow us too much • Can use the "partial" rede f ining • let_it_be declarations can be used in let, but not vice versa 59
60. Complex suites 60
61. Complex suites • Way one: pre-create posts & users using let_it_be with di ff erent attributes and test with that data • Way two: create data per test using let / let! 61
62. Complex suites • let_it_be: • E ff icient 👍 • Adding a new orderable f ield might force you to rewrite the whole suite 😕 62
63. Complex suites • let / let!: • Ine ff icient 😕 • Doesn’t add complexity when editing tests / code 👍 63
64. Complex suites Solution: 64
65. Complex suites Solution: 65
66. Complex suites • How big is gain from using let_it_be? • How stable is the list of attributes? • In the end, less e ff icient but more "maintainable" solution might be preferable. It’s Ruby, after all. 66
67. What else?
68. AnyFixture • Creates one record for whole run that you can re-use • Might provoke state leaks between f iles (make sure to reload) • Adding just one f ixture may give a signi f icant improvement • https://test-prof.evilmartians.io/recipes/any_ f ixture 68
69. Parallelising by f ile • On CI, our tests are currently split by example between 4 workers • The feeling is, it decreases the gain we’re getting from let_it_be • The possible side e ff ect: CI nodes become unbalanced • Experiment shows: • RSpec time for slowest node decreased from 6:12 to 5:33 (-11%) • The di ff erence was consumed by CI operations (7:18 vs 7:13) 69
70. Thank you! Makar Ermokhin P.S. We are hiring! github.com/Earendil95 twitter.com/earendil_code