The Whop & Chop: Cutting CI time in half

Transcript

1. The Whop & Chop Cutting CI Time in half Irina Nazarova
2. Thank you!
3. 3 400000000 800000000 1200000000 1600000000 April 2021 April 2022 April 2023 April 2024 April 2025 1.48B 1.08B Total transaction volume, $ 0.74B 0.52B fast-paced startup built on Rails and Next.js
6. The stack 6 Problem: Slow test suite Rspec thoughtbot/ factory_bot
7. 7 github.com/test-prof/test-prof
8. 8 Rspec Minitest TestProf: toolbox to profile slow tests -StackProf Integration -FactoryProf -TagProf -EventProf github.com/ tmm1/ stackprof
9. 9 Rspec Minitest TestProf: helpers to speed up slow tests -let_it_be: create test data once for an entire context -factory_default: avoid rebuilding associated objects by reusing a cached instance -before_all: runs setup only once per group
10. Let’s get to profiling! Find the report in tmp/test-prof/ Review fl amegraph with Speedscope.app Focus: top method time spent 10
11. StackProf report 11
12. Top time-consuming methods -Heavy database activity (Trilogy#query), suggesting lots of factories or queries. -Calls to external services like Stripe and OpenAI that should be mocked. -Signi fi cant amount of time spent in logging. 12
13. Let’s switch off logging in test! 13
14. A custom Sentry logger detected! 14 Let’s go a bit further
15. Easy! How much time did we save? 15 Before: Single process: ~25 min CI (16 workers): ~4 min After: Single process: ~12 min CI: ~2 min 50% CI time saved!
16. Next suspect: FactoryBot TagProf to watch for AR, factory creations and Sidekiq jobs factory.create events were now responsible for more than half of the total test execution time 16
17. Factory Optimization Strategy 1: let_it_be — Create Once, Use Many Replace let with let_it_be to create the record once for the entire describe block. Don’t use it on objects whose state is modi fi ed by one test. 17
18. Strategy 2: create_default — create resuable associations 18 Let’s fi rst pro fi le factories: compare total vs. top-level factories to spot an issue in associations Factory Optimization
19. Strategy 2: create_default — create resuable associations 19 Factory Optimization
20. What’s next? 20
21. Parallelization Swap parallel_tests for test-queue Problem: it splits tests by fi les. 21 parallel_tests test-queue Solution: split by individual tests.
22. Sound good? Well… Over 100 tests failed. The errors were bizarre and seemingly random, with no obvious connection to our changes. 22
23. The Flaky Test Hunt: A Guide to Isolation Our new test-queue setup had exposed years of hidden state leakage. 23
24. How we fought this… 24
25. Soon on evilmartians.com /chronicles 25
26. Shoutouts Julia Egorova Igor Platonov Travis Turner Vladimir Dementyev Reach out: irina@evilmartians.com