Slide 1

Slide 1 text

The Whop & Chop Cutting CI Time in half Irina Nazarova

Slide 2

Slide 2 text

Thank you!

Slide 3

Slide 3 text

3 400000000 800000000 1200000000 1600000000 April 2021 April 2022 April 2023 April 2024 April 2025 1.48B 1.08B Total transaction volume, $ 0.74B 0.52B fast-paced startup built on Rails and Next.js

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

The stack 6 Problem: Slow test suite Rspec thoughtbot/ factory_bot

Slide 7

Slide 7 text

7 github.com/test-prof/test-prof

Slide 8

Slide 8 text

8 Rspec Minitest TestProf: toolbox to profile slow tests -StackProf Integration -FactoryProf -TagProf -EventProf github.com/ tmm1/ stackprof

Slide 9

Slide 9 text

9 Rspec Minitest TestProf: helpers to speed up slow tests -let_it_be: create test data once for an entire context -factory_default: avoid rebuilding associated objects by reusing a cached instance -before_all: runs setup only once per group

Slide 10

Slide 10 text

Let’s get to profiling! Find the report in tmp/test-prof/ Review fl amegraph with Speedscope.app Focus: top method time spent 10

Slide 11

Slide 11 text

StackProf report 11

Slide 12

Slide 12 text

Top time-consuming methods -Heavy database activity (Trilogy#query), suggesting lots of factories or queries. -Calls to external services like Stripe and OpenAI that should be mocked. -Signi fi cant amount of time spent in logging. 12

Slide 13

Slide 13 text

Let’s switch off logging in test! 13

Slide 14

Slide 14 text

A custom Sentry logger detected! 14 Let’s go a bit further

Slide 15

Slide 15 text

Easy! How much time did we save? 15 Before: Single process: ~25 min CI (16 workers): ~4 min After: Single process: ~12 min CI: ~2 min 50% CI time saved!

Slide 16

Slide 16 text

Next suspect: FactoryBot TagProf to watch for AR, factory creations and Sidekiq jobs factory.create events were now responsible for more than half of the total test execution time 16

Slide 17

Slide 17 text

Factory Optimization Strategy 1: let_it_be — Create Once, Use Many Replace let with let_it_be to create the record once for the entire describe block. Don’t use it on objects whose state is modi fi ed by one test. 17

Slide 18

Slide 18 text

Strategy 2: create_default — create resuable associations 18 Let’s fi rst pro fi le factories: compare total vs. top-level factories to spot an issue in associations Factory Optimization

Slide 19

Slide 19 text

Strategy 2: create_default — create resuable associations 19 Factory Optimization

Slide 20

Slide 20 text

What’s next? 20

Slide 21

Slide 21 text

Parallelization Swap parallel_tests for test-queue Problem: it splits tests by fi les. 21 parallel_tests test-queue Solution: split by individual tests.

Slide 22

Slide 22 text

Sound good? Well… Over 100 tests failed. The errors were bizarre and seemingly random, with no obvious connection to our changes. 22

Slide 23

Slide 23 text

The Flaky Test Hunt: A Guide to Isolation Our new test-queue setup had exposed years of hidden state leakage. 23

Slide 24

Slide 24 text

How we fought this… 24

Slide 25

Slide 25 text

Soon on evilmartians.com /chronicles 25

Slide 26

Slide 26 text

Shoutouts Julia Egorova Igor Platonov Travis Turner Vladimir Dementyev Reach out: [email protected]