never needed to happen. Finding unnecessary work in Rails applications — a true story. Ahmad Hamza · Toptal · Rails Consultant ahmad19.github.io · linkedin/ahmad-hamza · github/ahmad19
retry: 0, lock: :until_and_while_executing, lock_ttl: 2.days Translation: this background job may run for two days straight — and no second copy is allowed to start while it does. Someone measured that. Then wrote it down as normal.
CENTER Product catalog 300+ accounts 700 million ads at peak ⛁ product ads db ──────▶ sync IN AUS CAN GB AE SA US ⋯300+ several per country · one shared API quota 150k – 20M ads per account
OF THE 300+ ACCOUNTS ONE ACCOUNT GOOGLE US · GMC Reviews every product ~150k ads ───▶ daily ───▶ ~50k ~100k DISAPPROVED LIVE prices · images · barcodes · policy ↓ cleaning that up is our job — one Sidekiq background job RAILS APP THE JOB RESULT cron: every 30 min DisableBadProductsWorker Account healthy ───▶ find the disapproved products → fix or disable them If this job falls behind: we pay to advertise products Google won't show — and past a threshold, Google suspends the entire account. Every ad in that country goes dark. ───▶ sync
One disapproved product Why did Google reject it? ───▶ RULES Fix it or disable the ad price mismatch missing image ───▶ missing barcode (GTIN) restricted category ⋯ and many more Per product: cheap. The expensive question is different — how do you find the disapproved products among 20 million ads? Hold that question. OUTCOME ───▶ then synced back to GMC — as you saw
HISTORY, UNEDITED Sep 12 fix: there is no unique in locking Nov 06 fix: separate some workers and queues coz its all mixed up Dec 03 Remove unnecessary batching from the disapproved-products worker Feb 19 Revert migration and single country worker Feb 20 Revert product sync batching Feb 20 Re-enable product sync queue processing, purge queue for inactive GMCs Feb 26 Prevent deadlock by retrying SQL bulk inserts with exponential back-off Sun 13:11 fix worker issue 1 in 4 commits was a fix, revert, clear, or disable — 593 of 2,457 commits in four months. When every part of a system is unhealthy at once, you can't tell cause from symptom. So you fix whatever is loudest — and the loudest thing is never the real problem.
— it was built for a smaller world. THE WORLD IT WAS DESIGNED FOR · 2019 4 merchant accounts Gulf countries only · a catalog one job could walk end-to-end · cleanup ran once THE WORLD IT WAS LIVING IN · 2024 2020 ⟶ worldwide a month 300+ accounts · 700M ads re-reviewed by Google daily · one shared quota synced every 30 minutes — same architecture None of that code was wrong when it was written — the scale made it wrong. Every batch size, timeout, and lock in the repo had been quietly outgrown. The clearest way to see it: watch one line live through the growth.
this number. It was a measurement. Jul 2023 Runs keep overlapping — a lock is added 12.hours + unique: :until_executed, lock_ttl: 12.hours Feb 2024 Runs outgrow it; the timeout is removed 12.hours - unique: :until_and_while_executing, lock_ttl: 12.hours + unique: :until_and_while_executing Sep 2024 Discovery: the lock never worked — wrong option name, silently ignored for 7 months “there is no unique in locking” unique: :until_and_while_executing Sep 2024 none # Sidekiq expects `lock:` — this did nothing Lock fixed properly — sized to survive a real run 2.days - unique: :until_and_while_executing + lock: :until_and_while_executing, lock_ttl: 2.days Mar 2025 …the number changes one last time lock: :until_executing, lock_ttl: ? ?
to “how do you find them?” — fetch everything. def perform(gmc_id, page_token = nil) page = content_api.list_product_statuses(gmc_id, max_results: 250, page_token: page_token) # every ad — approved or not return if page.items.empty? self.class.perform_async(gmc_id, page.next_page_token) # …and again. and again. bad = page.items.select { |ad| ad.disapproved? } # filtered on OUR side, after the download fix_or_disable(bad) # handed off to yet more workers end Google was never asked “which ads are disapproved?” It was asked for the entire catalog — 250 ads at a time — and the answer was thrown away in Ruby. The job stops when the pages run out. That is what the two days were.
THEM Nothing in that loop is slow. Now multiply it. 20,000,000 ÷ 250 80,000 ads in the biggest account ads per API call API calls just to read the list — the same cost even when zero ads are broken 700,000,000 ads across the 300+ accounts ≈ 2,800,000 API calls to cover all accounts once — all drawn from one shared quota And the cron asked it to start again every 30 minutes — 96 requests over a two-day run, 95 of them silently discarded by the lock. Of everything it downloaded, roughly 97% needed no action at all.
ALL OF IT REAL WORK, ALL OF IT SHIPPED Everything got faster. Nothing got smaller. Nov 2024 Workers and queues separated “coz its all mixed up” reads 20M Nov 2024 Storage layer under the worker swapped — removed, reverted, rewritten within a week reads 20M Dec 03 Batching removed — “unnecessary” reads 20M Dec 04 Batching added back — one day later reads 20M Dec 06 The loop parallelized — the job now enqueues itself, page by page reads 20M Feb 2025 A pull request lands, with the most boring title yet: “improving disabled disapprove worker” ? Every fix made the job better at doing the work. Not one asked whether the work is actually needed or not.
The fix wasn't faster code. It was a better question. THE OLD QUESTION · ASKED FOR 6 YEARS THE NEW QUESTION · REPORTING API “Give me every ad.” “Give me only the broken ones.” 20,000,000 answers · we search them in Ruby ⟶ Google filters before it answers · 5,000 per page SELECT product_view.id, product_view.item_issues FROM ProductView WHERE aggregated_destination_status IN ('NOT_ELIGIBLE_OR_DISAPPROVED', …) The actual query. The Reporting API existed the whole time — the entire fix is a WHERE clause.
tuned this number either. sidekiq_options queue: 'gmc_pipeline_fixer', lock: :until_executing, lock_ttl: 1.hour 100,000 ÷ 5,000 20 broken ads in the biggest account — even on a bad day ads per API call API calls — down from 80,000 Two days became one hour — not because the code got faster, but because most of the work never needed to happen.
CLAUSE The fires went out on their own. Nov – Feb Commits touching the ads pipeline — my first four months Mar – Jun The same four months, after the fix — and still falling since Jun The worker at the center of six years of firefighting — one oneline tweak, then nothing 154 91 quiet We never fixed the loud things. They went quiet on their own — that's how you know you found the real problem.
this one line taught me. 01 Make the work scale with the size of the problem — not the size of the system. 02 Timeouts, TTLs and batch sizes are measurements. When one looks insane — believe it. 03 The loudest problem is never the real one. Ask what question the system is answering. 04 Your platform may already have the API you're missing. Read the docs. The biggest performance win was realizing most of the work never needed to happen. — thank you. ahmad19.github.io · linkedin/ahmad-hamza · github/ahmad19
marcoroth
kishida
minodriven
siroemk
sorah
yasaichi
y0hgi
clockmaker
andpad
iwatsukayura
rkaga
kaityo256
carmenintech
afnizarnur
akademiya2063
jakevdp
sabderemane
szymonslowik
akosma
tamaranovitovic
morganepeng
techseoconnect
eileencodes
sstephenson
