Share
Embed
Not All Matches Are Equally Valuable: An Online Experiment of Retention-Focused Recommendation in a Job-Matching Platform
Tatsuya Ute
November 02, 2026
Transcript
16 slides
-
RecSys in HR 2026 Not All Matches Are Equally Valuable:
An Online Experiment of Retention-Focused Recommendation in a Job-Matching Platform Tatsuya Ute, Chiaki Ichimura, Yuta Saito | Sep 28, 2026 © 2026 Wantedly, Inc. -
Recommendation in Wantedly Visit Companies search for job seekers ("users")
and contact them first © 2026 Wantedly, Inc. -
What we have been optimizing We originally tuned the recommender
system to maximize scouts and matches. • Matching platforms commonly optimize immediate engagement: click-through rate, reply rate, total matches, etc. • Total matches are a proxy for the business outcomes we actually care about: revenue and user retention, etc. → Does maximizing matches actually help user retention? © 2026 Wantedly, Inc. -
Pre-analysis: matches and churn Churn risk seems higher for users
with very few recent matches [1] [1] this is an observation, not causality © 2026 Wantedly, Inc. -
Reframing the objective Conventional • Maximize the total number of
Retention-aware • users matches over all lists • • One more match anywhere counts Minimize the number of churn-risk • A user at risk can be prioritized even the same when they are not the Exposure concentrates on users who highest-probability match match easily Kishimoto et al. (ICLR 2026) formalize this objective and propose MRet, a learned re-ranker. Ours is simpler: we add a small boost to the existing ranking, and test it live. © 2026 Wantedly, Inc. -
A lightweight post-processing boost We add a boost to the
scores of churn-risk users only. The boost shrinks as the user gets more exposure. Scores in the figure are illustrative. © 2026 Wantedly, Inc. -
The score boost, precisely : baseline score : updated score,
clipped to [-1, 1] for boosted pairs : appearances within a ranking cutoff across companies : base boost factor : diversity decay factor : churn-risk users, fewer than 3 recent scouts [1] : minimum baseline score at which the boost applies [1] Risk counts scouts because a match also needs a reply, so the ranking most directly affects scout receipt. The threshold of three came from a churn prediction model on pre-treatment behavioral features. © 2026 Wantedly, Inc. -
Experiment setup Design • Four-week A/B test, preceded by a
Analysis • Active in the pre-period with an 28-day pre-period impression: ~20.9k control-exposed, • Randomized at the recruiter level ~20.4k treatment-exposed • One user can be ranked by recruiters • in both arms Churned if no login is recorded during the four-week test • Logistic regression with 14 covariates measured in the pre-period © 2026 Wantedly, Inc. -
Directionally lower churn, but not significant Odds ratio (<1 =
lower churn odds) p-value Main (overlap allowed) 0.957 0.128 One arm only [1] 0.948 0.223 Prior scouts: 0 [2] 0.956 0.200 Prior scouts: 1-2 [2] 0.948 0.359 Prior scouts: 3+ [2] 1.015 0.893 Neither main estimate is significant at p < 0.05. [1] Users seen by a single arm (6.6k control, 6.0k treatment). [2] Exploratory and underpowered; it does not establish heterogeneity. 0-2 is the targeted group. © 2026 Wantedly, Inc. -
Matching outcomes and the company side Match volume • +5.4%
point estimate (Poisson, Company guardrails • company controls) • Not significant, so not evidence of 0.77) • improvement • Not evidence that volume was preserved either © 2026 Wantedly, Inc. Actual company churn: +0.006 (p = Model-based company churn score: -0.007 (p = 0.34) • Weighted least squares, company-clustered robust SE -
Conclusion Maximizing matches alone may be misaligned with retention. •
Match value is not uniform: users with very few recent matches are more likely to leave • A post-processing boost re-ranks for retention with no retraining, and showed no evidence of company-side harm • Whether it actually reduces churn is still inconclusive after four weeks © 2026 Wantedly, Inc. -
Limitations What we cannot claim • • • One platform,
four weeks; may not How it was built • Compared only against the generalize to other markets or longer production baseline, not congestion- horizons or fairness-aware Cross-arm exposure limits a clean • Threshold and coefficients from user-level causal reading offline analyses, not learned from the Mechanism not established: objective exposure, scouts and matches not verified © 2026 Wantedly, Inc.