MTC2018 - Search precision tuning with query rewriting -Internship at mercari-

Speaker: 八木 悠斗(Yuto Yagi)

検索では適合率と再現率のバランスが鍵となります。ここでは、メルカリで行っているクエリリライトによる適合率の調整についてご紹介します。

Finding the right balance for precision and recall is necessary in search systems. I’ll introduce our project of tuning search precision with query rewriting.

Transcript

1. Search precision tuning with query rewriting Subtitle: Internship at mercari Yuto Yagi Product Management Intern (Search)
2. Product Management Intern Search Yuto Yagi
3. Search Precision Tuning with Query Rewriting Internship at Mercari Mercari Tech Conf 2018 Product Management Intern (Search) Yuto Yagi
4. Topics Mercari Search Problem (low precision) Solution (precision tuning) Results/insights
5. Default Search Ranking Default Ranking: “Newest First (新しい順)” • Freshness Higher score for new listings • Relevance Based on : - Title - Description - Category - Brand - Status, etc. • Spam Demote duplicates/spam
6. Problem Keyword = “ipad device” (ipad 本体)
7. Problem Keyword = “ipad device” (ipad 本体)
8. Problem Keyword = “ipad device” (ipad 本体) • Too much noise = Low “precision” • Source of user frustration
9. Difficulties • Not all keywords are the same ◦ Different types of search (informational vs transactional) Uniform measures don’t work • Hard to learn an item’s relevancy to a query ◦ Short lifespan of an item ◦ Items constantly edited
10. # Problematic Keywords • Find keywords that are ◦ popular: high search count ◦ transactional: keyword, chlonological user behavior ◦ low action rates: low CTR, STR, etc.
11. Determine User Intent Query Rewriting Implementation, A/B testing What We Did Per-keyword basis precision tuning
12. Determine User Intent
13. Determine User Intent
14. Determine User Intent • The key parameter = parameter that separates the correct items from the noise e.g. category, price range, brand
15. Query Rewriting • Boost the key parameter that reflect user intent ◦ Category ▪ For: accessory pollution ◦ Price range ▪ For: for-parts items, accessory pollution ◦ Brand ▪ For: generic products, off-brands substitutes
16. Implementation and A/B Testing The management tool • Semi-automated • Quick (3 days -> 1 hour)
17. Results for “ipad device” (ipad 本体) Improved action rate: Δ%CTR: + Δ%STR: + Boosted: - dominant category electronics > PC/tablets > tablets
18. Results for “Nintendo Switch device” (ニンテンドースイッチ 本体) Boosted: - category books/music/games > video games > home console books/music/games > video games > portable console - price range Δ%CTR: + Δ%STR: +
19. Insights • User intent still hard to measure • Necessity to gather more information Exclusion keywords (09/24 released) New search logs (e.g. dwell time) etc.
20. Summary • Per-keyword basis precision tuning • Quick PDCA enabled by the semi-automated tool • Insights for longer-term solutions
21. Thank you!