MTC2018 - Mercari's Search Service: Now and the Future

Speaker: Kenji Sugiki, Yanpeng Lin

The Search Team develops and improves on features for the Mercari app such as suggestions, related words, and the recommendations tab. In this session, Sugiki will introduce the various features they are developing in their goal to compete on the global stage.

Transcript

1. Mercari's Search Service: Now and the Future Kenji Sugiki Software Engineer (Search) Yanpeng Lin Software Engineer (Search)
2. The Present and Future of Mercari’s Search Services
3. Software Engineer Search Kenji Sugiki
4. Mercari’s Search Services at Present
5. Largest number of listings in Japan Total number of listings on the Mercari app 7/13/2018 million 1 billion
6. Lots of active traffic (listing→buying) GMV/MAU Trends GMV and MAU are increasing steadily. 100MM yen 10k people
7. Mercari’s uniqueness = C2C ・Life cycle of listings is fast 　・One of a kind (only one user can purchase) ・High frequency of listings and updates 　・High MAU and GMV 　・Price negotiations and edits occur frequently ・Master catalog is largely unsaved The quality and function of searches are very important
8. Search functions supporting Mercari Timeline Item search New listing notifications Related listings Keyword completion Related keywords Recommendation timeline Functions that have been added or improved in the last year
9. Current search platform: monolithic environment 　　　 　　　 mercari-api 　　　 　　　 ・・・ 　　　 ・・・ Master/slave configuration Data divided by service Proxy Reduce internal hassle
10. Internal search API: basic design QueryBuilder Reranking SolrFetcher PostItemFilter Search service QueryRewriter 　　　 Query generated from parameters Logic AB Test Dynamic query conversion Adjust/exclude certain items Search field Synonyms Token filter Internal service Timeline, related listings, etc. ResultBuilder Controller Additional data from DB Format Solr result Search pipeline
11. Search team’s current work ・Add functions, correct logic, AB tests ・Solr operations (synonyms, schema changes) ・Create next-generation search platform ・AI-based search optimization Search system + quality improvement ・Keyword completion ・Related keywords ・Personalized recommendations 　・Recommendation timeline Search functions + UI/UX improvement Responsible for all of the various functions to search listings
12. Example of Search Functions That Have Been Developed
13. Recommendation timeline tab Public release at the end of last year Simple design with a big effect
14. mercari-api Viewing history API Acquire frequent categories/brands from viewed listings user_id Build Solr filter query and cache Data acquisition New arrival notification Data inquiries Polling Architecture of the recommendation timeline 　　　
15. Related Keyword Service
16. Related keyword service Popular words by category Popular words by brand Words related to the keyword Can be applied to only categories or brands *Currently public release is only for iOS
17. mercari-api Related keywords Batch computation All data is cached Keyword Categories/brands Related keywords Architecture for related keywords (API side) Name-based data: keyword→keyword Related keyword: keyword→related keyword... Kubernetes Engine Save
18. Batch pipeline (daily cron-like execution) Kubernetes Engine Generate related keywords Import Mercari API Delete keyword differences Architecture for related keywords (batch side; CI/CD) Develop git push Automatic image building and deployment Notification of batch results via Slack
19. User Search keyword Item Search keyword N x M K x M BigQuery ① Acquire user event log: search with keyword then tap items ② Keyword normalization ・NFKC + lowercase ・Synonyms, restricted words, spacing ・Align according to most-tapped keywords ③ Determine categories/brands for each keyword ・Aggregate frequency of categories/brands of tapped items ・Calculate skewed distribution with unique UDFs ・Removal may occur ⑤ Merge both models Deduplicate if related keywords have an inclusion relation ④ Transform into sparse matrices to calculate keyword similarity - keyword, user_id - keyword, item_id Generation of related keywords
20. Search: The Inner Face of Query Autocomplete
21. Software Engineer Search Yanpeng Lin
22. Towards a Safer and More Secure Marketplace
23. Query Autocomplete (QAC) Merca Mercari store Mercari book Mercari material Mercado bag Release on demand Keyword quality control
24. QAC built upon microservice platform Client API gateway Mercari API Sakura GCP QAC QAC K8S Continuous testing Continuous building Continuous delivery QAC quality control service
25. Release on demand Pull request Tagging Continuous delivery Continuous testing Continuous building Continuous delivery
26. Keyword quality control Filter function 1 Filter function 2 Filter function ... Filter function ... Filter function N Recall strategy 1 Recall strategy 2 Recall strategy ... Recall strategy ... Recall strategy N QAC (quality control service) Matching function
27. Keyword quality control (text matching) Filter function 1 Filter function 2 Filter function ... Filter function ... Filter function N Recall strategy 1 Recall strategy 2 Recall strategy ... Recall strategy ... Recall strategy N Bag Shine muscat Erotica XXX Bag Shine muscat
28. Keyword quality control (semantic matching) Matching function Keyword Item Heterogeneous network embedding Super-bit hashing Lucene angle matching
29. Keyword quality control (semantic matching) Kid Halloween ?
30. Keyword quality control (semantic matching) Child Pirate Tigger Boo Peter Pan ・・・ Kid Halloween Woody
31. QAC microservice recap Client API gateway Mercari API Sakura GCP QAC QAC K8S Continuous testing Continuous building Continuous delivery QAC quality control service
32. The Future of Mercari’s Search Services
33. Creating a world-standard search team ・Recruiting more members ・From monolithic to microservice architecture ・Create a more scalable, stronger search platform ・Apply IR, NLP, and ML in ways we can show to the world 　・Structure semi-structured data by detecting item attributes and entities 　・High-speed PDCA for search improvements 　　・Consider search evaluation metrics unique to C2C 　　・Automatic rank tuning (LTR) 　・Participate in and present at various international conferences
34. In July and September, we attended international conferences and held briefing sessions! Please join us next year!