[mercari GEARS 2025] Foundations of AI - The Invisible Forces Driving Product Innovation

Transcript

1. Foundations of AI: The Invisible Forces Driving Product Innovation
 Sho Akiyama
 Mercari / Engineering Manager
 Andre Rusli
 Mercari / Research Engineer

2. • A founding member of the newly established Applied AI Research team
 • Previously worked as an ML engineer working in multiple different domains, including: LLM fine-tuning, AI-supported listing, embeddings for search, and on-device AI.
 • Engineering Manager for the AI/LLM team at Mercari.
 • Joined Mercari in 2024 as an individual contributor (MLOps and applied ML infrastructure).
 • Prior experience in mobile, web, and ML/MLOps engineering at companies including TRIAL Inc. (Fukuoka) and Cogent Labs (Tokyo).
 Engineering Manager, AI/LLM
 Sho Akiyama
 Andre Rusli, PhD
 
 
 Research Engineer, Applied AI
 

3. The journey of product innovation sparked by images.
 1) Origin – from a forgotten piece of experimental code
 2) Evolution – the rise of the embeddings revolution
 3) Explosion – the long-awaited revival of Image Search
 4) Expansion – from AI listing to contextual semantic search
 5) Future – continuous research and knowledge sharing
 List of Chapters

4. (1/5)
 The Origin

5. Origin
 Similar Looks
 Recommendations

6. 01 Dormant Idea
 A small experiment with SigLIP image embeddings sat unused in a personal repo, initially considered experimental and not ready for production.
 02 Spark of Curiosity
 Picked up again when unclaimed ideas surfaced. SigLIP was bootstrapped internally without external mandate.
 03 First Customer
 Recommendation team became the earliest adopter, validating real-world impact for existing MobileNet use case to try SigLIP. Story kicked off with a casual team building lunch that turned into a roadmap discussion.

7. Origin
 - The foundation of the journey wasn’t a top-down initiative but grassroots engineering curiosity.
 - Lesson: small experiments can become infra-level breakthroughs if coupled with product alignment.

8. (2/5)
 The Evolution

9. Initial reusable pipeline:
 Image ingestion
 ↓
 Embedding model
 ↓
 ANN index
 ↓
 Image similarity search service(s)
 Embeddings moved from “experimental” to “critical”
 

10. Embeddings moved from “experimental” to “critical infra.”
 
 Additionally, to get people onboarded and excited within the company, 
 “Embeddings as a Meme” → #z-embeddings Slack channel becomes shorthand for new use cases even with non-technical teams.
11. Embeddings moved from “experimental” to “critical infra.”
 - Migration from MobileNet to embeddings for Similar Looks
 - 1.5x increase in tap rate
 - +14% increase in purchase count via Item Detail Page

12. Evolution
 One of the first clear demonstrations that embeddings ≠ research toy → direct GMV impact.

13. (3/5)
 The Explosion

14. Explosion: Features
 (1)
 
 「Engineering」
 
 ML / Infrastructure / Backend / Client / Frontend
 (2)
 
 「Product」
 
 Metrics / Analysis / Milestones

15. Explosion: Features
 Similar Looks
 Recommendations
 Image Search

16. From “single-use critical infra.” to “reusable critical infra.”
 
 Reusable inference for image embeddings

17. Explosion
 Image Search

18. Explosion
 (1)
 
 「Engineering」
 
 ML / Infrastructure / Backend / Client / Frontend
 (2)
 
 「Product」
 
 Metrics / Analysis / Milestones
 (3)
 
 「Marketing」
 
 Events / CM / Onboarding / User Growth

19. Explosion
 Events / CM / SNS

20. Explosion
 (1)
 
 「Engineering」
 
 ML / Infrastructure / Backend / Client / Frontend
 (2)
 
 「Product」
 
 Metrics / Analysis / Milestones
 (3)
 
 「Marketing」
 
 Events / CM / Onboarding / User Growth
 (4)
 
 「Research」
 
 
 Conferences / Papers

21. Explosion
 Presented at multiple conferences around the world
 • FOSSASIA Summit in Bangkok, Thailand
 • 画像の認識・理解シンポジウム MIRU2025 in Kyoto, Japan
 • ACM KDD 2025 in Toronto, ON, Canada
 • 19th ACM Conference on Recommender Systems (RecSys 2025) in Prague, Czech
 Events / CM / SNS

22. (4/5)
 The Expansion

23. Beyond Image Search: Expanding to Other Features
 (2)
 
 「Search」
 
 Domain-
 specific Text Embeddings
 (1)
 
 「AI Listing」
 
 Enhancing Category Prediction

24. (1)
 
 「AI Listing」
 
 Category Prediction

25. image-to-
 images similarity
 (1)
 
 「AI Listing」
 
 Category Prediction
 Upload images
 Predict categories
 🪄 ✨
 
 Generated!
 ✅✅
 Category #1
 
 Category #2
 …
 
 Category #n
 Listing Completion Rate via AI Listing 📈🚀

26. (2)
 
 「Search」
 
 Context-
 Aware Text Embeddings
 “Onepiece summer”
 “Onepiece goods”
 “Coach How To”
 “Coach Mens”
 “Coach”
 “Super Rare Chanel Belt Affordable Used”
 👗🏖🩱 󰬬👒🧸 󰠅⚽📈 🛍👜🧥 ?
 ?

27. Solution: A fine-tuned text embeddings model
 
 Training set
 “Query” & “Item Title” pairs
 
 Base model
 cl-nagoya/ruri-small-v2 Loss Functions
 MultipleNegativeRankingLoss
 MatryoshkaLoss
 (2)
 
 「Search」
 
 Context-
 Aware Text Embeddings

28. Offline evaluation shows promising results
 
 
 
 
 
 In addition to a comprehensive qualitative manual checks by internal experts.
 Model
 # of dim
 nDCG@k
 nDCG@k 
 (long queries)
 Prec@k
 Recall@k
 Baseline
 32 (PCA)
 0.099
 0.142
 0.007
 0.272
 Fine-tuned
 32 (MRL)
 0.195
 0.235
 0.015
 0.607
 (2)
 
 「Search」
 
 Context-
 Aware Text Embeddings

29. Product 🤝 Engineering 🤝 Research
 Beyond the Features: An Organizational Shift
 The「Applied AI Research」Team
 is formed
 🎊🎊🎊

30. The Applied AI Research (AAIR) Team
 A product-aligned but execution-autonomous driver of AI innovation: delivering research that ships, by working ahead of the roadmap – not outside of it.
 01
 Semantic Understanding
 02
 Contextual Intelligence
 03
 Continuous AI Learning

31. (5/5)
 The Future

32. Multimodal embeddings
 Going beyond only-text, or only-image, understanding.
 Advanced multi-lingual NLP/U
 Enabling our platform to reach more global audiences.
 User embeddings
 Personalized search and discovery, moderation (TnS), etc.
 
 and many more exciting projects!
 Going Forward

33. Thank You!