Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Learning to Rank with Multimodal Data

Learning to Rank with Multimodal Data

This is a talk I gave with Dr. Kamelia Aryafar on how we use techniques for learning to rank at Etsy from the theory applied to production pipeline and challenges from two perspectives: (i) a single modality model using traditional text-based features, and (ii) a multimodal approach that includes both text and visual clues from Etsy listings.

Ea4344b9c06094807b3d1171d2363488?s=128

Nishan Subedi

July 21, 2016
Tweet

Transcript

  1. Learning to Rank with Multimodal Data @ Etsy NYC Machine

    Learning Meetup Kamelia Aryafar, Senior Data Scientist, @karyafar Nishan Subedi, Senior Software Engineer, @subedinishan Search Sciences, Etsy July 2016 1
  2. Etsy

  3. Etsy is a global marketplace where people around the world

    connect, both online and offline, to make, sell and buy unique goods. 3
  4. By the Numbers 1.6M active sellers AS OF DECEMBER 31,

    2015 24M active buyers AS OF DECEMBER 31, 2015 $2.39B annual GMS IN 2015 35+M items for sale AS OF DECEMBER 31, 2015 Photo by Kirsty-Lyn Jameson DISCLAIMER The statistics included on the following slides are updated quarterly.
  5. 819 employees around the world AS OF DECEMBER 31, 2015

    9 offices in 7 countries AS OF DECEMBER 31, 2015 Photo by Emily Andrews Work and Culture DISCLAIMER The statistics included on the following slides are updated quarterly.
  6. Large and Unique Seller Base 1.6M active sellers AS OF

    SEPTEMBER 30, 2015 95% of sellers run their Etsy shop from home 2014 ETSY SELLER SURVEY 76% consider their shop a business 2014 ETSY SELLER SURVEY Photo by Moira K. Lime DISCLAIMER The statistics included on the following slides are updated quarterly.
  7. Etsy Made in Canada Photo by Jean-Michael Seminaro 24M active

    buyers AS OF DECEMBER 31, 2015 92% of buyers agree Etsy offers products they can't find elsewhere 2014 ETSY BUYER SURVEY DISCLAIMER The statistics included on the following slides are updated quarterly.
  8. 8

  9. None
  10. None
  11. Learning To Rank 11

  12. Approaches to Learning to Rank • Pointwise - For an

    item, predict it’s grade (implicit ordering) - Labels come from interactions with items - Possible class imbalance • Pairwise - Ranking transformed to pairwise classification or regression - Labels depend on ordering of item pair - Ability to create balanced classes • Listwise - Input is entire set of documents associated with query - Output is their ranked list - Eg. Loss is a measure of the distance of ranking generated by the model to the perfect ranking for the set of documents
  13. Pairwise Learning Each training instance represents a pair of items

    from same set of search results in your logs. <item1, item2> Learner must learn to order item1 and item2 correctly, with respect to user preference decisions found in your logs.
  14. Features

  15. Label Creation for Pairwise Features {housewarming, gift, photo} - {housewarming,

    gift, ceramic, tile} → +1 {housewarming, gift, ceramic, tile } - {housewarming, gift, photo} → -1
  16. Train Classifier (SVM)

  17. Learning to Rank Pipeline

  18. Multimodal Learning to Rank

  19. None
  20. Image vs. Text Features 20 Texture Shape Color Title Tags

  21. Extracting Image Features 21

  22. title Feature Engineerings Deep Learning 22

  23. ImageNet 23 Photo from : http://www.image-net.org/

  24. Convolutional Neural Nets (CNNs) 24 Photo from: http://cs231n.stanford.edu/

  25. VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION Karen Simonyan

    & Andrew Zisserman Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg Model Specs: VGGnet
  26. Transfer Learning 26 Images Don’t Lie: Transferring Deep Visual Semantic

    Features to Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
  27. Photo by Corey Lynch

  28. Photo by Corey Lynch

  29. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale

    Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
  30. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale

    Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
  31. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale

    Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
  32. 32 Images Don’t Lie: Transferring Deep Visual Semantic Features to

    Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
  33. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale

    Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
  34. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale

    Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
  35. Production Pipeline 35

  36. Collecting Item Pairs with Labels 36

  37. Strategies for Model Application Real Time Offline Pros Can handle

    unseen items No cost to feature complexity Cons Latency cost ∝ complexity Query time feature computations Computations compound as considerations increase
  38. Real time model evaluation Fetch Model (cache, key-value store) Apply

    Ranking (ranking or reranking pass) User Query Top-k results Top-k results Reranked Index Ranking Model
  39. Gaining Confidence

  40. Gaining Confidence

  41. Performance Replays

  42. Performance Replays

  43. Ranking Replays

  44. Offline Evaluation Metrics

  45. Model Understanding: Side by Side

  46. Custom Queries & Explain Logs

  47. The future…

  48. Further Reading

  49. etsy.com/careers

  50. Thanks!