ランキングアルゴリズムにおける効率的なオンライン評価手法 ~インターリービング (Interleaving) の利用~

2019/2/19 の s-dev talks 〜サービス開発勉強会〜「仮説の立て方」(https://s-dev-talks.connpass.com/event/118037/) での発表内容です．

ランキングアルゴリズムをリリースする過程でのオンライン評価について概要をお話させていただきました．

Transcript

1. ©2019 Wantedly, Inc. ϥϯΩϯάΞϧΰϦζϜʹ͓͚Δ ޮ཰తͳΦϯϥΠϯධՁख๏ ΠϯλʔϦʔϏϯά (Interleaving) ͷར༻ s-dev talks ʙαʔϏε։ൃษڧձʙʮԾઆͷཱͯํʯ 19.Feb.2019 - Yuya Matsumura - @yu-ya4
2. ©2019 Wantedly, Inc. ✓ Yuya Matsumura (দଜ ༏໵) ✓ Software Engineer (ݕࡧɾਪનΤϯδχΞ) ✓ Wantedly, Inc. Recommendation Team (since April 2018) ✓ Interested in Information Retrieval, Machine Learning Self-Introduction @yu-ya4 @yu__ya4
3. ©2019 Wantedly, Inc. 㾎࿩͢͜ͱ
   w ϥϯΩϯάΞϧΰϦζϜʹ͓͚ΔΦϯϥΠϯධՁख๏ͷ֓ཁΛ ௚ײతʹཧղͰ͖ΔΑ͏౒Ίͨઆ໌
   㾎࿩͞ͳ͍͜ͱ
   w ਺ֶత౷ܭతͳཧ࿦ͷઆ໌ূ໌
   w ۩ମతͳ࣮૷ํ๏ About this talk
4. ©2019 Wantedly, Inc. ԾઆΛཱͯΔ
   Ծઆʹج͍ͮͯ৽͍͠ϥϯΩϯάΞϧΰϦζϜΛߟҊ
   աڈͷσʔλͳͲʹجͮ͘ΦϑϥΠϯςετʹΑΔݕূ
   "#
   ςετͳͲΛ༻͍ͨΦϯϥΠϯςετʹΑΔݕূ
   ϓϩμΫτͰϦϦʔε ϥϯΩϯάΞϧΰϦζϜมߋͷҰൠతͳྲྀΕ
5. ©2019 Wantedly, Inc. ABςετ ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷࢪࡦͷํ͕ ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ̍ ̎  ̍ ̎  Group A Ranking A Control Variation Group B Ranking B
6. ©2019 Wantedly, Inc. ABςετͷ՝୊ 㾎༗ҙͳ͕ࠩग़Δ·Ͱʹେྔͷαϯϓϧ਺ JNQSFTTJPO਺
   17਺ ͕ඞཁ
   㾎ۃ୺ͳϢʔβͷߦಈʹେ͖͘ӨڹΛड͚ΔՄೳੑ͕͋Δ Ex. *ݱࡏCVR 1%ͷ΋ͷΛ10%վળ(CVR 1.1%)͢Δʹ͸134,379΋ͷαϯϓϧ਺͕ඞཁ Ex. Ұํͷάϧʔϓʹɼ͢΂ͯͷίϯςϯπʹίϯόʔδϣϯ͢ΔϢʔβ͕͍ͨΒ…ʁ
7. ©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά(Interleaving) 2ͭͷϥϯΩϯά͔ΒަޓʹΞΠςϜΛબ୒͠৽ͨͳϥϯΩϯάΛ ੜ੒ͯ͠ɼಉҰͷϢʔβ܈ʹఏࣔͯ͠ੑೳΛධՁ͢Δ ̍ ̎  ̍ ̎  Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C
8. ©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯάͷੑೳ ࣮ݧతʹɺഒ͔Βഒैདྷͷ"#ςετΑΓ΋ޮ཰తͰ͋Δ ͜ͱ͕ ஌ΒΕ͍ͯΔ *ਖ਼֬ʹ͸ʮΠϯλʔϦʔϏϯά͸A/BςετΑΓ100ഒޮ཰తͳ৔߹͕͋Γɺ͓͓Αͦͷ৔߹ɺ 10~100ഒޮ཰తͰɺA/BςετΑΓޮ཰తͰͳ͍৔߹͸গͳ͍ʯ(Chapelle et al. 2012) (Wantedly Visit಺ͷϥϯΩϯάվળʹͯܭଌ)
9. ©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮ཰తͳཧ༝ 㾎
   ਓͷϢʔβʹ҉໧తʹෳ਺ͷΞϧΰϦζϜΛධՁͯ͠΋Β͏͜ͱ͕Ͱ͖Δ
   㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ
   㾎ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢
10. ©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮ཰తͳཧ༝ 㾎
    ਓͷϢʔβʹ҉໧తʹෳ਺ͷΞϧΰϦζϜΛධՁͯ͠΋Β͏͜ͱ͕Ͱ͖Δ
    㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ
    㾎ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢
11. ©2019 Wantedly, Inc. 1 ਓͷϢʔβʹ҉໧తʹෳ਺ͷΞϧΰϦζϜΛධՁͯ͠΋Β͏͜ͱ͕Ͱ͖Δ ̍ ̎  ̍ ̎  Ranking A Ranking B A/B Testing গͳ͍Ϣʔβ਺Ͱෳ਺ͷΞϧΰϦζϜΛൺֱධՁͰ͖Δ → Interleaving ̍ ̎ ̍ ̎
12. ©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮ཰తͳཧ༝ 㾎
    ਓͷϢʔβ͕҉໧తʹෳ਺ͷΞϧΰϦζϜΛධՁͯ͠΋Β͏͜ͱ͕Ͱ͖Δ
    㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ
    㾎ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢
13. ©2019 Wantedly, Inc. ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ ̍ ̎  ̍ ̎  Ranking A Ranking B A/B Testing ̍ ̍ ̍ ̍ ̍ ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ৔߹ ͕ࠩग़ʹ͍͘ →
14. ©2019 Wantedly, Inc. ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ Interleaving ̍ ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍ Ranking A ͱ Ranking B͕҉ʹ௚઀ൺֱ͞ΕΔͷͰ ͕ࠩग़΍͍͢ʂʂ →
15. ©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮ཰తͳཧ༝ 㾎
    ਓͷϢʔβʹ҉໧తʹෳ਺ͷΞϧΰϦζϜΛධՁͯ͠΋Β͏͜ͱ͕Ͱ͖Δ
    㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ
    㾎ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢
16. ©2019 Wantedly, Inc. ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢ ̍ ̎  ̍ ̎  Ranking A Ranking B A/B Testing  ̍ ̍  ̎ ̍ ಉ͡ΞΠςϜ͸ಉ͘͡Β͍ͷ CVR ʹͳΔ(͸ͣ)ͷͰɼ 2 ͭͷϥϯΩϯάͰ͕ࠩग़ʹ͍͘ → ʹ
17. ©2019 Wantedly, Inc. ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢ Interleaving  ̍ ̍ ̍ ̍ ̍ ̎ ̍  ̍ ಉ͡ΞΠςϜͰ΋ΑΓ্Ґʹ͍ͯͨ͠ϥϯΩϯά ΞϧΰϦζϜͷউͪʂʂ →
18. ©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯάͷ՝୊ 㾎࣮૷ίετ͕͔͔ΔɽଞͷϏδωεϩδοΫͱͷিಥ
    㾎෼͔Δͷ͸͋͘·Ͱ૬ରతͳੑೳͷྑ͠ѱ͠
    wΞϧΰϦζϜ୯ମͷੑೳ͸෼͔Βͳ͍
    wϢʔβͷϦςϯγϣϯ΍ΞϧΰϦζϜมߋʹ൐͏ଞͷػೳ΁ͷӨڹͳͲ Λଌఆ͠ʹ͍͘Γʹ͍͘
19. ©2019 Wantedly, Inc. 2 ஈ֊ͷΦϯϥΠϯධՁϓϩηε ΠϯλʔϦʔϏϯάͰ
    
    ࣍ධՁΛߦ͍༗ྗͳީิʹߜΓࠐΈɼ
    "#
    ςετͰ
    
    ࣍ධՁΛߦ͏
    
    ஈ֊ͷΦϯϥΠϯධՁγεςϜ https://medium.com/netflix-techblog/interleaving-in-online-experiments-at-netflix- a04ee392ec55
20. ©2019 Wantedly, Inc. Summary 㾎 ैདྷͷΦϯϥΠϯϥϯΩϯάධՁख๏
    "#
    ςετͷ՝୊
    w ༗ҙͳ͕ࠩग़Δ·Ͱʹେྔͷαϯϓϧ਺͕ඞཁ
    㾎 ޮ཰తͳධՁख๏ΠϯλʔϦʔϏϯάͷ঺հ
    w ഒ͔Βഒैདྷͷ
    "#
    ςετΑΓ΋ޮ཰తͰ͋ΔͱݴΘΕΔ
    㾎 ΠϯλʔϦʔϏϯάͷ՝୊
    w ͋͘·Ͱ૬ରతͳධՁ͔͠Ͱ͖ͳ͍
    㾎 
    ͭͷख๏Λ͍͍ͱ͜औΓͨ͠
    
    ஈ֊ͷΦϯϥΠϯධՁϓϩηε
    w ΠϯλʔϦʔϏϯάͰީิΛߜΓɼ"#
    ςετͰ࠷ऴతͳධՁΛԼ͢