Web界隈LT会_爆速改善_Interleaving.pdf

A/Bテストより10-100倍効率的と言われるランキング評価手法、Interleavingについて、ザックリと概要をお話ししました。

Transcript

1. ©2018 Wantedly, Inc. ര଎վળʂInterleaving ABςετΑΓޮ཰తͳϥϯΩϯάධՁΛٻΊͯ Webք۾LTձ 6.Jul.2018 - Yuya Matsumura - @yu-ya4
2. ©2018 Wantedly, Inc. •Yuya Matsumuraʢদଜ ༏໵ʣ •Wantedly, Inc. (ࣾձਓ4ϲ݄໨) •From Kyoto •Interested in Information Retrieval & Recommendation Self Introduction @yu-ya4
3. ©2018 Wantedly, Inc. ABςετͯ͠·͔͢ʁ
4. ©2018 Wantedly, Inc. ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷ ࢪࡦͷํ͕ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ABςετͱ͸ WantedlyͰ͸ɼجຊతʹ͢΂ͯͷมߋʹରͯ͠ABςετΛߦ͔ͬͯΒ ͷਖ਼ࣜϦϦʔεΛߦ͍ͬͯ·͢ʂ
5. ©2018 Wantedly, Inc. Wantedlyʹ͓͚ΔϥϯΩϯάͷABςετ A/B Testing ̍ ̎  ̍ ̎  Group A Ranking A Control Variation Group B Ranking B
6. ©2018 Wantedly, Inc. ༗ҙͳ͕ࠩग़Δ·Ͱʹ ABςετͷ՝୊ →PDCAαΠΫϧΛճ͢ͷ͕஗͘ͳΔʂ Ex. *ݱࡏCVR 1%ͷ΋ͷΛ10%վળ(CVR 1.1%)͢Δʹ͸134,379΋ͷαϯϓϧ਺͕ඞཁ *༗ҙਫ४Ћ=5%, ݕग़ྗ1-β=95%(https://keisan.casio.jp/exec/user/1489063696) େྔͷαϯϓϧ਺(impression਺, PV਺)͕ඞཁ
7. ©2018 Wantedly, Inc.
8. ©2018 Wantedly, Inc. ͦ͜ͰInterleaving
9. ©2018 Wantedly, Inc. 2ͭͷϥϯΩϯά͔Βަޓʹݕࡧ݁ՌΛ࠷্Ґ͔Βॱʹ બ୒͠1ͭͷϥϯΩϯάΛੜ੒ͯ͠Ϣʔβʹఏࣔ͢Δ Interleavingͱ͸ ̍ ̎  ̍ ̎  Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C ᶃ ᶄ ᶅ ᶆ
10. ©2018 Wantedly, Inc. Interleavingͷੑೳ →ര଎ͰPDCAαΠΫϧ͕ճͤΔʂ *ਖ਼֬ʹ͸ʮΠϯλʔϦʔϏϯά͸A/BςετΑΓ100ഒޮ཰తͳ৔߹͕͋Γɺ͓͓Αͦͷ৔߹ɺ 10~100ഒޮ཰తͰɺA/BςετΑΓޮ཰తͰͳ͍৔߹͸গͳ͍ʯ(Chapelle et al. 2012) Interleaving͸ ABςετΑΓ10ʙ100ഒޮ཰త*
11. ©2018 Wantedly, Inc.
12. ©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮ཰తͳཧ༝ ̍ ̎  ̍ ̎  Ranking A Ranking B A/B Testing ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ৔߹ ͕ࠩग़ʹ͍͘ → ̍ ̍ ̍ ̍ ̍
13. ©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮ཰తͳཧ༝ Interleaving ̍ Ranking A ͱ Ranking B͕҉ʹ௚઀ൺֱ͞ΕΔͷͰ ͕ࠩग़΍͍͢ʂʂ → ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍
14. ©2018 Wantedly, Inc. ࣮ࡍͷABςετͱInterleavingͷp-valueͷऩଋͷൺֱ (Wantedly Visit಺ͷϥϯΩϯάվળʹͯܭଌ)
15. ©2018 Wantedly, Inc. ͦͷ݁Ռ…
16. ©2018 Wantedly, Inc. ৽ଔೖࣾ1ϲ݄൒ͰWantedly Visitͷ ৽͍͠ϥϯΩϯάࢪࡦΛຊ൪ϦϦʔε (ࠓͰ΋ܧଓͯ͠ྑ͍݁ՌΛग़͠ଓ͚͍ͯ·͢)
17. ©2018 Wantedly, Inc. InterleavingΛ׆༻ͯ͠ര଎ϥϯΩϯάվળΛ