Web界隈LT会_爆速改善_Interleaving.pdf

Transcript

1. ©2018 Wantedly, Inc. ര଎վળʂInterleaving ABςετΑΓޮ཰తͳϥϯΩϯάධՁΛٻΊͯ Webք۾LTձ 6.Jul.2018 - Yuya Matsumura

   - @yu-ya4

2. ©2018 Wantedly, Inc. •Yuya Matsumuraʢদଜ ༏໵ʣ •Wantedly, Inc. (ࣾձਓ4ϲ݄໨) •From

   Kyoto •Interested in Information Retrieval & Recommendation Self Introduction @yu-ya4

3. ©2018 Wantedly, Inc. ABςετͯ͠·͔͢ʁ

4. ©2018 Wantedly, Inc. ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷ ࢪࡦͷํ͕ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ABςετͱ͸ WantedlyͰ͸ɼجຊతʹ͢΂ͯͷมߋʹରͯ͠ABςετΛߦ͔ͬͯΒ ͷਖ਼ࣜϦϦʔεΛߦ͍ͬͯ·͢ʂ

5. ©2018 Wantedly, Inc. Wantedlyʹ͓͚ΔϥϯΩϯάͷABςετ A/B Testing ̍ ̎  ̍

   ̎  Group A Ranking A Control Variation Group B Ranking B

6. ©2018 Wantedly, Inc. ༗ҙͳ͕ࠩग़Δ·Ͱʹ ABςετͷ՝୊ →PDCAαΠΫϧΛճ͢ͷ͕஗͘ͳΔʂ Ex. *ݱࡏCVR 1%ͷ΋ͷΛ10%վળ(CVR 1.1%)͢Δʹ͸134,379΋ͷαϯϓϧ਺͕ඞཁ

   *༗ҙਫ४Ћ=5%, ݕग़ྗ1-β=95%(https://keisan.casio.jp/exec/user/1489063696) େྔͷαϯϓϧ਺(impression਺, PV਺)͕ඞཁ

7. ©2018 Wantedly, Inc.

8. ©2018 Wantedly, Inc. ͦ͜ͰInterleaving

9. ©2018 Wantedly, Inc. 2ͭͷϥϯΩϯά͔Βަޓʹݕࡧ݁ՌΛ࠷্Ґ͔Βॱʹ બ୒͠1ͭͷϥϯΩϯάΛੜ੒ͯ͠Ϣʔβʹఏࣔ͢Δ Interleavingͱ͸ ̍ ̎  ̍

   ̎  Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C ᶃ ᶄ ᶅ ᶆ

10. ©2018 Wantedly, Inc. Interleavingͷੑೳ →ര଎ͰPDCAαΠΫϧ͕ճͤΔʂ *ਖ਼֬ʹ͸ʮΠϯλʔϦʔϏϯά͸A/BςετΑΓ100ഒޮ཰తͳ৔߹͕͋Γɺ͓͓Αͦͷ৔߹ɺ 10~100ഒޮ཰తͰɺA/BςετΑΓޮ཰తͰͳ͍৔߹͸গͳ͍ʯ(Chapelle et al. 2012)

    Interleaving͸ ABςετΑΓ10ʙ100ഒޮ཰త*

11. ©2018 Wantedly, Inc.

12. ©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮ཰తͳཧ༝ ̍ ̎  ̍ ̎ 

    Ranking A Ranking B A/B Testing ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ৔߹ ͕ࠩग़ʹ͍͘ → ̍ ̍ ̍ ̍ ̍

13. ©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮ཰తͳཧ༝ Interleaving ̍ Ranking A ͱ Ranking

    B͕҉ʹ௚઀ൺֱ͞ΕΔͷͰ ͕ࠩग़΍͍͢ʂʂ → ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍

14. ©2018 Wantedly, Inc. ࣮ࡍͷABςετͱInterleavingͷp-valueͷऩଋͷൺֱ (Wantedly Visit಺ͷϥϯΩϯάվળʹͯܭଌ)

15. ©2018 Wantedly, Inc. ͦͷ݁Ռ…

16. ©2018 Wantedly, Inc. ৽ଔೖࣾ1ϲ݄൒ͰWantedly Visitͷ ৽͍͠ϥϯΩϯάࢪࡦΛຊ൪ϦϦʔε (ࠓͰ΋ܧଓͯ͠ྑ͍݁ՌΛग़͠ଓ͚͍ͯ·͢)

17. ©2018 Wantedly, Inc. InterleavingΛ׆༻ͯ͠ര଎ϥϯΩϯάվળΛ