Web界隈LT会_爆速改善_Interleaving.pdf

 Web界隈LT会_爆速改善_Interleaving.pdf

A/Bテストより10-100倍効率的と言われるランキング評価手法、Interleavingについて、ザックリと概要をお話ししました。

3c0649db1ae5ca0ae57c76037243f501?s=128

Yuya Matsumura

July 06, 2018
Tweet

Transcript

  1. ©2018 Wantedly, Inc. ര଎վળʂInterleaving ABςετΑΓޮ཰తͳϥϯΩϯάධՁΛٻΊͯ Webք۾LTձ 6.Jul.2018 - Yuya Matsumura

    - @yu-ya4
  2. ©2018 Wantedly, Inc. •Yuya Matsumuraʢদଜ ༏໵ʣ •Wantedly, Inc. (ࣾձਓ4ϲ݄໨) •From

    Kyoto •Interested in Information Retrieval & Recommendation Self Introduction @yu-ya4
  3. ©2018 Wantedly, Inc. ABςετͯ͠·͔͢ʁ

  4. ©2018 Wantedly, Inc. ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷ ࢪࡦͷํ͕ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ABςετͱ͸ WantedlyͰ͸ɼجຊతʹ͢΂ͯͷมߋʹରͯ͠ABςετΛߦ͔ͬͯΒ ͷਖ਼ࣜϦϦʔεΛߦ͍ͬͯ·͢ʂ

  5. ©2018 Wantedly, Inc. Wantedlyʹ͓͚ΔϥϯΩϯάͷABςετ A/B Testing ̍ ̎  ̍

    ̎  Group A Ranking A Control Variation Group B Ranking B
  6. ©2018 Wantedly, Inc. ༗ҙͳ͕ࠩग़Δ·Ͱʹ ABςετͷ՝୊ →PDCAαΠΫϧΛճ͢ͷ͕஗͘ͳΔʂ Ex. *ݱࡏCVR 1%ͷ΋ͷΛ10%վળ(CVR 1.1%)͢Δʹ͸134,379΋ͷαϯϓϧ਺͕ඞཁ

    *༗ҙਫ४Ћ=5%, ݕग़ྗ1-β=95%(https://keisan.casio.jp/exec/user/1489063696) େྔͷαϯϓϧ਺(impression਺, PV਺)͕ඞཁ
  7. ©2018 Wantedly, Inc.

  8. ©2018 Wantedly, Inc. ͦ͜ͰInterleaving

  9. ©2018 Wantedly, Inc. 2ͭͷϥϯΩϯά͔Βަޓʹݕࡧ݁ՌΛ࠷্Ґ͔Βॱʹ બ୒͠1ͭͷϥϯΩϯάΛੜ੒ͯ͠Ϣʔβʹఏࣔ͢Δ Interleavingͱ͸ ̍ ̎  ̍

    ̎  Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C ᶃ ᶄ ᶅ ᶆ
  10. ©2018 Wantedly, Inc. Interleavingͷੑೳ →ര଎ͰPDCAαΠΫϧ͕ճͤΔʂ *ਖ਼֬ʹ͸ʮΠϯλʔϦʔϏϯά͸A/BςετΑΓ100ഒޮ཰తͳ৔߹͕͋Γɺ͓͓Αͦͷ৔߹ɺ 10~100ഒޮ཰తͰɺA/BςετΑΓޮ཰తͰͳ͍৔߹͸গͳ͍ʯ(Chapelle et al. 2012)

    Interleaving͸ ABςετΑΓ10ʙ100ഒޮ཰త*
  11. ©2018 Wantedly, Inc.

  12. ©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮ཰తͳཧ༝ ̍ ̎  ̍ ̎ 

    Ranking A Ranking B A/B Testing ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ৔߹ ͕ࠩग़ʹ͍͘ → ̍ ̍ ̍ ̍ ̍
  13. ©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮ཰తͳཧ༝ Interleaving ̍ Ranking A ͱ Ranking

    B͕҉ʹ௚઀ൺֱ͞ΕΔͷͰ ͕ࠩग़΍͍͢ʂʂ → ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍
  14. ©2018 Wantedly, Inc. ࣮ࡍͷABςετͱInterleavingͷp-valueͷऩଋͷൺֱ (Wantedly Visit಺ͷϥϯΩϯάվળʹͯܭଌ)

  15. ©2018 Wantedly, Inc. ͦͷ݁Ռ…

  16. ©2018 Wantedly, Inc. ৽ଔೖࣾ1ϲ݄൒ͰWantedly Visitͷ ৽͍͠ϥϯΩϯάࢪࡦΛຊ൪ϦϦʔε (ࠓͰ΋ܧଓͯ͠ྑ͍݁ՌΛग़͠ଓ͚͍ͯ·͢)

  17. ©2018 Wantedly, Inc. InterleavingΛ׆༻ͯ͠ര଎ϥϯΩϯάվળΛ