Tweet
Tweet

Slide 1

Slide 1 text

©2018 Wantedly, Inc. ര଎վળʂInterleaving ABςετΑΓޮ཰తͳϥϯΩϯάධՁΛٻΊͯ Webք۾LTձ 6.Jul.2018 - Yuya Matsumura - @yu-ya4

Slide 2

Slide 2 text

©2018 Wantedly, Inc. •Yuya Matsumuraʢদଜ ༏໵ʣ •Wantedly, Inc. (ࣾձਓ4ϲ݄໨) •From Kyoto •Interested in Information Retrieval & Recommendation Self Introduction @yu-ya4

Slide 3

Slide 3 text

©2018 Wantedly, Inc. ABςετͯ͠·͔͢ʁ

Slide 4

Slide 4 text

©2018 Wantedly, Inc. ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷ ࢪࡦͷํ͕ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ABςετͱ͸ WantedlyͰ͸ɼجຊతʹ͢΂ͯͷมߋʹରͯ͠ABςετΛߦ͔ͬͯΒ ͷਖ਼ࣜϦϦʔεΛߦ͍ͬͯ·͢ʂ

Slide 5

Slide 5 text

©2018 Wantedly, Inc. Wantedlyʹ͓͚ΔϥϯΩϯάͷABςετ A/B Testing ̍ ̎ ̍ ̎ Group A Ranking A Control Variation Group B Ranking B

Slide 6

Slide 6 text

©2018 Wantedly, Inc. ༗ҙͳ͕ࠩग़Δ·Ͱʹ ABςετͷ՝୊ →PDCAαΠΫϧΛճ͢ͷ͕஗͘ͳΔʂ Ex. *ݱࡏCVR 1%ͷ΋ͷΛ10%վળ(CVR 1.1%)͢Δʹ͸134,379΋ͷαϯϓϧ਺͕ඞཁ *༗ҙਫ४Ћ=5%, ݕग़ྗ1-β=95%(https://keisan.casio.jp/exec/user/1489063696) େྔͷαϯϓϧ਺(impression਺, PV਺)͕ඞཁ

Slide 7

Slide 7 text

©2018 Wantedly, Inc.

Slide 8

Slide 8 text

©2018 Wantedly, Inc. ͦ͜ͰInterleaving

Slide 9

Slide 9 text

©2018 Wantedly, Inc. 2ͭͷϥϯΩϯά͔Βަޓʹݕࡧ݁ՌΛ࠷্Ґ͔Βॱʹ બ୒͠1ͭͷϥϯΩϯάΛੜ੒ͯ͠Ϣʔβʹఏࣔ͢Δ Interleavingͱ͸ ̍ ̎ ̍ ̎ Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C ᶃ ᶄ ᶅ ᶆ

Slide 10

Slide 10 text

©2018 Wantedly, Inc. Interleavingͷੑೳ →ര଎ͰPDCAαΠΫϧ͕ճͤΔʂ *ਖ਼֬ʹ͸ʮΠϯλʔϦʔϏϯά͸A/BςετΑΓ100ഒޮ཰తͳ৔߹͕͋Γɺ͓͓Αͦͷ৔߹ɺ 10~100ഒޮ཰తͰɺA/BςετΑΓޮ཰తͰͳ͍৔߹͸গͳ͍ʯ(Chapelle et al. 2012) Interleaving͸ ABςετΑΓ10ʙ100ഒޮ཰త*

Slide 11

Slide 11 text

©2018 Wantedly, Inc.

Slide 12

Slide 12 text

©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮ཰తͳཧ༝ ̍ ̎ ̍ ̎ Ranking A Ranking B A/B Testing ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ৔߹ ͕ࠩग़ʹ͍͘ → ̍ ̍ ̍ ̍ ̍

Slide 13

Slide 13 text

©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮ཰తͳཧ༝ Interleaving ̍ Ranking A ͱ Ranking B͕҉ʹ௚઀ൺֱ͞ΕΔͷͰ ͕ࠩग़΍͍͢ʂʂ → ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍

Slide 14

Slide 14 text

©2018 Wantedly, Inc. ࣮ࡍͷABςετͱInterleavingͷp-valueͷऩଋͷൺֱ (Wantedly Visit಺ͷϥϯΩϯάվળʹͯܭଌ)

Slide 15

Slide 15 text

©2018 Wantedly, Inc. ͦͷ݁Ռ…

Slide 16

Slide 16 text

©2018 Wantedly, Inc. ৽ଔೖࣾ1ϲ݄൒ͰWantedly Visitͷ ৽͍͠ϥϯΩϯάࢪࡦΛຊ൪ϦϦʔε (ࠓͰ΋ܧଓͯ͠ྑ͍݁ՌΛग़͠ଓ͚͍ͯ·͢)

Slide 17

Slide 17 text

©2018 Wantedly, Inc. InterleavingΛ׆༻ͯ͠ര଎ϥϯΩϯάվળΛ