Tweet
Tweet

Slide 1

Slide 1 text

©2019 Wantedly, Inc. ϥϯΩϯάΞϧΰϦζϜʹ͓͚Δ ޮ཰తͳΦϯϥΠϯධՁख๏ ΠϯλʔϦʔϏϯά (Interleaving) ͷར༻ s-dev talks ʙαʔϏε։ൃษڧձʙʮԾઆͷཱͯํʯ 19.Feb.2019 - Yuya Matsumura - @yu-ya4

Slide 2

Slide 2 text

©2019 Wantedly, Inc. ✓ Yuya Matsumura (দଜ ༏໵) ✓ Software Engineer (ݕࡧɾਪનΤϯδχΞ) ✓ Wantedly, Inc. Recommendation Team (since April 2018) ✓ Interested in Information Retrieval, Machine Learning Self-Introduction @yu-ya4 @yu__ya4

Slide 3

Slide 3 text

©2019 Wantedly, Inc. 㾎࿩͢͜ͱ w ϥϯΩϯάΞϧΰϦζϜʹ͓͚ΔΦϯϥΠϯධՁख๏ͷ֓ཁΛ ௚ײతʹཧղͰ͖ΔΑ͏౒Ίͨઆ໌ 㾎࿩͞ͳ͍͜ͱ w ਺ֶత౷ܭతͳཧ࿦ͷઆ໌ূ໌ w ۩ମతͳ࣮૷ํ๏ About this talk

Slide 4

Slide 4 text

©2019 Wantedly, Inc. ԾઆΛཱͯΔ Ծઆʹج͍ͮͯ৽͍͠ϥϯΩϯάΞϧΰϦζϜΛߟҊ աڈͷσʔλͳͲʹجͮ͘ΦϑϥΠϯςετʹΑΔݕূ "#ςετͳͲΛ༻͍ͨΦϯϥΠϯςετʹΑΔݕূ ϓϩμΫτͰϦϦʔε ϥϯΩϯάΞϧΰϦζϜมߋͷҰൠతͳྲྀΕ

Slide 5

Slide 5 text

©2019 Wantedly, Inc. ABςετ ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷࢪࡦͷํ͕ ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ̍ ̎ ̍ ̎ Group A Ranking A Control Variation Group B Ranking B

Slide 6

Slide 6 text

©2019 Wantedly, Inc. ABςετͷ՝୊ 㾎༗ҙͳ͕ࠩग़Δ·Ͱʹେྔͷαϯϓϧ਺ JNQSFTTJPO਺ 17਺ ͕ඞཁ 㾎ۃ୺ͳϢʔβͷߦಈʹେ͖͘ӨڹΛड͚ΔՄೳੑ͕͋Δ Ex. *ݱࡏCVR 1%ͷ΋ͷΛ10%վળ(CVR 1.1%)͢Δʹ͸134,379΋ͷαϯϓϧ਺͕ඞཁ Ex. Ұํͷάϧʔϓʹɼ͢΂ͯͷίϯςϯπʹίϯόʔδϣϯ͢ΔϢʔβ͕͍ͨΒ…ʁ

Slide 7

Slide 7 text

©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά(Interleaving) 2ͭͷϥϯΩϯά͔ΒަޓʹΞΠςϜΛબ୒͠৽ͨͳϥϯΩϯάΛ ੜ੒ͯ͠ɼಉҰͷϢʔβ܈ʹఏࣔͯ͠ੑೳΛධՁ͢Δ ̍ ̎ ̍ ̎ Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C

Slide 8

Slide 8 text

©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯάͷੑೳ ࣮ݧతʹɺഒ͔Βഒैདྷͷ"#ςετΑΓ΋ޮ཰తͰ͋Δ͜ͱ͕ ஌ΒΕ͍ͯΔ *ਖ਼֬ʹ͸ʮΠϯλʔϦʔϏϯά͸A/BςετΑΓ100ഒޮ཰తͳ৔߹͕͋Γɺ͓͓Αͦͷ৔߹ɺ 10~100ഒޮ཰తͰɺA/BςετΑΓޮ཰తͰͳ͍৔߹͸গͳ͍ʯ(Chapelle et al. 2012) (Wantedly Visit಺ͷϥϯΩϯάվળʹͯܭଌ)

Slide 9

Slide 9 text

©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮ཰తͳཧ༝ 㾎ਓͷϢʔβʹ҉໧తʹෳ਺ͷΞϧΰϦζϜΛධՁͯ͠΋Β͏͜ͱ͕Ͱ͖Δ 㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ 㾎ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢

Slide 10

Slide 10 text

©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮ཰తͳཧ༝ 㾎ਓͷϢʔβʹ҉໧తʹෳ਺ͷΞϧΰϦζϜΛධՁͯ͠΋Β͏͜ͱ͕Ͱ͖Δ 㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ 㾎ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢

Slide 11

Slide 11 text

©2019 Wantedly, Inc. 1 ਓͷϢʔβʹ҉໧తʹෳ਺ͷΞϧΰϦζϜΛධՁͯ͠΋Β͏͜ͱ͕Ͱ͖Δ ̍ ̎ ̍ ̎ Ranking A Ranking B A/B Testing গͳ͍Ϣʔβ਺Ͱෳ਺ͷΞϧΰϦζϜΛൺֱධՁͰ͖Δ → Interleaving ̍ ̎ ̍ ̎

Slide 12

Slide 12 text

©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮ཰తͳཧ༝ 㾎ਓͷϢʔβ͕҉໧తʹෳ਺ͷΞϧΰϦζϜΛධՁͯ͠΋Β͏͜ͱ͕Ͱ͖Δ 㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ 㾎ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢

Slide 13

Slide 13 text

©2019 Wantedly, Inc. ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ ̍ ̎ ̍ ̎ Ranking A Ranking B A/B Testing ̍ ̍ ̍ ̍ ̍ ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ৔߹ ͕ࠩग़ʹ͍͘ →

Slide 14

Slide 14 text

©2019 Wantedly, Inc. ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ Interleaving ̍ ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍ Ranking A ͱ Ranking B͕҉ʹ௚઀ൺֱ͞ΕΔͷͰ ͕ࠩग़΍͍͢ʂʂ →

Slide 15

Slide 15 text

©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮ཰తͳཧ༝ 㾎ਓͷϢʔβʹ҉໧తʹෳ਺ͷΞϧΰϦζϜΛධՁͯ͠΋Β͏͜ͱ͕Ͱ͖Δ 㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒ΂ͯ௚઀ൺֱͰ͖Δ 㾎ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢

Slide 16

Slide 16 text

©2019 Wantedly, Inc. ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢ ̍ ̎ ̍ ̎ Ranking A Ranking B A/B Testing ̍ ̍ ̎ ̍ ಉ͡ΞΠςϜ͸ಉ͘͡Β͍ͷ CVR ʹͳΔ(͸ͣ)ͷͰɼ 2 ͭͷϥϯΩϯάͰ͕ࠩग़ʹ͍͘ → ʹ

Slide 17

Slide 17 text

©2019 Wantedly, Inc. ಉ͡ΞΠςϜ͕͋ͬͯ΋ͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़΍͍͢ Interleaving ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̍ ಉ͡ΞΠςϜͰ΋ΑΓ্Ґʹ͍ͯͨ͠ϥϯΩϯά ΞϧΰϦζϜͷউͪʂʂ →

Slide 18

Slide 18 text

©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯάͷ՝୊ 㾎࣮૷ίετ͕͔͔ΔɽଞͷϏδωεϩδοΫͱͷিಥ 㾎෼͔Δͷ͸͋͘·Ͱ૬ରతͳੑೳͷྑ͠ѱ͠ wΞϧΰϦζϜ୯ମͷੑೳ͸෼͔Βͳ͍ wϢʔβͷϦςϯγϣϯ΍ΞϧΰϦζϜมߋʹ൐͏ଞͷػೳ΁ͷӨڹͳͲ Λଌఆ͠ʹ͍͘Γʹ͍͘

Slide 19

Slide 19 text

©2019 Wantedly, Inc. 2 ஈ֊ͷΦϯϥΠϯධՁϓϩηε ΠϯλʔϦʔϏϯάͰ࣍ධՁΛߦ͍༗ྗͳީิʹߜΓࠐΈɼ "#ςετͰ࣍ධՁΛߦ͏ஈ֊ͷΦϯϥΠϯධՁγεςϜ https://medium.com/netflix-techblog/interleaving-in-online-experiments-at-netflix- a04ee392ec55

Slide 20

Slide 20 text

©2019 Wantedly, Inc. Summary 㾎 ैདྷͷΦϯϥΠϯϥϯΩϯάධՁख๏"#ςετͷ՝୊ w ༗ҙͳ͕ࠩग़Δ·Ͱʹେྔͷαϯϓϧ਺͕ඞཁ 㾎 ޮ཰తͳධՁख๏ΠϯλʔϦʔϏϯάͷ঺հ w ഒ͔Βഒैདྷͷ"#ςετΑΓ΋ޮ཰తͰ͋ΔͱݴΘΕΔ 㾎 ΠϯλʔϦʔϏϯάͷ՝୊ w ͋͘·Ͱ૬ରతͳධՁ͔͠Ͱ͖ͳ͍ 㾎 ͭͷख๏Λ͍͍ͱ͜औΓͨ͠ஈ֊ͷΦϯϥΠϯධՁϓϩηε w ΠϯλʔϦʔϏϯάͰީิΛߜΓɼ"#ςετͰ࠷ऴతͳධՁΛԼ͢