Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
©2019 Wantedly, Inc. ϥϯΩϯάΞϧΰϦζϜʹ͓͚Δ ޮతͳΦϯϥΠϯධՁख๏ ΠϯλʔϦʔϏϯά (Interleaving) ͷར༻ s-dev talks ʙαʔϏε։ൃษڧձʙʮԾઆͷཱͯํʯ 19.Feb.2019 - Yuya Matsumura - @yu-ya4
Slide 2
Slide 2 text
©2019 Wantedly, Inc. ✓ Yuya Matsumura (দଜ ༏) ✓ Software Engineer (ݕࡧɾਪનΤϯδχΞ) ✓ Wantedly, Inc. Recommendation Team (since April 2018) ✓ Interested in Information Retrieval, Machine Learning Self-Introduction @yu-ya4 @yu__ya4
Slide 3
Slide 3 text
©2019 Wantedly, Inc. 㾎͢͜ͱ w ϥϯΩϯάΞϧΰϦζϜʹ͓͚ΔΦϯϥΠϯධՁख๏ͷ֓ཁΛ ײతʹཧղͰ͖ΔΑ͏Ίͨઆ໌ 㾎͞ͳ͍͜ͱ w ֶత౷ܭతͳཧͷઆ໌ূ໌ w ۩ମతͳ࣮ํ๏ About this talk
Slide 4
Slide 4 text
©2019 Wantedly, Inc. ԾઆΛཱͯΔ Ծઆʹج͍ͮͯ৽͍͠ϥϯΩϯάΞϧΰϦζϜΛߟҊ աڈͷσʔλͳͲʹجͮ͘ΦϑϥΠϯςετʹΑΔݕূ "#ςετͳͲΛ༻͍ͨΦϯϥΠϯςετʹΑΔݕূ ϓϩμΫτͰϦϦʔε ϥϯΩϯάΞϧΰϦζϜมߋͷҰൠతͳྲྀΕ
Slide 5
Slide 5 text
©2019 Wantedly, Inc. ABςετ ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷࢪࡦͷํ͕ ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ̍ ̎ ̍ ̎ Group A Ranking A Control Variation Group B Ranking B
Slide 6
Slide 6 text
©2019 Wantedly, Inc. ABςετͷ՝ 㾎༗ҙͳ͕ࠩग़Δ·Ͱʹେྔͷαϯϓϧ JNQSFTTJPO 17 ͕ඞཁ 㾎ۃͳϢʔβͷߦಈʹେ͖͘ӨڹΛड͚ΔՄೳੑ͕͋Δ Ex. *ݱࡏCVR 1%ͷͷΛ10%վળ(CVR 1.1%)͢Δʹ134,379ͷαϯϓϧ͕ඞཁ Ex. Ұํͷάϧʔϓʹɼͯ͢ͷίϯςϯπʹίϯόʔδϣϯ͢ΔϢʔβ͕͍ͨΒ…ʁ
Slide 7
Slide 7 text
©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά(Interleaving) 2ͭͷϥϯΩϯά͔ΒަޓʹΞΠςϜΛબ͠৽ͨͳϥϯΩϯάΛ ੜͯ͠ɼಉҰͷϢʔβ܈ʹఏࣔͯ͠ੑೳΛධՁ͢Δ ̍ ̎ ̍ ̎ Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C
Slide 8
Slide 8 text
©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯάͷੑೳ ࣮ݧతʹɺഒ͔Βഒैདྷͷ"#ςετΑΓޮతͰ͋Δ͜ͱ͕ ΒΕ͍ͯΔ *ਖ਼֬ʹʮΠϯλʔϦʔϏϯάA/BςετΑΓ100ഒޮతͳ߹͕͋Γɺ͓͓Αͦͷ߹ɺ 10~100ഒޮతͰɺA/BςετΑΓޮతͰͳ͍߹গͳ͍ʯ(Chapelle et al. 2012) (Wantedly VisitͷϥϯΩϯάվળʹͯܭଌ)
Slide 9
Slide 9 text
©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮతͳཧ༝ 㾎ਓͷϢʔβʹ҉తʹෳͷΞϧΰϦζϜΛධՁͯ͠Β͏͜ͱ͕Ͱ͖Δ 㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ 㾎ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢
Slide 10
Slide 10 text
©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮతͳཧ༝ 㾎ਓͷϢʔβʹ҉తʹෳͷΞϧΰϦζϜΛධՁͯ͠Β͏͜ͱ͕Ͱ͖Δ 㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ 㾎ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢
Slide 11
Slide 11 text
©2019 Wantedly, Inc. 1 ਓͷϢʔβʹ҉తʹෳͷΞϧΰϦζϜΛධՁͯ͠Β͏͜ͱ͕Ͱ͖Δ ̍ ̎ ̍ ̎ Ranking A Ranking B A/B Testing গͳ͍ϢʔβͰෳͷΞϧΰϦζϜΛൺֱධՁͰ͖Δ → Interleaving ̍ ̎ ̍ ̎
Slide 12
Slide 12 text
©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮతͳཧ༝ 㾎ਓͷϢʔβ͕҉తʹෳͷΞϧΰϦζϜΛධՁͯ͠Β͏͜ͱ͕Ͱ͖Δ 㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ 㾎ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢
Slide 13
Slide 13 text
©2019 Wantedly, Inc. ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ ̍ ̎ ̍ ̎ Ranking A Ranking B A/B Testing ̍ ̍ ̍ ̍ ̍ ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ߹ ͕ࠩग़ʹ͍͘ →
Slide 14
Slide 14 text
©2019 Wantedly, Inc. ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ Interleaving ̍ ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍ Ranking A ͱ Ranking B͕҉ʹൺֱ͞ΕΔͷͰ ͕ࠩग़͍͢ʂʂ →
Slide 15
Slide 15 text
©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮతͳཧ༝ 㾎ਓͷϢʔβʹ҉తʹෳͷΞϧΰϦζϜΛධՁͯ͠Β͏͜ͱ͕Ͱ͖Δ 㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ 㾎ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢
Slide 16
Slide 16 text
©2019 Wantedly, Inc. ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢ ̍ ̎ ̍ ̎ Ranking A Ranking B A/B Testing ̍ ̍ ̎ ̍ ಉ͡ΞΠςϜಉ͘͡Β͍ͷ CVR ʹͳΔ(ͣ)ͷͰɼ 2 ͭͷϥϯΩϯάͰ͕ࠩग़ʹ͍͘ → ʹ
Slide 17
Slide 17 text
©2019 Wantedly, Inc. ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢ Interleaving ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̍ ಉ͡ΞΠςϜͰΑΓ্Ґʹ͍ͯͨ͠ϥϯΩϯά ΞϧΰϦζϜͷউͪʂʂ →
Slide 18
Slide 18 text
©2019 Wantedly, Inc. ΠϯλʔϦʔϏϯάͷ՝ 㾎࣮ίετ͕͔͔ΔɽଞͷϏδωεϩδοΫͱͷিಥ 㾎͔Δͷ͋͘·Ͱ૬ରతͳੑೳͷྑ͠ѱ͠ wΞϧΰϦζϜ୯ମͷੑೳ͔Βͳ͍ wϢʔβͷϦςϯγϣϯΞϧΰϦζϜมߋʹ͏ଞͷػೳͷӨڹͳͲ Λଌఆ͠ʹ͍͘Γʹ͍͘
Slide 19
Slide 19 text
©2019 Wantedly, Inc. 2 ஈ֊ͷΦϯϥΠϯධՁϓϩηε ΠϯλʔϦʔϏϯάͰ࣍ධՁΛߦ͍༗ྗͳީิʹߜΓࠐΈɼ "#ςετͰ࣍ධՁΛߦ͏ஈ֊ͷΦϯϥΠϯධՁγεςϜ https://medium.com/netflix-techblog/interleaving-in-online-experiments-at-netflix- a04ee392ec55
Slide 20
Slide 20 text
©2019 Wantedly, Inc. Summary 㾎 ैདྷͷΦϯϥΠϯϥϯΩϯάධՁख๏"#ςετͷ՝ w ༗ҙͳ͕ࠩग़Δ·Ͱʹେྔͷαϯϓϧ͕ඞཁ 㾎 ޮతͳධՁख๏ΠϯλʔϦʔϏϯάͷհ w ഒ͔Βഒैདྷͷ"#ςετΑΓޮతͰ͋ΔͱݴΘΕΔ 㾎 ΠϯλʔϦʔϏϯάͷ՝ w ͋͘·Ͱ૬ରతͳධՁ͔͠Ͱ͖ͳ͍ 㾎 ͭͷख๏Λ͍͍ͱ͜औΓͨ͠ஈ֊ͷΦϯϥΠϯධՁϓϩηε w ΠϯλʔϦʔϏϯάͰީิΛߜΓɼ"#ςετͰ࠷ऴతͳධՁΛԼ͢