Slide 1
Slide 1 text
©2019 Wantedly, Inc.
ϥϯΩϯάΞϧΰϦζϜʹ͓͚Δ
ޮతͳΦϯϥΠϯධՁख๏
ΠϯλʔϦʔϏϯά (Interleaving) ͷར༻
s-dev talks ʙαʔϏε։ൃษڧձʙʮԾઆͷཱͯํʯ
19.Feb.2019 - Yuya Matsumura - @yu-ya4
Slide 2
Slide 2 text
©2019 Wantedly, Inc.
✓ Yuya Matsumura (দଜ ༏)
✓ Software Engineer (ݕࡧɾਪનΤϯδχΞ)
✓ Wantedly, Inc. Recommendation Team (since April 2018)
✓ Interested in Information Retrieval, Machine Learning
Self-Introduction
@yu-ya4
@yu__ya4
Slide 3
Slide 3 text
©2019 Wantedly, Inc.
㾎͢͜ͱ
w ϥϯΩϯάΞϧΰϦζϜʹ͓͚ΔΦϯϥΠϯධՁख๏ͷ֓ཁΛ
ײతʹཧղͰ͖ΔΑ͏Ίͨઆ໌
㾎͞ͳ͍͜ͱ
w ֶత౷ܭతͳཧͷઆ໌ূ໌
w ۩ମతͳ࣮ํ๏
About this talk
Slide 4
Slide 4 text
©2019 Wantedly, Inc.
ԾઆΛཱͯΔ
Ծઆʹج͍ͮͯ৽͍͠ϥϯΩϯάΞϧΰϦζϜΛߟҊ
աڈͷσʔλͳͲʹجͮ͘ΦϑϥΠϯςετʹΑΔݕূ
"#ςετͳͲΛ༻͍ͨΦϯϥΠϯςετʹΑΔݕূ
ϓϩμΫτͰϦϦʔε
ϥϯΩϯάΞϧΰϦζϜมߋͷҰൠతͳྲྀΕ
Slide 5
Slide 5 text
©2019 Wantedly, Inc.
ABςετ
͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷࢪࡦͷํ͕
ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ
̍
̎
̍
̎
Group A
Ranking A
Control Variation
Group B Ranking B
Slide 6
Slide 6 text
©2019 Wantedly, Inc.
ABςετͷ՝
㾎༗ҙͳ͕ࠩग़Δ·Ͱʹେྔͷαϯϓϧ JNQSFTTJPO
17
͕ඞཁ
㾎ۃͳϢʔβͷߦಈʹେ͖͘ӨڹΛड͚ΔՄೳੑ͕͋Δ
Ex. *ݱࡏCVR 1%ͷͷΛ10%վળ(CVR 1.1%)͢Δʹ134,379ͷαϯϓϧ͕ඞཁ
Ex. Ұํͷάϧʔϓʹɼͯ͢ͷίϯςϯπʹίϯόʔδϣϯ͢ΔϢʔβ͕͍ͨΒ…ʁ
Slide 7
Slide 7 text
©2019 Wantedly, Inc.
ΠϯλʔϦʔϏϯά(Interleaving)
2ͭͷϥϯΩϯά͔ΒަޓʹΞΠςϜΛબ͠৽ͨͳϥϯΩϯάΛ
ੜͯ͠ɼಉҰͷϢʔβ܈ʹఏࣔͯ͠ੑೳΛධՁ͢Δ
̍
̎
̍
̎
Ranking A Ranking B
̍
̎
̍
̎
Ranking C
Slide 8
Slide 8 text
©2019 Wantedly, Inc.
ΠϯλʔϦʔϏϯάͷੑೳ
࣮ݧతʹɺഒ͔Βഒैདྷͷ"#ςετΑΓޮతͰ͋Δ͜ͱ͕
ΒΕ͍ͯΔ
*ਖ਼֬ʹʮΠϯλʔϦʔϏϯάA/BςετΑΓ100ഒޮతͳ߹͕͋Γɺ͓͓Αͦͷ߹ɺ
10~100ഒޮతͰɺA/BςετΑΓޮతͰͳ͍߹গͳ͍ʯ(Chapelle et al. 2012)
(Wantedly VisitͷϥϯΩϯάվળʹͯܭଌ)
Slide 9
Slide 9 text
©2019 Wantedly, Inc.
ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮతͳཧ༝
㾎ਓͷϢʔβʹ҉తʹෳͷΞϧΰϦζϜΛධՁͯ͠Β͏͜ͱ͕Ͱ͖Δ
㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ
㾎ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢
Slide 10
Slide 10 text
©2019 Wantedly, Inc.
ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮతͳཧ༝
㾎ਓͷϢʔβʹ҉తʹෳͷΞϧΰϦζϜΛධՁͯ͠Β͏͜ͱ͕Ͱ͖Δ
㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ
㾎ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢
Slide 11
Slide 11 text
©2019 Wantedly, Inc.
1 ਓͷϢʔβʹ҉తʹෳͷΞϧΰϦζϜΛධՁͯ͠Β͏͜ͱ͕Ͱ͖Δ
̍
̎
̍
̎
Ranking A Ranking B
A/B Testing
গͳ͍ϢʔβͰෳͷΞϧΰϦζϜΛൺֱධՁͰ͖Δ
→
Interleaving
̍
̎
̍
̎
Slide 12
Slide 12 text
©2019 Wantedly, Inc.
ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮతͳཧ༝
㾎ਓͷϢʔβ͕҉తʹෳͷΞϧΰϦζϜΛධՁͯ͠Β͏͜ͱ͕Ͱ͖Δ
㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ
㾎ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢
Slide 13
Slide 13 text
©2019 Wantedly, Inc.
ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ
̍
̎
̍
̎
Ranking A Ranking B
A/B Testing
̍
̍
̍ ̍
̍
̍
Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ߹
͕ࠩग़ʹ͍͘
→
Slide 14
Slide 14 text
©2019 Wantedly, Inc.
ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ
Interleaving
̍
̍
̍ ̍
̍
̍
̎
̍
̎
̍
Ranking A ͱ Ranking B͕҉ʹൺֱ͞ΕΔͷͰ
͕ࠩग़͍͢ʂʂ
→
Slide 15
Slide 15 text
©2019 Wantedly, Inc.
ΠϯλʔϦʔϏϯά͕ AB ςετΑΓޮతͳཧ༝
㾎ਓͷϢʔβʹ҉తʹෳͷΞϧΰϦζϜΛධՁͯ͠Β͏͜ͱ͕Ͱ͖Δ
㾎ಉ͡Α͏ͳੑೳͷΞΠςϜΛฒͯൺֱͰ͖Δ
㾎ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢
Slide 16
Slide 16 text
©2019 Wantedly, Inc.
ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢
̍
̎
̍
̎
Ranking A Ranking B
A/B Testing
̍
̍
̎
̍
ಉ͡ΞΠςϜಉ͘͡Β͍ͷ CVR ʹͳΔ(ͣ)ͷͰɼ
2 ͭͷϥϯΩϯάͰ͕ࠩग़ʹ͍͘
→
ʹ
Slide 17
Slide 17 text
©2019 Wantedly, Inc.
ಉ͡ΞΠςϜ͕͋ͬͯͲͪΒ͔ʹࢦ໊͞ΕΔͷͰ͕ࠩग़͍͢
Interleaving
̍
̍ ̍
̍
̍
̎
̍
̍
ಉ͡ΞΠςϜͰΑΓ্Ґʹ͍ͯͨ͠ϥϯΩϯά
ΞϧΰϦζϜͷউͪʂʂ
→
Slide 18
Slide 18 text
©2019 Wantedly, Inc.
ΠϯλʔϦʔϏϯάͷ՝
㾎࣮ίετ͕͔͔ΔɽଞͷϏδωεϩδοΫͱͷিಥ
㾎͔Δͷ͋͘·Ͱ૬ରతͳੑೳͷྑ͠ѱ͠
wΞϧΰϦζϜ୯ମͷੑೳ͔Βͳ͍
wϢʔβͷϦςϯγϣϯΞϧΰϦζϜมߋʹ͏ଞͷػೳͷӨڹͳͲ
Λଌఆ͠ʹ͍͘Γʹ͍͘
Slide 19
Slide 19 text
©2019 Wantedly, Inc.
2 ஈ֊ͷΦϯϥΠϯධՁϓϩηε
ΠϯλʔϦʔϏϯάͰ࣍ධՁΛߦ͍༗ྗͳީิʹߜΓࠐΈɼ
"#ςετͰ࣍ධՁΛߦ͏ஈ֊ͷΦϯϥΠϯධՁγεςϜ
https://medium.com/netflix-techblog/interleaving-in-online-experiments-at-netflix-
a04ee392ec55
Slide 20
Slide 20 text
©2019 Wantedly, Inc.
Summary
㾎 ैདྷͷΦϯϥΠϯϥϯΩϯάධՁख๏"#ςετͷ՝
w ༗ҙͳ͕ࠩग़Δ·Ͱʹେྔͷαϯϓϧ͕ඞཁ
㾎 ޮతͳධՁख๏ΠϯλʔϦʔϏϯάͷհ
w ഒ͔Βഒैདྷͷ"#ςετΑΓޮతͰ͋ΔͱݴΘΕΔ
㾎 ΠϯλʔϦʔϏϯάͷ՝
w ͋͘·Ͱ૬ରతͳධՁ͔͠Ͱ͖ͳ͍
㾎 ͭͷख๏Λ͍͍ͱ͜औΓͨ͠ஈ֊ͷΦϯϥΠϯධՁϓϩηε
w ΠϯλʔϦʔϏϯάͰީิΛߜΓɼ"#ςετͰ࠷ऴతͳධՁΛԼ͢