レコメンデーションのターゲットメトリックスの設定 / ML loft 3

Wantedly Visitのレコメンデーションシステムの全体像を話をした後、ユーザエンゲージメントを良くするためのターゲットの設定方法についてお話しました。

Transcript

1. ©2019 Wantedly, Inc. Ϩίϝϯσʔγϣϯͷ λʔήοτϝτϦοΫε ML @ Loft #3 Jun 21, 2019 - Rei Kubonaga ϢʔβΤϯήʔδϝϯτΛྑ͘͢ΔͨΊͷλʔήοτઃఆ
2. ©2019 Wantedly, Inc. !LVCPOBHBSFJ
   
   3FJ
   ,VCPOBHB
   w 8BOUFEMZ
   ਓ໨ͷࣾһ
   w 8BOUFEMZ
   7JTJUͷ։ൃνʔϜͷϚωʔδϟʔ
   w ϨίϝϯσʔγϣϯνʔϜϦʔμʔͱͯ͠νʔϜͷ্ཱͪ͛Λߦ͏ ࣗݾ঺հ
3. ©2019 Wantedly, Inc. Wantedly Visit ࢲͨͪͷϓϩμΫτ!ձࣾ๚໰αʔϏε
4. ©2019 Wantedly, Inc. ϨίϝϯσʔγϣϯΛ࢖͍ͬͯΔ৔ॴ Ϣʔβը໘Ͱ͸ɺΞϓϦΛ։͘ͱ࠷ॳʹදࣔ͞ΕΔը໘ͷ ϨίϝϯσʔγϣϯΛ։ൃ͍ͯ͠·͢ɻ ʮ͓͢͢ΊͷืूʯʮϑΟϧλϦϯάʯʮΩʔϫʔυݕࡧʯ ͱͦΕͧΕҟͳΔΞϧΰϦζϜͰ࣮૷͍ͯ͠·͢ɻ ఆ఺؍ଌ͍ͯ͠Δࢦඪͱͯ͠͸ɺ ʮืू͕දࣔ͞Ε͔ͯΒɺ࿩Λฉ͖ʹߦׂ͘߹ʯ ͳͲݟ͍ͯ·͢ɻ ࠷ॳͷ
   ͓͢͢Ίืू ϑΟϧλϦϯ ά Ωʔϫʔυ
   ݕࡧ
5. ©2019 Wantedly, Inc. ࠓ೔࿩͠Λ͢Δ͜ͱ ืूΛද͔ࣔͯ͠Βɺ࿩Λฉ͖ʹߦׂ͘߹ͷਪҠ (2017೥3݄͔Β2019೥5݄ɻ2017೥3݄Λ100%ͱ͍ͯ͠Δ) 2೥લʹɺϨίϝϯσʔγϣϯνʔϜΛൃ଍͠ɺ ػցֶशΛ࢖ͬͨϨίϝϯσʔγϣϯΛ௨ͯ͡ɺ ϢʔβମݧΛେ͖͘վળ͠·ͨ͠ɻ ࣮ࡍɺʮืूΛද͔ࣔͯ͠Βɺ࿩Λฉ͖ʹߦ֬͘཰ʯͱ͍ ͏ॏཁͳࢦඪ͕2ഒҎ্ʹվળ͠·ͨ͠ɻ ೥݄ͱൺֱ ͯ͠ͷվળ /FX
   "MHPSJUIN /FX
   "MHPSJUIN /FX
   "MHPSJUIN /FX
   "MHPSJUIN /FX
   "MHPSJUIN
6. ©2019 Wantedly, Inc. Wantedly VisitͷϨίϝϯσʔγϣϯ .-
   NPEFM Ϣʔβͷߦಈσʔλ Ϣʔβͷσʔλ ΞΠςϜͷσʔλ ݕࡧΫΤϦ Ϣʔβ Τϯήʔδϝϯτ είΞ σʔλ λʔήοτ
7. ©2019 Wantedly, Inc. σʔλλΠϓ Ϣʔβͷߦಈσʔλ ɾInteraction (ΞΠςϜʹରͯ͠ɺӾཡɺΫϦοΫɺϒοΫϚʔΫͳͲ) ɾInteger (ΞΫηε਺, ࠷ऴΞΫηε೔ͳͲ) Ϣʔβ / ΞΠςϜͷσʔλ ɾInteger (બ୒͍ͯ͠Δ৬छɺ೥ྸͷൣғ, ϒοΫϚʔΫ਺ͳͲ) ɾBoolean (Ԡื͔ͨ͠ɺ࠷ۙొ࿥͔ͨ͠ɺֶੜ͔ɺΞϓϦΛ࢖͍ͬͯΔ͔) ɾText (ืूཁ߲ɺࣗݾ঺հɺֶྺɺ৬ྺ) ɾImage (ืूͷը૾ɺϓϩϑΟʔϧը૾)
8. ©2019 Wantedly, Inc. ϢʔβΤϯήʔδϝϯτείΞ αʔϏεʹର͢ΔѪணΛද͢είΞ ɾϢʔβɺΞΠςϜ͝ͱʹɺϢʔβΤϯήʔδϝϯτΛͲͷ͘Β͍ߴΊΒΕΔ͔දͨ͠είΞ Ϣʔβମݧͷ࣭Λఆٛ͢Δࢦඪ ɾΑ͘ΫϦοΫ͞ΕΔɺΑ͘Ԡื͞ΕΔɺ௕͘࢖ͬͯ͘ΕΔɺසൟʹ࢖ͬͯ͘ΕΔͳͲ λʔήοτ͸Τϯήʔδϝϯτʹ૬͕ؔ͋Δ਺ ɾΫϦοΫ͞ΕΔ͜ͱʹΑͬͯɺΤϯήʔδϝϯτ͕ඞͣ͠΋ߴ͘ͳΔΘ͚Ͱ͸ͳ͍
9. ©2019 Wantedly, Inc. Ϩίϝϯσʔγϣϯͷ࣮૷ -JHIU(#.
   // Ϣʔβͷಛ௃ ΞΠςϜͷಛ௃ ΫΤϦͷಛ௃ ૊Έ߹Θͤͨಛ௃ Ϣʔβ Τϯήʔδϝϯτ είΞ (SBQI
   1SPDFTTJOH /-1 Ϣʔβͷߦಈσʔλ Ϣʔβͷଐੑ ΞΠςϜͷଐੑ ݕࡧΫΤϦ Interaction data Text Integer, Boolean Feature Data Target
10. ©2019 Wantedly, Inc. Ϩίϝϯσʔγϣϯʹ͓͚Δλʔήοτ λʔήοτͱ͸ͳʹ͔ ɾϥϯΩϯά͕Ϣʔβʔʹͱͬͯྑ͔͔ͬͨѱ͔͔ͬͨΛ൑அ͢ΔͨΊͷ਺ࣈ λʔήοτͷઃఆͷ೉͠͞ ɾΫϦοΫͯ͠΋ίϯςϯπΛݟͯͳ͍͔΋͠Εͳ͍ ɾϒοΫϚʔΫ΍Ԡืͩͱใु͕ͳׂ͍߹͕૿͑Δ ɾͨ͘͞Μݟ͗͢Δͱ͍͏͜ͱ͸ɺٯʹѱ͍ମݧ͔΋͠Εͳ͍ ɾPCͱΞϓϦͰϢʔβͷಈ͖͸ҟͳΔ VisitͰ͸ɺϢʔβΤϯήʔδϝϯτʹ૬͕ؔ͋Δ਺ࣈΛλʔήοτͱͯ͠ઃఆ
11. ©2019 Wantedly, Inc. Τϯήʔδϝϯτͱ͸ɺstickiness͕ߴ͍͜ͱ /FX
    6TFS "DUJWF
    6TFS %PSNBOU
    6TFS Acquisition Activation Disengagement Disengagement Re-engagement Churn stickiness ͍͔ʹActiveͰ͋Γଓ͚Δ͔ ๚໰ස౓ɺར༻࣌ؒͰ൑அ
12. ©2019 Wantedly, Inc. ΤϯήʔδϝϯτΛܾΊΔํ๏ ΠϯλϏϡʔ
    αʔϕΠ
    ΧελϚʔδϟʔχʔ
    FUD ΦϯϥΠϯ
    ෼ੳ ɾఆྔత ɾεέʔϧͯ͠ࢼͤΔ ɾϢʔβͷແҙࣝͷಈ͖ ɾ٬؍త ɾ୹ظత,௕ظత ɾఆੑ ɾ෦෼తʹߦ͏ ɾ࣮ݧதͷಈ͖ ɾओ؍త,٬؍త ɾ௕ظత
13. ©2019 Wantedly, Inc. ΦϯϥΠϯͷ෼ੳͰߦ͏͜ͱ .FBTVSFNFOU .FUSJD ,1* αʔϏεͷதͰखʹೖΔ ఆྔతͳ஋ ɾӾཡ਺ ɾӾཡ೔ ɾΫϦοΫ਺ Measurement͔Βܭࢉɾ෼ੳ ͠΍͍͢ܗʹ੔ܗͨ͠਺ࣈ ɾΫϦοΫ཰ ɾΞΫςΟϒϢʔβ ɾMRR/AUC/Precision/NDCG Business/Service্ͷ໨ඪʹରͯ͠ ॏཁͳӨڹΛ༩͑Δࢦඪ ɾϢʔβΤϯήʔδϝϯτείΞ ɾاۀͷܧଓ཰ɾARPU ɾ৽نϢʔβͷϦςϯγϣϯ཰ Measurement͔ΒMetricͱKPIΛܭࢉ͢Δ͜ͱ
14. ©2019 Wantedly, Inc. Τϯήʔδϝϯτͷܭࢉྫ .FBTVSFNFOU .FUSJD ,1* ɾ͋ΔΞΠςϜͷӾཡ࣌ؒ ɾ͋ΔΞΠςϜΛදࣔͨ͠ॱ൪ ɾ͋ΔΞΠςϜͷΫϦοΫͨ࣌ؒ͠ ɾ͋ΔΞΠςϜ΁ͷΞΫγϣϯ (ϒοΫϚʔΫɺԠื) ɾηογϣϯͷ࣌ؒ ɾηογϣϯதͷӾཡ਺ͱԠื਺ ɾ࠶๚໰ͨ࣌ؒ͠ ɾΫϦοΫ཰/ϒοΫϚʔΫ཰ ɾԠื཰/࠾༻཰ ɾMRR/AUC/nDCG (্Ґʹදࣔͨ͠΋ͷ͕ ΞΫγϣϯ͞Ε͔ͨ) ɾܧଓ཰ ɾར༻࣌ؒ ɾΞΠςϜΧόϨοδ (ಛఆͷاۀʹूதͯ͠ͳ͍͔) ɾϢʔβΤϯήʔδϝϯτείΞ ෳ਺ͷࢦඪΛ૊Έ߹ΘͤΔ
    ֶशʹ࢖͏Ϣʔβͷݶఆ΋
    ߦͬͨ
15. ©2019 Wantedly, Inc. Τϯήʔδϝϯτͷܭࢉ 5VOJOH &WBMVBUF 0OMJOF
    5FTU Tuning -> Online Test -> EvaluateͷαΠΫϧΛΞϧΰϦζ Ϝ͚ͩͰͳ͘ɺϢʔβΤϯήʔδϝϯτείΞʹ͍ͭͯ΋ߦ ͏ ɾείΞࣗମͷ૊Έ߹Θͤํ͸ແݶɻৗʹਖ਼͍͠ղ͸ͳ͍ ɾΑΓΑ͍਺ࣈ͸ৗʹ͋Γɺඞͣ͠΋ෳࡶʹ͢Ε͹Α͍Θ ͚Ͱ͸ͳ͍ɻ(ΫϦοΫ࣌ؒ => ฏۉ, ۠ؒ, ࿈ଓ) ɾαʔϏεͷϑΣΠζʹΑͬͯ΋࠷దԽ͢Δ΋ͷ͸มΘΔ ɾΦϑϥΠϯͷςετͱΦϯϥΠϯͷςετ͕ಉ݁͡Ռ ͔ɺಛʹҟͳͬͨ࣌ʹཧ༝Λௐ΂Δ
16. ©2019 Wantedly, Inc. • DNNͷར༻ • ਪનཧ༝ͷઆ໌ • όϯσΟοτ • ϑΣΞωε • ਪનʹదͨ͠UI ͜Ε͔Β͍ͨ͜͠ͱ
17. ©2019 Wantedly, Inc. ؔ࿈ͨ͠ϒϩά΋ॻ͍ͯ·͢ https://www.wantedly.com/companies/wantedly/post_articles/155593 https://www.wantedly.com/companies/wantedly/post_articles/169261
18. ©2019 Wantedly, Inc. Mobile FrontendʗBackend Infrastructure New! We are hiring!! https://www.wantedly.com/companies/wantedly