ウォンテッドリーにおける推薦システムのオフライン評価の仕組み

第35回 MLOps 勉強会：https://mlops.connpass.com/event/297976/

ウォンテッドリーでは、多様なユーザーと会社の理想的なマッチングを実現するために、会社訪問アプリ「Wantedly Visit」の推薦システム開発に力を入れています。今回の発表では、推薦チームが取り組んだオフライン評価の仕組み化について紹介します。この仕組みの導入により、推薦施策のリリース頻度や1回あたりの成果量が改善し、より早くユーザーに価値を提供できるようになりました。

Transcript

1. © 2023 Wantedly, Inc.
   ΢ΥϯςουϦʔʹ͓͚ΔਪનγεςϜͷ

   ΦϑϥΠϯධՁͷ࢓૊Έ
   ୈ35ճ MLOps ษڧձ
   Oct. 31 2023 - Shuhei Goda
   

   View Slide

2. © 2023 Wantedly, Inc.
   ͸͡Ίʹ
   

   View Slide

3. © 2023 Wantedly, Inc.
   ໊લɿ
   ߹ా पฏ(Shuhei Goda)
   ॴଐͱ໾ׂɿ
   ● ΢ΥϯςουϦʔגࣜձࣾ (2019ʙ)
   ● ਪનνʔϜ
   ● ML Tech Lead & Product Manager
   Kaggle Tierɿ
   Kaggle Competitions Grandmaster
   ࣗݾ঺հ
   @hakubishin3
   @jy_msc
   

   View Slide

4. © 2023 Wantedly, Inc.
   ਪનνʔϜ͕औΓ૊ΜͩΦϑϥΠϯධՁͷ࢓૊ΈԽʹ͍ͭͯ঺հ
   ● ձࣾ๚໰ΞϓϦʮWantedly VisitʯͰͷਪનγεςϜͷ։ൃ
   
   
   ● ΦϑϥΠϯධՁͷ޻ఔͰ͍͔ͭ͘ͷ՝୊͕ଘࡏ͓ͯ͠Γɺ։ൃޮ཰͕ѱ͍ঢ়ଶʹ
   
   
   ● ΦϑϥΠϯධՁΛ࢓૊ΈԽ͠ɺ̍ࢪࡦ͋ͨΓͷ੒ՌྔͱϦϦʔεස౓͕େ͖͘޲্
   ຊൃදʹ͍ͭͯ
   

   View Slide

5. © 2023 Wantedly, Inc.
   ձࣾɾαʔϏεͷ঺հ
   

   View Slide

6. © 2023 Wantedly,
   Inc.
   ڀۃͷదࡐదॴʹΑΓɺ

   γΰτͰίίϩΦυϧͻͱΛ;΍͢
   © 2023 Wantedly, Inc.
   ࢲͨͪͷϛογϣϯ
   

   View Slide

7. © 2023 Wantedly, Inc.
   iOS, Android and Web
   ؾܰʹձࣾ๚໰
   ϛογϣϯ΍Ձ஋؍΁ͷڞײͰϚονϯά
   ● څ༩΍෱རްੜͳͲͷ৚݅Ͱ͸ͳ͘ɺ૝͍͕͋Ε͹ձࣾͷ
   ن໛ʹͱΒΘΕͳ͍
   ·ͣʮ࿩Λฉ͖ʹߦ͘ʯͱ͍͏৽͍͠ମݧ
   ● ݸਓͱاۀ͕ϑϥοτͳ໨ઢͰग़ձ͑Δ͜ͱͰɺΑΓັྗ
   తͳ৔ॴΛݟ͚ͭΔ͜ͱ͕Մೳʹ
   ձࣾ๚໰ΞϓϦʮWantedly Visitʯ
   

   View Slide

8. © 2023 Wantedly, Inc.
   Wantedly Visit ʹ͓͚Δ ML ׆༻ʹ͍ͭͯ
   

   View Slide

9. © 2023 Wantedly, Inc.
   Ϣʔβʔ͝ͱʹ࠷దԽ͞Εͨืूͷਪન
   ػցֶशΛ׆༻͢Δ͜ͱͰɺ
   

   Ϣʔβʔͷ࢓ࣄɾձࣾʹؔ͢Δᅂ޷Λଊ͑ɺ
   

   Ϣʔβʔͱձࣾͷཧ૝తͳϚονϯάΛ࣮ݱ͢Δ
   ืूҰཡʢϗʔϜը໘ʣ
   

   View Slide

10. © 2023 Wantedly, Inc.
    αʔϏεʹ͓͚Δਪનͷ࢓૊Έ
    ਪન༻
    API
    ਪન
    

    Ϟσϧ
    ਪન
    ݁Ռ
    ϦΫΤετ
    

    ʢuser_id, ΫΤϦύϥϝʔλ, …ʣ
    ਪન݁Ռ ࠓճͷؔ৺͸ίί
    

    View Slide

11. © 2023 Wantedly, Inc.
    ਪનϞσϧͷ։ൃϓϩηε
    αʔϏεϩάΛ༻͍ͨΦϑϥΠϯධՁͷ޻ఔΛઃ͚͍ͯΔ
    ● ΦϯϥΠϯςετʹ͓͍ͯɺϢʔβʔͷମݧΛᆝଛ͢ΔϦεΫΛ཈͑ΔͨΊ
    
    
    ● ߴ଎ʹ࣮ݧΛճ͠ɺنఆͷ඼࣭Λຬͨ͢ਪનϞσϧΛ࠷୹Ͱ࡞ΕΔΑ͏ʹ͢ΔͨΊ
    
    
    ○ ҰൠతʹɺΦϑϥΠϯධՁίετ ʻ ΦϯϥΠϯධՁίετ
    ࣮૷
    ΦϑϥΠϯ
    ධՁ
    ϦϦʔε
    

    ൑அ
    ΦϯϥΠϯ
    

    ςετ
    نఆͷ඼࣭ʹୡ͢Δ·Ͱ
    

    ࣮ݧʢ࣮૷→ΦϑϥΠϯධՁʣΛ܁Γฦ͢
    

    View Slide

12. © 2023 Wantedly, Inc.
    ΦϑϥΠϯධՁʹ·ͭΘΔ՝୊
    

    View Slide

13. © 2023 Wantedly, Inc.
    ਪનࢪࡦʹ͓͚Δ՝୊
    ਪનࢪࡦͷ։ൃޮ཰͕ߴ͘ͳ͍
    ● 1ࢪࡦʹ͔͔Δ͕࣌ؒ௕͍
    
    
    ● 1ࢪࡦ͋ͨΓͷ੒Ռͷظ଴஋͕ߴ͘ͳ͍
    
    
    ○ ۩ମతʹݴ͏ͱɺΦϯϥΠϯςετͷ݁Ռ͕૝ఆ௨ΓʹͳΒͳ͍͜ͱ͕ଟ͍
    
    
    㱺 ͜ΕΒ͸༷ʑͳཁҼʹΑͬͯੜ͍ͯ͡Δɺͦͷ͏ͪ1͕ͭʮΦϑϥΠϯධՁʯ
    

    View Slide

14. © 2023 Wantedly, Inc.
    ΦϑϥΠϯධՁʹ·ͭΘΔ՝୊ᶃ - 1ࢪࡦʹ͔͔Δ͕࣌ؒ௕͍
    ࢪࡦͷͨͼʹΦϑϥΠϯධՁͷ४උʹ͕͔͔࣌ؒΔ
    ● ࣮૷ʹ͕͔͔࣌ؒΔ
    
    
    ○ جຊతʹεΫϥονͰධՁ༻ͷίʔυΛ࣮૷
    
    
    ○ աڈͷධՁ༻ίʔυ͕࢒͍ͬͯͨͱͯ͠΋ɺຊ൪ίʔυ΄Ͳߴ͍඼࣭Ͱ
    

    ॻ͔Ε͍ͯΔΘ͚Ͱ͸ͳ͍ͨΊɺ׬શʹ࢖͍ճ͢͜ͱ͸೉͍͠
    
    
    ● ϨϏϡʔʹ͕͔͔࣌ؒΔ
    
    
    ○ ࣮ߦ؀ڥ͕ϩʔΧϧͰ࠶ݱ͕೉͍͠ʢར༻σʔλ΍؀ڥͷ࠶ݱʹखؒऔΔʣ
    
    
    ○ notebook ͷϨϏϡʔ͕େม
    

    View Slide

15. © 2023 Wantedly, Inc.
    ΦϯϥΠϯධՁͱ͋·Γ૬ؔ͠ͳ͍ΦϑϥΠϯධՁΛ͍ͯ͠Δ
    ● աڈͷࢪࡦ͔Βͷ஌ݟ͕ݸʑਓʹดͯ͡͠·͍΍͘͢҉໧஌ʹ
    
    
    ● ͦͷͨΊɺઃܭͷ΍Γํ΍ߟྀ౓߹͍͸ݸਓʹґଘʢޱ఻΍ϨϏϡʔͰ஌ࣝ఻ൖʣ
    
    
    ● ઃܭऀ΍ϨϏϡΞʔ͸ࢪࡦͷͨͼʹ࣍ͷΑ͏ͳ೰ΈΛ࣋ͭ͜ͱʹ
    
    
    ○ ͲͷΑ͏ʹධՁઃܭΛਐΊΕ͹͍͍ͷ͔෼͔Βͳ͍
    
    
    ○ ධՁʹ͓͍ͯͲ͏͍͏ཁૉΛߟྀ͢Δ΂͖͔෼͔Βͳ͍
    
    
    ○ Ͳ͜ʹམͱ͕݀͋͠Δͷ͔෼͔Βͳ͍
    ΦϑϥΠϯධՁʹ·ͭΘΔ՝୊ᶄ - ΦϯϥΠϯςετͷ݁Ռ͕૝ఆ௨ΓʹͳΒͳ͍
    

    View Slide

16. © 2023 Wantedly, Inc.
    ՝୊ղܾͷͨΊʹऔΓ૊Μͩ͜ͱ
    

    View Slide

17. © 2023 Wantedly, Inc.
    1. ҉໧஌Λ໌֬ʹ͢Δ
    2. ΦϑϥΠϯධՁʹؔ͢ΔυΩϡϝϯτΛ੔උ͠ɺೝࣝΛἧ͑Δ
    3. ϑϨʔϜϫʔΫΛ࡞੒ͯ͠ɺ࣮૷/ϨϏϡʔίετΛԼ͛Δ
    औΓ૊Έͷॱং
    

    View Slide

18. © 2023 Wantedly, Inc.
    ·ͣ͸҉໧஌ͱԽ͍ͯͨ͠΋ͷΛચ͍ग़͢
    ● աڈͷࢪࡦ࣮ࢪऀʹର͢ΔώΞϦϯάΛ࣮ࢪ
    
    
    ● աڈͷࢪࡦͷΦϑϥΠϯධՁͷઃܭ಺༰΍࣮૷ɺධՁ݁ՌΛ΋ͱʹཧղΛਂΊΔ
    
    
    ● Ͳ͏͍ͬͨ͜ͱ͕ߟྀ͞Ε͍͔ͯͨΒ͏·͍͍ͬͯͨ͘ͷ͔ɺ΋͘͠͸
    

    ߟྀ͞Ε͍ͯͳͯ͘͏·͍͔͘ͳ͔ͬͨͷ͔Λ೺Ѳ͢Δ
    औΓ૊Έᶃ҉໧஌Λ໌֬ʹ͢Δ
    

    View Slide

19. © 2023 Wantedly, Inc.
    ྫɿ࣮ࡍʹϢʔβ͕ݟΔ΋ͷͱҟͳΔਪન݁ՌΛධՁͯ͠͠·͍ͬͯΔ
    ● ࣮ࡍʹϢʔβʔ͕αʔϏεͰӾཡ͢Δืूͷฒͼॱ͸ɺਪનϞσϧͷ݁ՌΛ
    

    ͦͷ··ฦ͢ͷͰ͸ͳ͘ɺͦͷ্ʹԿ͔͠ΒͷޙॲཧΛߦͬͨ݁ՌΛฦ͍ͯ͠Δ
    
    
    ○ ৬छ΍࠾༻ܗଶͱ͍ͬͨݕࡧϑΟϧλʔͷઃఆʢϢʔβͷϦΫΤετʹدΔ΋ͷʣ
    
    
    ○ ެ։͕ఀࢭ͞ΕͨืूΛআ֎͢Δॲཧʢਪનؔ܎ͳ͘αʔϏεͱͯ͠ͷ࢓༷ʣ
    औΓ૊Έᶃ҉໧஌Λ໌֬ʹ͢Δ
    

    View Slide

20. © 2023 Wantedly, Inc.
    ྫɿධՁ͢Δ΂͖͜ͱΛਖ਼͘͠ධՁͰ͖ΔϝτϦΫεʹͳ͍ͬͯΔ͔
    ● ྫ͑͹ʮػցֶशʹΑΔݕࡧΩʔϫʔυͷϥϯΩϯάͰશମͷίϯόʔδϣϯͷ
    

    ޲্཰ΛධՁ͢ΔࡍɺΩʔϫʔυ͝ͱʹnDCGΛܭࢉͯ͠ฏۉ஋ΛऔͬͯධՁ͢Δʯ
    

    㱺ຊ౰ʹͦΕͰ͍͍ͷͩΖ͏͔
    
    
    ○ Α͘࢖ΘΕΔΩʔϫʔυͱ͋·Γ࢖ΘΕͳ͍ΩʔϫʔυΛಉ͡ॏΈͰධՁ͢Δ
    

    ͷ͸ద੾ͩΖ͏͔ʁ࣮ࡍͷར༻ঢ়گΛߟྀͨ͠΄͏͕Α͍ͷͰ͸ʁ
    औΓ૊Έᶃ҉໧஌Λ໌֬ʹ͢Δ
    

    View Slide

21. © 2023 Wantedly, Inc.
    ΦϑϥΠϯධՁͰߟྀ͢Δ΂͖؍఺Λ੔ཧ͢Δ
    ● ਫ਼៛ͳγϛϡϨʔλʔΛ࡞Ε͍ͯΔͩΖ͏͔
    
    
    ○ ࣮ࡍʹϢʔβʔ͕ݟΔ΋ͷΛ࠶ݱͰ͖͍ͯΔ͔
    
    
    ○ αʔϏεʹ͓͚ΔϢʔβʔͷߦಈ܏޲Λ࠶ݱͰ͖͍ͯΔ͔
    
    
    ● ϝτϦΫεͷઃܭ͸ద੾͔
    
    
    ○ ຊ౰ʹධՁ͍ͨ͜͠ͱʹରͯۙ͠ࣅͰ͖͍ͯΔ͔
    
    
    ● ධՁର৅͸ద੾͔
    
    
    ○ ධՁͰ͖ͳ͍Ϣʔβʔ/ΞΠςϜ΋ฆΕࠐΜͰ͍ͳ͍͔
    
    
    ○ ٯʹධՁՄೳͳϢʔβʔ/ΞΠςϜΛͲ͔͜ͷஈ֊Ͱམͱ͍ͯ͠ͳ͍͔
    
    
    ○ ൺֱର৅ͱਖ਼͘͠ൺֱͰ͖ΔΑ͏ʹͳ͍ͬͯΔ͔ɺ༗རෆར͸ͳ͍͔
    औΓ૊Έᶃ҉໧஌Λ໌֬ʹ͢Δ
    

    View Slide

22. © 2023 Wantedly, Inc.
    ΦϑϥΠϯධՁͷ΍ΓํΛݟ͑ΔԽ͠ɺνʔϜ಺ͷೝࣝΛἧ͑Δ
    ● Why, What
    
    
    ● ઃܭͰҙࣝ͢Δ΂͖͜ͱ
    
    
    ● ઃܭͷϓϩηε
    
    
    ● ۩ମྫ
    
    
    ● Ξϯνύλʔϯ
    औΓ૊ΈᶄΦϑϥΠϯධՁʹؔ͢ΔυΩϡϝϯτΛ੔උ͠ɺೝࣝΛἧ͑Δ
    

    View Slide

23. © 2023 Wantedly, Inc.
    ਪનϞσϧͷΦϑϥΠϯධՁͷॲཧϑϩʔΛ࡞੒ʢେཻ͖͘౓Ͱఆٛʣ
    औΓ૊ΈᶄΦϑϥΠϯධՁʹؔ͢ΔυΩϡϝϯτΛ੔උ͠ɺೝࣝΛἧ͑Δ
    ਪન݁ՌΛ
    
    
    ࠶ݱ͢Δ
    ਪન݁ՌΛ
    
    
    ධՁ͢Δ
    

    View Slide

24. © 2023 Wantedly, Inc.
    ਪનϞσϧͷΦϑϥΠϯධՁͷॲཧϑϩʔΛ࡞੒ʢࡉཻ͔͍౓Ͱఆٛʣ
    औΓ૊ΈᶄΦϑϥΠϯධՁʹؔ͢ΔυΩϡϝϯτΛ੔උ͠ɺೝࣝΛἧ͑Δ
    ࣮ࡍͷ΋ͷΑΓ؆ૉʹදݱ
    

    View Slide

25. © 2023 Wantedly, Inc.
    ఆٛͨ͠ॲཧϑϩʔͷߏ੒Λ΋ͱʹ࡞੒
    ● ࣮૷ίετΛԼ͛ΒΕΔΑ͏ʹ͢Δ
    
    
    ○ ࢖͍ճͤΔॲཧΛؔ਺Խ
    
    
    ○ ConfigϕʔεͰධՁ࣮૷Ͱ͖ΔΑ͏ʹ
    
    
    ● ϨϏϡʔίετΛԼ͛ΒΕΔΑ͏ʹ͢Δ
    
    
    ○ ࣮ߦ؀ڥΛϩʔΧϧ͔ΒEKS Pod΁
    
    
    ○ GitHubͰPR࡞ͬͯϨϏϡʔ͢Δମ੍
    
    
    ○ ConfigϕʔεͰϨϏϡʔͰ͖ΔΑ͏ʹ
    औΓ૊ΈᶅϑϨʔϜϫʔΫΛ࡞੒ͯ͠ɺ࣮૷/ϨϏϡʔίετΛԼ͛Δ
    μϛʔͷConfig
    

    View Slide

26. © 2023 Wantedly, Inc.
    ݁Ռ͸Ͳ͏ͳͬͨʁ
    

    View Slide

27. © 2023 Wantedly, Inc.
    ਪનࢪࡦͷ։ൃޮ཰͕޲্ 🎉
    ● 1ࢪࡦʹ͔͔Δ͕࣌ؒ୹ॖɿ൒೔ʙ2೔΄Ͳ࡟ݮ
    
    
    ○ ࣮૷ίετͷ࡟ݮ
    
    
    ○ ϨϏϡʔίετͷ࡟ݮ
    
    
    ● ΦϯϥΠϯςετ݁Ռ͕ࠓ·ͰҎ্ʹ૝ఆʹۙ͘ͳͬͨ
    
    
    ○ ΦϯϥΠϯςετͷ༏ઌॱҐ෇͚ΛΑΓਖ਼֬ʹͰ͖ΔΑ͏ʹͳͬͨ
    
    
    ○ ࣍ͷϖʔδͰৄࡉΛهड़
    ਪનࢪࡦʹରͯ͠༗ҙٛͳมԽ
    

    View Slide

28. © 2023 Wantedly, Inc.
    2. ࢓૊ΈΛ࡞Δલޙ
    0%
    0%
    ॎ࣠ɿΦϯϥΠϯϝτϦΫεͷมԽ཰
    
    
    ԣ࣠ɿΦϑϥΠϯϝτϦΫεͷมԽ཰
    ΦϑϥΠϯϝτϦΫεͱΦϯϥΠϯϝτϦΫεͷ૬͕ؔ޲্
    1. ͔ͳΓੲ
    3. ࢓૊Έ͕࢖ΘΕΔ
    Α͏ʹͳͬͨޙ
    +x%
    -x%
    -x%
    +x%
    

    View Slide

29. © 2023 Wantedly, Inc.
    ࢀߟɿBooking.com ͷ৔߹
    3. ࢓૊Έ͕࢖ΘΕΔ
    Α͏ʹͳͬͨޙ
    https://blog.kevinhu.me/2021/04/25/25-Paper-Reading-
    

    Booking.com-Experiences/bernardi2019.pdf
    

    View Slide

30. © 2023 Wantedly, Inc.
    ͍͞͝ʹ
    

    View Slide

31. © 2023 Wantedly, Inc.
    ਪનνʔϜʹ͓͚ΔΦϯϥΠϯςετʹؔ͢ΔऔΓ૊Έ @ MLOpsษڧձ
    

    View Slide

32. © 2023 Wantedly, Inc.
    We are hiring!
    ਓͱձࣾͷཧ૝తͳϚονϯάΛ௥ٻ͢Δ
    

    σʔλαΠΤϯςΟετɾػցֶशΤϯδχΞΛੵۃ࠾༻தʂ
    
    
    ·ͣ͸ΧδϡΞϧʹ͓࿩͠·͠ΐ͏ʂ
    ΧδϡΞϧ໘ஊϖʔδ ߹ాͷ X ΞΧ΢ϯτ
    

    (DM͍͚ͨͩΕ͹ରԠ͠·͢)
    

    View Slide