BigQuery as a Recommendation System

BigQuery as a Recommendation System

9/11(火) Data Driven Developer Meetup #1 (https://d3m.connpass.com/event/98934/)での発表内容です
BigQuery を活用した Wantedly Visit における Recommendation System の事例についてお話させていただきました.

3c0649db1ae5ca0ae57c76037243f501?s=128

Yuya Matsumura

September 11, 2018
Tweet

Transcript

  1. ©2018 Wantedly, Inc. BigQuery as a Recommendation System BigQuery Λ׆༻ͨ͠ਪનγεςϜࣄྫ(In

    Wantedly Visit) Data Driven Developer Meetup #1 11.Sep.2018 - Yuya Matsumura - @yu-ya4
  2. ©2018 Wantedly, Inc. •Yuya Matsumuraʢদଜ ༏໵ʣ •Wantedly, Inc. (ࣾձਓ6ϲ݄໨) •Recommendation

    Engineer •From Kyoto Self Introduction @yu-ya4 @yu__ya4 https://www.wantedly.com/users/2390451
  3. ©2018 Wantedly, Inc.  ਪનγεςϜͱ͸  8IZVTF#JH2VFSZ   )PXVTF#JH2VFSZ

    Agenda ※ຊൃද͸ɼ2018೥9݄ݱࡏͷWantedly Visit ʹ͓͚ΔਪનγεςϜͷ։ൃϑϩʔɼӡ༻ํ๏ʹج͓ͮ͘࿩Ͱ͢ɽ ಉ͡࿩͕ଞͷόοΫάϥ΢ϯυͰ΋ඞͣਖ਼͍͠ͱ͸ݶΓ·ͤΜɽ
  4. ©2018 Wantedly, Inc.  ਪનγεςϜͱ͸  8IZVTF#JH2VFSZ   )PXVTF#JH2VFSZ

    Agenda
  5. ©2018 Wantedly, Inc. It is often necessary to make choices

    without sufficient personal experience of the alternatives. In everyday life, we rely on recommendations from other people either by word of mouth, recommendation letters, movie and book reviews printed in newspapers, or general surveys such as Zagat’s restaurant guides. Recommender systems assist and augment this natural social process. ਪનγεςϜͱ͸ [Resnick+ 97] ࣗ෼ͷܦݧ͔ΒͷΈͰ͸͋·ΓΑ͘Θ͔Βͳ͍΋ͷͷத͔ΒɼͲ͏ ͯ͠΋ͲΕ͔Λબ͹ͳ͚Ε͹ͳΒͳ͍ͱ͍͏͜ͱ͸Α͋͘Δɽ͜ͷ Α͏ͳࡍΘͨͨͪ͠͸ɼޱίϛɼਪનঢ়ɼ ৽ฉͷॻධ΍өըධɼϨ ετϥϯΨΠυͳͲͷଞਓ͔ΒͷਪનʹཔΔ͜ͱΛ ೔ৗతʹߦͬͯ ͍Δɽ ৘ใਪનγεςϜͱ͸ɼ͜͏ͨࣾ͠ձͰී௨ʹߦΘΕ͍ͯΔ Ұ࿈ͷߦҝΛิॿͨ͠Γɼଅਐͨ͠Γ͢Δ΋ͷͰ͋Δɽ [Resnick 97] P. Resnick and H. R. Varian. Recommender systems. Communications of the ACM, Vol. 40, No. 3, pp. 56–58, 1997.
  6. ©2018 Wantedly, Inc. Recommenders: Tools to help identify worthwhile stuff

    ਪનγεςϜͱ͸ [Konstan+ 03] ਪનγεςϜ: Ձ஋ͷ͋Δ΋ͷ͕ͲΕ͔ಛఆ͢Δͷʹ໾ཱͭಓ۩ [Konstan 03] J. A. Konstan and J. Riedl. Recommender systems: Collaborating in commerce and communities. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, Tutorial, 2003.
  7. ©2018 Wantedly, Inc. ৘ใਪનͷྫ ಛఆͷϢʔβͷᅂ޷ʹ ߹ͬͨ঎඼ͷਪન

  8. ©2018 Wantedly, Inc. ৘ใਪનͷྫ ྨࣅɾؔ࿈঎඼ ͷਪન (amazon.co.jp) ଞͷϢʔβʹΑΔධՁ

  9. ©2018 Wantedly, Inc. ৘ใਪનͷྫ ਓؾ঎඼ͷਪન (amazon.co.jp) ৽ண঎඼ͷਪન

  10. ©2018 Wantedly, Inc. Everything is a Recommendation

  11. ©2018 Wantedly, Inc.  ਪનγεςϜͱ͸  8IZVTF#JH2VFSZ   )PXVTF#JH2VFSZ

    Agenda
  12. ©2018 Wantedly, Inc. ͱʹ͔͘ૣ͍ Why use BigQuery?

  13. ©2018 Wantedly, Inc. ݱࡏͷWantedly Visit ͷ Recommendation TeamʹٻΊΒΕΔ͜ͱ վળαΠΫϧΛૉૣͨ͘͘͞Μճ͢͜ͱ

  14. ©2018 Wantedly, Inc. ݱࡏͷWantedly Visit ͷ Recommendation ࣄ৘  γϯϓϧͳมߋͰվળͰ͖ͦ͏ͳՕॴ͕ෳ਺࢒͍ͬͯΔ

     νʔϜ͕খن໛ ࢲΛؚΊ໊ͯ   αʔϏε͸খن໛͡Όͳ͍ ͨ͘͞ΜͷϢʔβ ͭͭͷվળʹΊͪΌͪ͘Ό࣌ؒΛ͔͚͍͍ͯΘ͚Ͱ͸ͳ͍
  15. ©2018 Wantedly, Inc. ͨͱ͑͹… Ϣʔβͷืू΁ͷΫϦοΫ৘ใΛར༻ͯ͠ɼ ͋ΔϢʔβʹืूΛਪન͍ͨ͠ ⭕ ⭕ ⭕ ⭕

    ⭕ ⭕ ⭕ ʁ ⭕ - - - - - - - ืू Ϣʔβ
  16. ©2018 Wantedly, Inc. աڈͷΫϦοΫ৘ใΛجʹͨ͠Ϣʔβؒͷྨࣅ౓Λࢉग़ͯ͠ɼ ࣅ͍ͯΔϢʔβ͕ΫϦοΫͨ͠ืूΛਪન͠Α͏ʂ (ϢʔβϕʔεڠௐϑΟϧλϦϯά) ྨࣅ౓௿ ྨࣅ౓ߴ ⭕ ⭕

    ⭕ ⭕ ⭕ ⭕ ⭕ ʁ ⭕ - - - - - - - ਪન ืू Ϣʔβ ͨͱ͑͹…
  17. ©2018 Wantedly, Inc. ୯७ͦ͏Ͱߟ͑Δ͜ͱ/΍Δ͜ͱ͸ͨ͘͞Μ… վળαΠΫϧ͕ͳ͔ͳ͔ճͤͳ͍ ࣮ߦ଎౓͸໰୊ͳ͍ʁ σʔλྔ͍͢͝ ΦϑϥΠϯධՁ͸Ͳ͏͢Δʁ ΦϯϥΠϯධՁ͸ʁ ࣮૷ݴޠ͸ʁ

    ෼ࢄॲཧͤ͞Δʁ σʔλͷύΠϓϥΠϯઃܭ͸ʁ ϚγϯϦιʔε଍ΓͯΔʁ σʔλͷݕূ͸Ͳ͏͢Δʁ
  18. ©2018 Wantedly, Inc.

  19. ©2018 Wantedly, Inc. BigQueryͰશ෦΍ͬͪΌ͑

  20. ©2018 Wantedly, Inc.  ਪનγεςϜͱ͸  8IZVTF#JH2VFSZ   )PXVTF#JH2VFSZ

    Agenda
  21. ©2018 Wantedly, Inc. Overview of architecture of Recommendation System in

    Wantedly Visit BigQuery merge Job ͷهड़(΄΅SQL) Scheduler ʹొ࿥ ΫΤϦͷ࣮ߦ ࣮ߦ݁ՌΛRDBʹอଘ Ωϟογϡͱͯ͠ར༻
  22. ©2018 Wantedly, Inc. BigQuery by Ruby

  23. ©2018 Wantedly, Inc. Jobͷఆٛ

  24. ©2018 Wantedly, Inc. pros. w 42-͚ͩॻ͚Ε͹͍͍ͷͰɼ࣮૷ίετ͕௿͍ɽ w ແݶͷϦιʔε w (PPHMFઌੜʹ՝ۚ͢Δ͚ͩͰര଎ʂ

    w #JH2VFSZ಺Ͱॲཧ͕׬݁ w σʔλͷҠಈߟ͑Δඞཁͳ͠ w %PNP #*πʔϧ ্Ͱεέδϡʔϥ͕࣮ߦՄೳ w "#ςετͳͲͷΦϯϥΠϯධՁΛܧଓతʹ௥͍΍͍͢
  25. ©2018 Wantedly, Inc. cons. w ΫΤϦ͕ଐਓԽ͠΍͍͢ w ਓͷॻ͍ͨΫΤϦಡΉͷŨŽŘ w σόοά͠ʹ͍͘

    w ຊ౰ʹਖ਼͍͠ग़ྗʁ w ෳࡶա͗Δॲཧ͸͠ΜͲ͍ w ;ͱͨ͠ॠؒʹ1ZUIPO͕࿀͘͠ͳΔ ݸਓతͳҙݟ
  26. ©2018 Wantedly, Inc. ·ͱΊ BigQueryųƄƃž