データ活用の際にハマってしまって辛かったこと/tsurami-of-data

Transcript

1. ©2019 Wantedly, Inc. σʔλ׆༻ͷࡍʹϋϚͬͯ͠·ͬͯ ਏ͔ͬͨ͜ͱ σʔλͱMLपลΤϯδχΞϦϯάΛߟ͑Δձ #2 19.July.2019 - Yuya

   Matsumura @yu-ya4

2. ©2019 Wantedly, Inc. ✓ Yuya Matsumura (দଜ ༏໵) ✓ Software

   Engineer (ݕࡧɾਪનΤϯδχΞ) ✓ Wantedly, Inc. Recommendation Team (since April 2018) ✓ Interested in Information Retrieval, Machine Learning Self-Introduction @yu-ya4 @yu__ya4

3. ©2018 Wantedly, Inc.

4. ©2019 Wantedly, Inc. Wantedly ʹ͓͚Δੵۃతͳσʔλ׆༻ 㾎੔උ͞Εͨσʔλج൫
   w ͋ΒΏΔϩάσʔλ͸
   #JH2VFSZ
   ʹอଘ
   w ओཁͳػೳ͝ͱͷϩάʢMPHVTFS@TFBSDI@MPH::::..%%ʣ
   

   w EBJMZ
   Ͱͷ
   3%#
   ͷεφοϓγϣοτʢSECVTFST::::..%%ʣ
   w EBJMZ
   Ͱͷਪન݁ՌʢSFDPNNOFESBOLJOH::::..%%ʣ
   w ෼ੳ͠΍͍͢Α͏ʹɼલॲཧΛߦ͍ϩάσʔλΛ੔ܗͨ͠தؒςʔϒϧͷ ࡞੒ɾڞ༗

5. ©2019 Wantedly, Inc. Wantedly ʹ͓͚Δੵۃతͳσʔλ׆༻ 㾎σʔλ׆༻ͷຽओԽ
   w ϏδωεαΠυ΋ؚΊɼࣾһશһ͕
   #JH2VFSZ
   ͰσʔλʹΞΫηεͰ͖Δ؀ڥ
   w #JH2VFSZ
   ͷ࢖͍ํͱυϝΠϯ஌ࣝ֫ಘ͢ΔͨΊͷ৽ਓݚमͷ࣮ࢪ
   

   w ΞϓϦέʔγϣϯΤϯδχΞ͕ࣗ෼ͨͪͰσʔλ෼ੳΛߦͬͯ՝୊Λ೺Ѳͯ͠ ࢪࡦΛ࣮ࢪ͠ɼͦͷ݁ՌͷධՁ·ͰΛߦ͏ମ੍ IUUQTTQFBLFSEFDLDPNZVZBXBOUFEMZOFXDPNFSUSBJOJOHCJHRVFSZUSBJOJOH

6. ©2019 Wantedly, Inc. σʔλ׆༻ͷࡍʹϋϚͬͯ͠·ͬͯ ਏ͔ͬͨࣄྫΛ͍͔ͭ͘͝঺հ(´ʀТʀʆ)

7. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

   rdb.users20190718 → ᶅ recommend.ranking20190718 →ɹ

8. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

   rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά

9. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

   rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά

10. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά ←͔͠΋΋ͷʹΑͬͯ͸ ɹ2019೥7݄18೔ʹ༻͍ͨϥϯΩϯάΛද͢ ʢೖࣾؒ΋ͳ͍΅͕͘࡞ͬͨɻɻɻʣ

11. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά ͜ͷ͋ͱ໓஡ۤ஡υΩϡϝϯτ࡞ͬͨɻ

12. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜ ϦΫΤετύϥϝλΈ͍ͨͳ΋ͷΛ json

    ܗࣜͷ STRING Ͱอଘ͍ͯ͠Δ ύϥϝλ͕૿͑ͨΓݮͬͨΓ͢Δ౎౓ɼςʔϒϧͷ Schema Λมߋ͢Δ ඞཁ͕ͳ͍ͷͰϕϯϦ →

13. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜ • ಛఆͷ

    filter ͕࢖ΘΕͨͱ͖ͷ log ͚ͩूΊͯ෼ੳ͍ͨ͠ʂ • ͍ΖΜͳάϥϑΛ࡞ͬͯ BI πʔϧʹొ࿥ͩʂ • ͜ͷσʔλΛ࢖ͬͯ model Λ࡞੒ʂ

14. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜʹ຋࿔͞ΕΔDS • ಛఆͷ

    filter ͕࢖ΘΕͨͱ͖ͷ log ͚ͩूΊͯ෼ੳ͍ͨ͠ʂ • ͍ΖΜͳάϥϑΛ࡞ͬͯ BI πʔϧʹొ࿥ͩʂ • ͜ͷσʔλΛ࢖ͬͯ model Λ࡞੒ʂ ύϥϝλ͕૿͑ͨΓݮͬͨΓ͢Δ౎౓ɼ෼ੳ༻ͷΫΤϦ΋BIπʔϧͷ άϥϑ΋ model ΋࢒೦ͳ͜ͱʹ… →

15. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ null

16. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ "null"

17. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ json ܗࣜͷ STRING ͷΧϥϜΛॲཧͯ͠෼ੳ༻ͷதؒςʔϒϧΛ࡞੒ͨ͠ࡍ ʹ͜͏ͳͬͯ͠·͍ͬͯͨɻ "null"

18. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ null Λআ͍ͨ ARRAY Λ࡞͍ͬͯΖ͍Ζॲཧͨ͠ޙʹɼͦͷ ARRAY ͷཁૉ

    ͷ਺Λ਺͑ΔʢARRAY_LENGTH() ʣͬͯॲཧΛͨ͠ͱ͖ʹ௧͍໨ʹ… "null"

19. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ json ܗࣜͰ݁ՌΛݟΕ͹ؾ͚ͮͨɻʢແཧͰ͸ʁʣ "null"

20. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ ৽UIͰ͸վળ͞Εͨ "null"

21. ©2019 Wantedly, Inc. ͓ΘΓ 㾎ਏ͔ͬͨ͜ͱʢࣗۀࣗಘʣ
    w ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧTV⒏Y
    w KTPO
    ܗࣜͰهड़͞Εͨ
    453*/(
    ͷΧϥϜʹ຋࿔͞ΕΔ%4
    w

    OVMM