データ活用の際にハマってしまって辛かったこと/tsurami-of-data

2019年7月19日 データとML周辺エンジニアリングを考える会 #2 (https://data-engineering.connpass.com/event/136756/) でのLT資料です。

5分間の LT です。
データ活用の際にハマってしまって時間を溶かしたツラミ（自業自得）を共有します。誰かがログの設計やデータ活用の際に救われたら嬉しいです。

Transcript

1. ©2019 Wantedly, Inc. σʔλ׆༻ͷࡍʹϋϚͬͯ͠·ͬͯ ਏ͔ͬͨ͜ͱ σʔλͱMLपลΤϯδχΞϦϯάΛߟ͑Δձ #2 19.July.2019 - Yuya Matsumura @yu-ya4
2. ©2019 Wantedly, Inc. ✓ Yuya Matsumura (দଜ ༏໵) ✓ Software Engineer (ݕࡧɾਪનΤϯδχΞ) ✓ Wantedly, Inc. Recommendation Team (since April 2018) ✓ Interested in Information Retrieval, Machine Learning Self-Introduction @yu-ya4 @yu__ya4
3. ©2018 Wantedly, Inc.
4. ©2019 Wantedly, Inc. Wantedly ʹ͓͚Δੵۃతͳσʔλ׆༻ 㾎੔උ͞Εͨσʔλج൫
   w ͋ΒΏΔϩάσʔλ͸
   #JH2VFSZ
   ʹอଘ
   w ओཁͳػೳ͝ͱͷϩάʢMPHVTFS@TFBSDI@MPH::::..%%ʣ
   w EBJMZ
   Ͱͷ
   3%#
   ͷεφοϓγϣοτʢSECVTFST::::..%%ʣ
   w EBJMZ
   Ͱͷਪન݁ՌʢSFDPNNOFESBOLJOH::::..%%ʣ
   w ෼ੳ͠΍͍͢Α͏ʹɼલॲཧΛߦ͍ϩάσʔλΛ੔ܗͨ͠தؒςʔϒϧͷ ࡞੒ɾڞ༗
5. ©2019 Wantedly, Inc. Wantedly ʹ͓͚Δੵۃతͳσʔλ׆༻ 㾎σʔλ׆༻ͷຽओԽ
   w ϏδωεαΠυ΋ؚΊɼࣾһશһ͕
   #JH2VFSZ
   ͰσʔλʹΞΫηεͰ͖Δ؀ڥ
   w #JH2VFSZ
   ͷ࢖͍ํͱυϝΠϯ஌ࣝ֫ಘ͢ΔͨΊͷ৽ਓݚमͷ࣮ࢪ
   w ΞϓϦέʔγϣϯΤϯδχΞ͕ࣗ෼ͨͪͰσʔλ෼ੳΛߦͬͯ՝୊Λ೺Ѳͯ͠ ࢪࡦΛ࣮ࢪ͠ɼͦͷ݁ՌͷධՁ·ͰΛߦ͏ମ੍ IUUQTTQFBLFSEFDLDPNZVZBXBOUFEMZOFXDPNFSUSBJOJOHCJHRVFSZUSBJOJOH
6. ©2019 Wantedly, Inc. σʔλ׆༻ͷࡍʹϋϚͬͯ͠·ͬͯ ਏ͔ͬͨࣄྫΛ͍͔ͭ͘͝঺հ(´ʀТʀʆ)
7. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ rdb.users20190718 → ᶅ recommend.ranking20190718 →ɹ
8. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά
9. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά
10. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά ←͔͠΋΋ͷʹΑͬͯ͸ ɹ2019೥7݄18೔ʹ༻͍ͨϥϯΩϯάΛද͢ ʢೖࣾؒ΋ͳ͍΅͕͘࡞ͬͨɻɻɻʣ
11. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά ͜ͷ͋ͱ໓஡ۤ஡υΩϡϝϯτ࡞ͬͨɻ
12. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜ ϦΫΤετύϥϝλΈ͍ͨͳ΋ͷΛ json ܗࣜͷ STRING Ͱอଘ͍ͯ͠Δ ύϥϝλ͕૿͑ͨΓݮͬͨΓ͢Δ౎౓ɼςʔϒϧͷ Schema Λมߋ͢Δ ඞཁ͕ͳ͍ͷͰϕϯϦ →
13. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜ • ಛఆͷ filter ͕࢖ΘΕͨͱ͖ͷ log ͚ͩूΊͯ෼ੳ͍ͨ͠ʂ • ͍ΖΜͳάϥϑΛ࡞ͬͯ BI πʔϧʹొ࿥ͩʂ • ͜ͷσʔλΛ࢖ͬͯ model Λ࡞੒ʂ
14. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜʹ຋࿔͞ΕΔDS • ಛఆͷ filter ͕࢖ΘΕͨͱ͖ͷ log ͚ͩूΊͯ෼ੳ͍ͨ͠ʂ • ͍ΖΜͳάϥϑΛ࡞ͬͯ BI πʔϧʹొ࿥ͩʂ • ͜ͷσʔλΛ࢖ͬͯ model Λ࡞੒ʂ ύϥϝλ͕૿͑ͨΓݮͬͨΓ͢Δ౎౓ɼ෼ੳ༻ͷΫΤϦ΋BIπʔϧͷ άϥϑ΋ model ΋࢒೦ͳ͜ͱʹ… →
15. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ null
16. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ "null"
17. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ json ܗࣜͷ STRING ͷΧϥϜΛॲཧͯ͠෼ੳ༻ͷதؒςʔϒϧΛ࡞੒ͨ͠ࡍ ʹ͜͏ͳͬͯ͠·͍ͬͯͨɻ "null"
18. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ null Λআ͍ͨ ARRAY Λ࡞͍ͬͯΖ͍Ζॲཧͨ͠ޙʹɼͦͷ ARRAY ͷཁૉ ͷ਺Λ਺͑ΔʢARRAY_LENGTH() ʣͬͯॲཧΛͨ͠ͱ͖ʹ௧͍໨ʹ… "null"
19. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ json ܗࣜͰ݁ՌΛݟΕ͹ؾ͚ͮͨɻʢແཧͰ͸ʁʣ "null"
20. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ ৽UIͰ͸վળ͞Εͨ "null"
21. ©2019 Wantedly, Inc. ͓ΘΓ 㾎ਏ͔ͬͨ͜ͱʢࣗۀࣗಘʣ
    w ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧTV⒏Y
    w KTPO
    ܗࣜͰهड़͞Εͨ
    453*/(
    ͷΧϥϜʹ຋࿔͞ΕΔ%4
    w OVMM