データ活用の際にハマってしまって辛かったこと/tsurami-of-data

 データ活用の際にハマってしまって辛かったこと/tsurami-of-data

2019年7月19日 データとML周辺エンジニアリングを考える会 #2 (https://data-engineering.connpass.com/event/136756/) でのLT資料です。

5分間の LT です。
データ活用の際にハマってしまって時間を溶かしたツラミ(自業自得)を共有します。誰かがログの設計やデータ活用の際に救われたら嬉しいです。

3c0649db1ae5ca0ae57c76037243f501?s=128

Yuya Matsumura

July 19, 2019
Tweet

Transcript

  1. ©2019 Wantedly, Inc. σʔλ׆༻ͷࡍʹϋϚͬͯ͠·ͬͯ ਏ͔ͬͨ͜ͱ σʔλͱMLपลΤϯδχΞϦϯάΛߟ͑Δձ #2 19.July.2019 - Yuya

    Matsumura @yu-ya4
  2. ©2019 Wantedly, Inc. ✓ Yuya Matsumura (দଜ ༏໵) ✓ Software

    Engineer (ݕࡧɾਪનΤϯδχΞ) ✓ Wantedly, Inc. Recommendation Team (since April 2018) ✓ Interested in Information Retrieval, Machine Learning Self-Introduction @yu-ya4 @yu__ya4
  3. ©2018 Wantedly, Inc.

  4. ©2019 Wantedly, Inc. Wantedly ʹ͓͚Δੵۃతͳσʔλ׆༻ 㾎੔උ͞Εͨσʔλج൫ w ͋ΒΏΔϩάσʔλ͸#JH2VFSZʹอଘ w ओཁͳػೳ͝ͱͷϩάʢMPHVTFS@TFBSDI@MPH::::..%%ʣ

    w EBJMZͰͷ3%#ͷεφοϓγϣοτʢSECVTFST::::..%%ʣ w EBJMZͰͷਪન݁ՌʢSFDPNNOFESBOLJOH::::..%%ʣ w ෼ੳ͠΍͍͢Α͏ʹɼલॲཧΛߦ͍ϩάσʔλΛ੔ܗͨ͠தؒςʔϒϧͷ ࡞੒ɾڞ༗
  5. ©2019 Wantedly, Inc. Wantedly ʹ͓͚Δੵۃతͳσʔλ׆༻ 㾎σʔλ׆༻ͷຽओԽ w ϏδωεαΠυ΋ؚΊɼࣾһશһ͕#JH2VFSZͰσʔλʹΞΫηεͰ͖Δ؀ڥ w #JH2VFSZͷ࢖͍ํͱυϝΠϯ஌ࣝ֫ಘ͢ΔͨΊͷ৽ਓݚमͷ࣮ࢪ

    w ΞϓϦέʔγϣϯΤϯδχΞ͕ࣗ෼ͨͪͰσʔλ෼ੳΛߦͬͯ՝୊Λ೺Ѳͯ͠ ࢪࡦΛ࣮ࢪ͠ɼͦͷ݁ՌͷධՁ·ͰΛߦ͏ମ੍ IUUQTTQFBLFSEFDLDPNZVZBXBOUFEMZOFXDPNFSUSBJOJOHCJHRVFSZUSBJOJOH
  6. ©2019 Wantedly, Inc. σʔλ׆༻ͷࡍʹϋϚͬͯ͠·ͬͯ ਏ͔ͬͨࣄྫΛ͍͔ͭ͘͝঺հ(´ʀТʀʆ)

  7. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → ᶅ recommend.ranking20190718 →ɹ
  8. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά
  9. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά
  10. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά ←͔͠΋΋ͷʹΑͬͯ͸ ɹ2019೥7݄18೔ʹ༻͍ͨϥϯΩϯάΛද͢ ʢೖࣾؒ΋ͳ͍΅͕͘࡞ͬͨɻɻɻʣ
  11. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά ͜ͷ͋ͱ໓஡ۤ஡υΩϡϝϯτ࡞ͬͨɻ
  12. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜ ϦΫΤετύϥϝλΈ͍ͨͳ΋ͷΛ json

    ܗࣜͷ STRING Ͱอଘ͍ͯ͠Δ ύϥϝλ͕૿͑ͨΓݮͬͨΓ͢Δ౎౓ɼςʔϒϧͷ Schema Λมߋ͢Δ ඞཁ͕ͳ͍ͷͰϕϯϦ →
  13. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜ • ಛఆͷ

    filter ͕࢖ΘΕͨͱ͖ͷ log ͚ͩूΊͯ෼ੳ͍ͨ͠ʂ • ͍ΖΜͳάϥϑΛ࡞ͬͯ BI πʔϧʹొ࿥ͩʂ • ͜ͷσʔλΛ࢖ͬͯ model Λ࡞੒ʂ
  14. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜʹ຋࿔͞ΕΔDS • ಛఆͷ

    filter ͕࢖ΘΕͨͱ͖ͷ log ͚ͩूΊͯ෼ੳ͍ͨ͠ʂ • ͍ΖΜͳάϥϑΛ࡞ͬͯ BI πʔϧʹొ࿥ͩʂ • ͜ͷσʔλΛ࢖ͬͯ model Λ࡞੒ʂ ύϥϝλ͕૿͑ͨΓݮͬͨΓ͢Δ౎౓ɼ෼ੳ༻ͷΫΤϦ΋BIπʔϧͷ άϥϑ΋ model ΋࢒೦ͳ͜ͱʹ… →
  15. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ null

  16. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ "null"

  17. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ json ܗࣜͷ STRING ͷΧϥϜΛॲཧͯ͠෼ੳ༻ͷதؒςʔϒϧΛ࡞੒ͨ͠ࡍ ʹ͜͏ͳͬͯ͠·͍ͬͯͨɻ "null"

  18. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ null Λআ͍ͨ ARRAY Λ࡞͍ͬͯΖ͍Ζॲཧͨ͠ޙʹɼͦͷ ARRAY ͷཁૉ

    ͷ਺Λ਺͑ΔʢARRAY_LENGTH() ʣͬͯॲཧΛͨ͠ͱ͖ʹ௧͍໨ʹ… "null"
  19. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ json ܗࣜͰ݁ՌΛݟΕ͹ؾ͚ͮͨɻʢແཧͰ͸ʁʣ "null"

  20. ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ ৽UIͰ͸վળ͞Εͨ "null"

  21. ©2019 Wantedly, Inc. ͓ΘΓ 㾎ਏ͔ͬͨ͜ͱʢࣗۀࣗಘʣ w ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧTV⒏Y w KTPOܗࣜͰهड़͞Εͨ453*/(ͷΧϥϜʹ຋࿔͞ΕΔ%4 w

    OVMM