データ活用の際にハマってしまって辛かったこと/tsurami-of-data

3c0649db1ae5ca0ae57c76037243f501?s=47 Yuya Matsumura
July 19, 2019
Technology
1
3.2k

 データ活用の際にハマってしまって辛かったこと/tsurami-of-data

2019年7月19日 データとML周辺エンジニアリングを考える会 #2 (https://data-engineering.connpass.com/event/136756/) でのLT資料です。

5分間の LT です。
データ活用の際にハマってしまって時間を溶かしたツラミ（自業自得）を共有します。誰かがログの設計やデータ活用の際に救われたら嬉しいです。

3c0649db1ae5ca0ae57c76037243f501?s=128

Yuya Matsumura

July 19, 2019
Tweet

Want more?

3c0649db1ae5ca0ae57c76037243f501?s=48 yuya4
3
1.3k
3c0649db1ae5ca0ae57c76037243f501?s=48 yuya4
1
1.9k
3c0649db1ae5ca0ae57c76037243f501?s=48 yuya4
8
5.1k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
13
810
A9a491b0fcbe0fbce3d64063a37add99?s=48 rmw
4
440
766b9746b59e6f09e280cd33cf4ed419?s=48 skipperchong
0
280
61857dafbd287b3027c4dcea9008ad3c?s=48 carmenhchung
8
580
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
9
310
C0390e8b11079f8761c0cd3274ed61cb?s=48 productmarketing
3
350
C35e01544a0dd94a2cf1619ee8a42ebb?s=48 clairelew
4
840
6bb82b13cda86d3b821fa44100335ddf?s=48 thoeni
1
220
B3ace07412a5178ea778e7eec161fc0a?s=48 jcasabona
4
290

Transcript

  1. 1.

    ©2019 Wantedly, Inc. σʔλ׆༻ͷࡍʹϋϚͬͯ͠·ͬͯ ਏ͔ͬͨ͜ͱ σʔλͱMLपลΤϯδχΞϦϯάΛߟ͑Δձ #2 19.July.2019 - Yuya

    Matsumura @yu-ya4
  2. 2.

    ©2019 Wantedly, Inc. ✓ Yuya Matsumura (দଜ ༏໵) ✓ Software

    Engineer (ݕࡧɾਪનΤϯδχΞ) ✓ Wantedly, Inc. Recommendation Team (since April 2018) ✓ Interested in Information Retrieval, Machine Learning Self-Introduction @yu-ya4 @yu__ya4
  3. 3.

    ©2018 Wantedly, Inc.

  4. 4.

    ©2019 Wantedly, Inc. Wantedly ʹ͓͚Δੵۃతͳσʔλ׆༻ 㾎੔උ͞Εͨσʔλج൫ w ͋ΒΏΔϩάσʔλ͸#JH2VFSZʹอଘ w ओཁͳػೳ͝ͱͷϩάʢMPHVTFS@TFBSDI@MPH::::..%%ʣ

    w EBJMZͰͷ3%#ͷεφοϓγϣοτʢSECVTFST::::..%%ʣ w EBJMZͰͷਪન݁ՌʢSFDPNNOFESBOLJOH::::..%%ʣ w ෼ੳ͠΍͍͢Α͏ʹɼલॲཧΛߦ͍ϩάσʔλΛ੔ܗͨ͠தؒςʔϒϧͷ ࡞੒ɾڞ༗
  5. 5.

    ©2019 Wantedly, Inc. Wantedly ʹ͓͚Δੵۃతͳσʔλ׆༻ 㾎σʔλ׆༻ͷຽओԽ w ϏδωεαΠυ΋ؚΊɼࣾһશһ͕#JH2VFSZͰσʔλʹΞΫηεͰ͖Δ؀ڥ w #JH2VFSZͷ࢖͍ํͱυϝΠϯ஌ࣝ֫ಘ͢ΔͨΊͷ৽ਓݚमͷ࣮ࢪ

    w ΞϓϦέʔγϣϯΤϯδχΞ͕ࣗ෼ͨͪͰσʔλ෼ੳΛߦͬͯ՝୊Λ೺Ѳͯ͠ ࢪࡦΛ࣮ࢪ͠ɼͦͷ݁ՌͷධՁ·ͰΛߦ͏ମ੍ IUUQTTQFBLFSEFDLDPNZVZBXBOUFEMZOFXDPNFSUSBJOJOHCJHRVFSZUSBJOJOH
  6. 6.

    ©2019 Wantedly, Inc. σʔλ׆༻ͷࡍʹϋϚͬͯ͠·ͬͯ ਏ͔ͬͨࣄྫΛ͍͔ͭ͘͝঺հ(´ʀТʀʆ)

  7. 7.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → ᶅ recommend.ranking20190718 →ɹ
  8. 8.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά
  9. 9.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά
  10. 10.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά ←͔͠΋΋ͷʹΑͬͯ͸ ɹ2019೥7݄18೔ʹ༻͍ͨϥϯΩϯάΛද͢ ʢೖࣾؒ΋ͳ͍΅͕͘࡞ͬͨɻɻɻʣ
  11. 11.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶃ ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧsuffix ᶃ log.user_search_log20190718 → 2019೥7݄18೔ͷ1೔෼ͷϩά͕ೖͬͨςʔϒϧ ᶄ

    rdb.users20190718 → 2019೥7݄18೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ → 2019೥7݄18೔ʹ࡞੒͞Εͨ2019೥7݄17೔24:00࣌఺ͰͷRDBͷεφοϓγϣοτ ᶅ recommend.ranking20190718 → 2019೥7݄18೔ʹ༻͍ͨϥϯΩϯά → 2019೥7݄18೔·ͰͷσʔλΛݩʹ࡞੒͞Εͨ2019೥7݄19೔ʹ༻͍ͨϥϯΩϯά ͜ͷ͋ͱ໓஡ۤ஡υΩϡϝϯτ࡞ͬͨɻ
  12. 12.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜ ϦΫΤετύϥϝλΈ͍ͨͳ΋ͷΛ json

    ܗࣜͷ STRING Ͱอଘ͍ͯ͠Δ ύϥϝλ͕૿͑ͨΓݮͬͨΓ͢Δ౎౓ɼςʔϒϧͷ Schema Λมߋ͢Δ ඞཁ͕ͳ͍ͷͰϕϯϦ →
  13. 13.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜ • ಛఆͷ

    filter ͕࢖ΘΕͨͱ͖ͷ log ͚ͩूΊͯ෼ੳ͍ͨ͠ʂ • ͍ΖΜͳάϥϑΛ࡞ͬͯ BI πʔϧʹొ࿥ͩʂ • ͜ͷσʔλΛ࢖ͬͯ model Λ࡞੒ʂ
  14. 14.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶄ json ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜʹ຋࿔͞ΕΔDS • ಛఆͷ

    filter ͕࢖ΘΕͨͱ͖ͷ log ͚ͩूΊͯ෼ੳ͍ͨ͠ʂ • ͍ΖΜͳάϥϑΛ࡞ͬͯ BI πʔϧʹొ࿥ͩʂ • ͜ͷσʔλΛ࢖ͬͯ model Λ࡞੒ʂ ύϥϝλ͕૿͑ͨΓݮͬͨΓ͢Δ౎౓ɼ෼ੳ༻ͷΫΤϦ΋BIπʔϧͷ άϥϑ΋ model ΋࢒೦ͳ͜ͱʹ… →
  15. 15.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ null

  16. 16.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ "null"

  17. 17.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ json ܗࣜͷ STRING ͷΧϥϜΛॲཧͯ͠෼ੳ༻ͷதؒςʔϒϧΛ࡞੒ͨ͠ࡍ ʹ͜͏ͳͬͯ͠·͍ͬͯͨɻ "null"

  18. 18.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ null Λআ͍ͨ ARRAY Λ࡞͍ͬͯΖ͍Ζॲཧͨ͠ޙʹɼͦͷ ARRAY ͷཁૉ

    ͷ਺Λ਺͑ΔʢARRAY_LENGTH() ʣͬͯॲཧΛͨ͠ͱ͖ʹ௧͍໨ʹ… "null"
  19. 19.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ json ܗࣜͰ݁ՌΛݟΕ͹ؾ͚ͮͨɻʢແཧͰ͸ʁʣ "null"

  20. 20.

    ©2019 Wantedly, Inc. ਏ͔ͬͨ͜ͱᶅ ৽UIͰ͸վળ͞Εͨ "null"

  21. 21.

    ©2019 Wantedly, Inc. ͓ΘΓ 㾎ਏ͔ͬͨ͜ͱʢࣗۀࣗಘʣ w ҙຯͷҟͳΔ೔෇Λද͢ςʔϒϧTV⒏Y w KTPOܗࣜͰهड़͞Εͨ453*/(ͷΧϥϜʹ຋࿔͞ΕΔ%4 w

    OVMM