2019年7月19日 データとML周辺エンジニアリングを考える会 #2 (https://data-engineering.connpass.com/event/136756/) でのLT資料です。
5分間の LT です。 データ活用の際にハマってしまって時間を溶かしたツラミ(自業自得)を共有します。誰かがログの設計やデータ活用の際に救われたら嬉しいです。
©2019 Wantedly, Inc.σʔλ׆༻ͷࡍʹϋϚͬͯ͠·ͬͯਏ͔ͬͨ͜ͱσʔλͱMLपลΤϯδχΞϦϯάΛߟ͑Δձ #219.July.2019 - Yuya Matsumura @yu-ya4
View Slide
©2019 Wantedly, Inc.✓ Yuya Matsumura (দଜ ༏)✓ Software Engineer (ݕࡧɾਪનΤϯδχΞ)✓ Wantedly, Inc. Recommendation Team (since April 2018)✓ Interested in Information Retrieval, Machine LearningSelf-Introduction@yu-ya4@yu__ya4
©2018 Wantedly, Inc.
©2019 Wantedly, Inc.Wantedly ʹ͓͚Δੵۃతͳσʔλ׆༻㾎උ͞Εͨσʔλج൫w ͋ΒΏΔϩάσʔλ#JH2VFSZʹอଘw ओཁͳػೳ͝ͱͷϩάʢ[email protected]@MPH::::..%%ʣw EBJMZͰͷ3%#ͷεφοϓγϣοτʢSECVTFST::::..%%ʣw EBJMZͰͷਪન݁ՌʢSFDPNNOFESBOLJOH::::..%%ʣw ੳ͍͢͠Α͏ʹɼલॲཧΛߦ͍ϩάσʔλΛܗͨ͠தؒςʔϒϧͷ࡞ɾڞ༗
©2019 Wantedly, Inc.Wantedly ʹ͓͚Δੵۃతͳσʔλ׆༻㾎σʔλ׆༻ͷຽओԽw ϏδωεαΠυؚΊɼࣾһશһ͕#JH2VFSZͰσʔλʹΞΫηεͰ͖Δڥw #JH2VFSZͷ͍ํͱυϝΠϯࣝ֫ಘ͢ΔͨΊͷ৽ਓݚमͷ࣮ࢪw ΞϓϦέʔγϣϯΤϯδχΞ͕ࣗͨͪͰσʔλੳΛߦͬͯ՝ΛѲͯ͠ࢪࡦΛ࣮ࢪ͠ɼͦͷ݁ՌͷධՁ·ͰΛߦ͏ମ੍IUUQTTQFBLFSEFDLDPNZVZBXBOUFEMZOFXDPNFSUSBJOJOHCJHRVFSZUSBJOJOH
©2019 Wantedly, Inc.σʔλ׆༻ͷࡍʹϋϚͬͯ͠·ͬͯਏ͔ͬͨࣄྫΛ͍͔ͭ͘͝հ(´ʀТʀʆ)
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶃΛද͢ςʔϒϧsuffixᶃ log.user_search_log20190718→ 20197݄18ͷ1ͷϩά͕ೖͬͨςʔϒϧᶄ rdb.users20190718→ᶅ recommend.ranking20190718→ɹ
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶃΛද͢ςʔϒϧsuffixᶃ log.user_search_log20190718→ 20197݄18ͷ1ͷϩά͕ೖͬͨςʔϒϧᶄ rdb.users20190718→ 20197݄1824:00࣌ͰͷRDBͷεφοϓγϣοτᶅ recommend.ranking20190718→ 20197݄18ʹ༻͍ͨϥϯΩϯά
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶃҙຯͷҟͳΔΛද͢ςʔϒϧsuffixᶃ log.user_search_log20190718→ 20197݄18ͷ1ͷϩά͕ೖͬͨςʔϒϧᶄ rdb.users20190718→ 20197݄1824:00࣌ͰͷRDBͷεφοϓγϣοτ→ 20197݄18ʹ࡞͞Εͨ20197݄1724:00࣌ͰͷRDBͷεφοϓγϣοτᶅ recommend.ranking20190718→ 20197݄18ʹ༻͍ͨϥϯΩϯά→ 20197݄18·ͰͷσʔλΛݩʹ࡞͞Εͨ20197݄19ʹ༻͍ͨϥϯΩϯά
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶃҙຯͷҟͳΔΛද͢ςʔϒϧsuffixᶃ log.user_search_log20190718→ 20197݄18ͷ1ͷϩά͕ೖͬͨςʔϒϧᶄ rdb.users20190718→ 20197݄1824:00࣌ͰͷRDBͷεφοϓγϣοτ→ 20197݄18ʹ࡞͞Εͨ20197݄1724:00࣌ͰͷRDBͷεφοϓγϣοτᶅ recommend.ranking20190718→ 20197݄18ʹ༻͍ͨϥϯΩϯά→ 20197݄18·ͰͷσʔλΛݩʹ࡞͞Εͨ20197݄19ʹ༻͍ͨϥϯΩϯά←͔͠ͷʹΑͬͯɹ20197݄18ʹ༻͍ͨϥϯΩϯάΛද͢ʢೖࣾؒͳ͍΅͕͘࡞ͬͨɻɻɻʣ
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶃҙຯͷҟͳΔΛද͢ςʔϒϧsuffixᶃ log.user_search_log20190718→ 20197݄18ͷ1ͷϩά͕ೖͬͨςʔϒϧᶄ rdb.users20190718→ 20197݄1824:00࣌ͰͷRDBͷεφοϓγϣοτ→ 20197݄18ʹ࡞͞Εͨ20197݄1724:00࣌ͰͷRDBͷεφοϓγϣοτᶅ recommend.ranking20190718→ 20197݄18ʹ༻͍ͨϥϯΩϯά→ 20197݄18·ͰͷσʔλΛݩʹ࡞͞Εͨ20197݄19ʹ༻͍ͨϥϯΩϯά͜ͷ͋ͱ໓ۤυΩϡϝϯτ࡞ͬͨɻ
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶄjson ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜϦΫΤετύϥϝλΈ͍ͨͳͷΛ json ܗࣜͷ STRING Ͱอଘ͍ͯ͠Δύϥϝλ͕૿͑ͨΓݮͬͨΓ͢Δɼςʔϒϧͷ Schema Λมߋ͢Δඞཁ͕ͳ͍ͷͰϕϯϦ→
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶄjson ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜ• ಛఆͷ filter ͕ΘΕͨͱ͖ͷ log ͚ͩूΊͯੳ͍ͨ͠ʂ• ͍ΖΜͳάϥϑΛ࡞ͬͯ BI πʔϧʹొͩʂ• ͜ͷσʔλΛͬͯ model Λ࡞ʂ
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶄjson ܗࣜͰهड़͞Εͨ STRING ͷΧϥϜʹ࿔͞ΕΔDS• ಛఆͷ filter ͕ΘΕͨͱ͖ͷ log ͚ͩूΊͯੳ͍ͨ͠ʂ• ͍ΖΜͳάϥϑΛ࡞ͬͯ BI πʔϧʹొͩʂ• ͜ͷσʔλΛͬͯ model Λ࡞ʂύϥϝλ͕૿͑ͨΓݮͬͨΓ͢Δɼੳ༻ͷΫΤϦBIπʔϧͷάϥϑ model ೦ͳ͜ͱʹ…→
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶅ null
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶅ "null"
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶅjson ܗࣜͷ STRING ͷΧϥϜΛॲཧͯ͠ੳ༻ͷதؒςʔϒϧΛ࡞ͨ͠ࡍʹ͜͏ͳͬͯ͠·͍ͬͯͨɻ"null"
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶅnull Λআ͍ͨ ARRAY Λ࡞͍ͬͯΖ͍Ζॲཧͨ͠ޙʹɼͦͷ ARRAY ͷཁૉͷΛ͑ΔʢARRAY_LENGTH() ʣͬͯॲཧΛͨ͠ͱ͖ʹ௧͍ʹ…"null"
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶅjson ܗࣜͰ݁ՌΛݟΕؾ͚ͮͨɻʢແཧͰʁʣ"null"
©2019 Wantedly, Inc.ਏ͔ͬͨ͜ͱᶅ৽UIͰվળ͞Εͨ"null"
©2019 Wantedly, Inc.͓ΘΓ㾎ਏ͔ͬͨ͜ͱʢࣗۀࣗಘʣw ҙຯͷҟͳΔΛද͢ςʔϒϧTV⒏Yw KTPOܗࣜͰهड़͞Εͨ453*/(ͷΧϥϜʹ࿔͞ΕΔ%4w OVMM