Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Data Analysis Platform Development @M3, inc.
Search
hsasakawa
September 10, 2018
0
3.4k
Data Analysis Platform Development @M3, inc.
hsasakawa
September 10, 2018
Tweet
Share
More Decks by hsasakawa
See All by hsasakawa
Suffix Trees and Suffix Arrays
hsasakawa
0
110
行動ログ処理基盤の構築
hsasakawa
0
3.3k
冪等性を考慮したデータ連携ジョブの設計
hsasakawa
6
2k
Data platform development on M3 USA
hsasakawa
0
1.1k
Featured
See All Featured
Faster Mobile Websites
deanohume
306
31k
Agile that works and the tools we love
rasmusluckow
328
21k
Done Done
chrislema
182
16k
VelocityConf: Rendering Performance Case Studies
addyosmani
328
24k
Typedesign – Prime Four
hannesfritz
41
2.5k
Git: the NoSQL Database
bkeepers
PRO
428
65k
Gamification - CAS2011
davidbonilla
80
5.2k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
49
2.3k
A better future with KSS
kneath
238
17k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Transcript
Data Analysis Platform Development merpay × M3 ػցֶश NIGHT Hirohito
Sasakawa data engineer @ M3, inc.
ࠓ͢͜ͱ w .ͷσʔλੳج൫ʹ͍ͭͯ w σʔλ࿈ܞδϣϒͷ։ൃ w ·ͱΊ
M3ͷσʔλੳج൫ʹ͍ͭͯ
M3ͷσʔλੳج൫ͷݱঢ় 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ
M3ͷσʔλੳج൫ͷݱঢ় 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ ҆Ձ͔ͭߴͳ #JH2VFSZΛ ׆༻͍͖͍ͯͨ͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங Ωʔϫʔυݕࡧ
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங Ωʔϫʔυݕࡧ ࠷ۙӾཡͨ͠ਓ ࠷Α͘Ӿཡͨ͠ਓ Λද͍ࣔ͠߹ΘͤͨΓͰ͖ΔΑ͏ʹ
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλ࿈ܞδϣϒͷ࣮
ڊਓͷݞʹΔ If I have seen further it is by standing
on the shoulders of giants. - Bernardus Carnotensis ԕ͘·ͰݟͤΔͷɺͻͱ͑ʹڊਓͷݞʹ͍ͬͯΔ͔Β γϟϧτϧͷϕϧφʔϧ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ σʔλͷόϧΫϩʔυπʔϧ ༷ʑͳιʔε λʔήοτʹ ରԠ ϓϥάΠϯػߏ 3%#.4
#JH2VFSZ 4 $47 +TPOpMF :BNMϕʔεͷ؆୯ͳهड़Ͱ ઃఆΛॻ͚Δ ςʔϒϧͨΓΧϥϜ ߦ ͘Β͍ͷهड़ྔ ΤϯδχΞͳΒ͋Ε ॻ͚ΔΑ͏ʹ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ ϫʔΫϑϩʔཧπʔϧ ॲཧͷ࿈ ϫʔΫϑϩʔ Λ ίʔυͰཧ͢Δπʔϧ ΠέͯΔ$SPO δϣϒͷґଘؔΛ:BNM
ϕʔε Ͱهड़ ಠཱͳॲཧฒྻ࣮ߦՄೳ Ԇࢹػೳ ֤छ௨ ਖ਼ৗɺҟৗऴྃ࣌ ϦτϥΠػೳ ਓؒॲཧͷલޙͰႈʹͳΔ Α͏ʹॲཧΛॻ͚ͩ͘ͰΑ͍ ॲཧಉ࢜ͷؔ%JHEBH͕ཧɺ ࣮ߦͯ͘͠ΕΔ
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w EJHEBHͷXPSLqPXઃఆϑΝΠϧɺͦͷϑΝΠϧͷ αϒσΟϨΫτϦͷϑΝΠϧ͔͠ॲཧରʹͰ͖ͳ͍
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w EJHEBHͷXPSLqPXઃఆϑΝΠϧɺͦͷϑΝΠϧͷ αϒσΟϨΫτϦͷϑΝΠϧ͔͠ॲཧରʹͰ͖ͳ͍ EC@XPSLqPXEJHɺ EC@MPBEҎԼͷFNCVMLͷ ϩʔυઃఆϑΝΠϧɺ DPOGҎԼͷڞ௨ઃఆΛ ࢀরͰ͖ͳ͍
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w σΟϨΫτϦߏΛΉΈʹਂͨ͘͘͠ͳ͍ w ιϑτϦϯΫͰରԠ
ϫʔΫϑϩʔઃܭͰࠔ͍ͬͯΔ͜ͱ w FNCVMLͷQMVHJOΛCVOEMFͰཧ͢Δ߹ͷઃܭ w ݱঢ়͍͍ํ๏͕ݟ͔ͭΒͣɺXPSLqPXͷઌ಄Ͱ FNCVMLCVOEMFJOTUBMM͢ΔλεΫΛىಈ w ͍͍ํ๏͕͋Εڭ͍͑ͯͩ͘͞
·ͱΊ
·ͱΊ w .ͷσʔλੳج൫ʹ͍ͭͯ w #JH2VFSZͷར༻Λਪਐ͍͖͍ͯͨ͠ w ੳج൫ͷ՝ w Ͳ͜ʹɺͳΜͷσʔλ͕͋Δʁ w
#JH2VFSZʹ͞Ε͍ͯͳ͍σʔλ͕͋Δ w σʔλ࿈ܞδϣϒͷ࣮ w σʔλ࿈ܞδϣϒΛ؆୯ʹॻ͚ΔEJHEBH FNCVMLศར w ਓؒຊདྷ͢Δ͖ࣄʹूத