Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Data Analysis Platform Development @M3, inc.
Search
hsasakawa
September 10, 2018
0
3.3k
Data Analysis Platform Development @M3, inc.
hsasakawa
September 10, 2018
Tweet
Share
More Decks by hsasakawa
See All by hsasakawa
Suffix Trees and Suffix Arrays
hsasakawa
0
78
行動ログ処理基盤の構築
hsasakawa
0
3.2k
冪等性を考慮したデータ連携ジョブの設計
hsasakawa
4
1.7k
Data platform development on M3 USA
hsasakawa
0
970
Featured
See All Featured
Writing Fast Ruby
sferik
620
60k
Principles of Awesome APIs and How to Build Them.
keavy
120
16k
Fantastic passwords and where to find them - at NoRuKo
philnash
36
2.5k
Happy Clients
brianwarren
91
6.4k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
1
1.3k
Designing the Hi-DPI Web
ddemaree
276
33k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
186
16k
Building Adaptive Systems
keathley
30
1.8k
Bootstrapping a Software Product
garrettdimon
PRO
301
110k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
321
20k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
5
1.5k
Transcript
Data Analysis Platform Development merpay × M3 ػցֶश NIGHT Hirohito
Sasakawa data engineer @ M3, inc.
ࠓ͢͜ͱ w .ͷσʔλੳج൫ʹ͍ͭͯ w σʔλ࿈ܞδϣϒͷ։ൃ w ·ͱΊ
M3ͷσʔλੳج൫ʹ͍ͭͯ
M3ͷσʔλੳج൫ͷݱঢ় 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ
M3ͷσʔλੳج൫ͷݱঢ় 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ ҆Ձ͔ͭߴͳ #JH2VFSZΛ ׆༻͍͖͍ͯͨ͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங Ωʔϫʔυݕࡧ
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங Ωʔϫʔυݕࡧ ࠷ۙӾཡͨ͠ਓ ࠷Α͘Ӿཡͨ͠ਓ Λද͍ࣔ͠߹ΘͤͨΓͰ͖ΔΑ͏ʹ
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλ࿈ܞδϣϒͷ࣮
ڊਓͷݞʹΔ If I have seen further it is by standing
on the shoulders of giants. - Bernardus Carnotensis ԕ͘·ͰݟͤΔͷɺͻͱ͑ʹڊਓͷݞʹ͍ͬͯΔ͔Β γϟϧτϧͷϕϧφʔϧ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ σʔλͷόϧΫϩʔυπʔϧ ༷ʑͳιʔε λʔήοτʹ ରԠ ϓϥάΠϯػߏ 3%#.4
#JH2VFSZ 4 $47 +TPOpMF :BNMϕʔεͷ؆୯ͳهड़Ͱ ઃఆΛॻ͚Δ ςʔϒϧͨΓΧϥϜ ߦ ͘Β͍ͷهड़ྔ ΤϯδχΞͳΒ͋Ε ॻ͚ΔΑ͏ʹ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ ϫʔΫϑϩʔཧπʔϧ ॲཧͷ࿈ ϫʔΫϑϩʔ Λ ίʔυͰཧ͢Δπʔϧ ΠέͯΔ$SPO δϣϒͷґଘؔΛ:BNM
ϕʔε Ͱهड़ ಠཱͳॲཧฒྻ࣮ߦՄೳ Ԇࢹػೳ ֤छ௨ ਖ਼ৗɺҟৗऴྃ࣌ ϦτϥΠػೳ ਓؒॲཧͷલޙͰႈʹͳΔ Α͏ʹॲཧΛॻ͚ͩ͘ͰΑ͍ ॲཧಉ࢜ͷؔ%JHEBH͕ཧɺ ࣮ߦͯ͘͠ΕΔ
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w EJHEBHͷXPSLqPXઃఆϑΝΠϧɺͦͷϑΝΠϧͷ αϒσΟϨΫτϦͷϑΝΠϧ͔͠ॲཧରʹͰ͖ͳ͍
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w EJHEBHͷXPSLqPXઃఆϑΝΠϧɺͦͷϑΝΠϧͷ αϒσΟϨΫτϦͷϑΝΠϧ͔͠ॲཧରʹͰ͖ͳ͍ EC@XPSLqPXEJHɺ EC@MPBEҎԼͷFNCVMLͷ ϩʔυઃఆϑΝΠϧɺ DPOGҎԼͷڞ௨ઃఆΛ ࢀরͰ͖ͳ͍
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w σΟϨΫτϦߏΛΉΈʹਂͨ͘͘͠ͳ͍ w ιϑτϦϯΫͰରԠ
ϫʔΫϑϩʔઃܭͰࠔ͍ͬͯΔ͜ͱ w FNCVMLͷQMVHJOΛCVOEMFͰཧ͢Δ߹ͷઃܭ w ݱঢ়͍͍ํ๏͕ݟ͔ͭΒͣɺXPSLqPXͷઌ಄Ͱ FNCVMLCVOEMFJOTUBMM͢ΔλεΫΛىಈ w ͍͍ํ๏͕͋Εڭ͍͑ͯͩ͘͞
·ͱΊ
·ͱΊ w .ͷσʔλੳج൫ʹ͍ͭͯ w #JH2VFSZͷར༻Λਪਐ͍͖͍ͯͨ͠ w ੳج൫ͷ՝ w Ͳ͜ʹɺͳΜͷσʔλ͕͋Δʁ w
#JH2VFSZʹ͞Ε͍ͯͳ͍σʔλ͕͋Δ w σʔλ࿈ܞδϣϒͷ࣮ w σʔλ࿈ܞδϣϒΛ؆୯ʹॻ͚ΔEJHEBH FNCVMLศར w ਓؒຊདྷ͢Δ͖ࣄʹूத