Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Data Analysis Platform Development @M3, inc.
Search
hsasakawa
September 10, 2018
0
3.4k
Data Analysis Platform Development @M3, inc.
hsasakawa
September 10, 2018
Tweet
Share
More Decks by hsasakawa
See All by hsasakawa
Suffix Trees and Suffix Arrays
hsasakawa
0
110
行動ログ処理基盤の構築
hsasakawa
0
3.3k
冪等性を考慮したデータ連携ジョブの設計
hsasakawa
6
2k
Data platform development on M3 USA
hsasakawa
0
1.1k
Featured
See All Featured
Bash Introduction
62gerente
611
210k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.3k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
193
16k
Docker and Python
trallard
44
3.3k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
Six Lessons from altMBA
skipperchong
27
3.6k
What's in a price? How to price your products and services
michaelherold
244
12k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.3k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
A Philosophy of Restraint
colly
203
16k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.2k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
45
9.4k
Transcript
Data Analysis Platform Development merpay × M3 ػցֶश NIGHT Hirohito
Sasakawa data engineer @ M3, inc.
ࠓ͢͜ͱ w .ͷσʔλੳج൫ʹ͍ͭͯ w σʔλ࿈ܞδϣϒͷ։ൃ w ·ͱΊ
M3ͷσʔλੳج൫ʹ͍ͭͯ
M3ͷσʔλੳج൫ͷݱঢ় 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ
M3ͷσʔλੳج൫ͷݱঢ় 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ ҆Ձ͔ͭߴͳ #JH2VFSZΛ ׆༻͍͖͍ͯͨ͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங Ωʔϫʔυݕࡧ
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங Ωʔϫʔυݕࡧ ࠷ۙӾཡͨ͠ਓ ࠷Α͘Ӿཡͨ͠ਓ Λද͍ࣔ͠߹ΘͤͨΓͰ͖ΔΑ͏ʹ
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλ࿈ܞδϣϒͷ࣮
ڊਓͷݞʹΔ If I have seen further it is by standing
on the shoulders of giants. - Bernardus Carnotensis ԕ͘·ͰݟͤΔͷɺͻͱ͑ʹڊਓͷݞʹ͍ͬͯΔ͔Β γϟϧτϧͷϕϧφʔϧ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ σʔλͷόϧΫϩʔυπʔϧ ༷ʑͳιʔε λʔήοτʹ ରԠ ϓϥάΠϯػߏ 3%#.4
#JH2VFSZ 4 $47 +TPOpMF :BNMϕʔεͷ؆୯ͳهड़Ͱ ઃఆΛॻ͚Δ ςʔϒϧͨΓΧϥϜ ߦ ͘Β͍ͷهड़ྔ ΤϯδχΞͳΒ͋Ε ॻ͚ΔΑ͏ʹ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ ϫʔΫϑϩʔཧπʔϧ ॲཧͷ࿈ ϫʔΫϑϩʔ Λ ίʔυͰཧ͢Δπʔϧ ΠέͯΔ$SPO δϣϒͷґଘؔΛ:BNM
ϕʔε Ͱهड़ ಠཱͳॲཧฒྻ࣮ߦՄೳ Ԇࢹػೳ ֤छ௨ ਖ਼ৗɺҟৗऴྃ࣌ ϦτϥΠػೳ ਓؒॲཧͷલޙͰႈʹͳΔ Α͏ʹॲཧΛॻ͚ͩ͘ͰΑ͍ ॲཧಉ࢜ͷؔ%JHEBH͕ཧɺ ࣮ߦͯ͘͠ΕΔ
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w EJHEBHͷXPSLqPXઃఆϑΝΠϧɺͦͷϑΝΠϧͷ αϒσΟϨΫτϦͷϑΝΠϧ͔͠ॲཧରʹͰ͖ͳ͍
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w EJHEBHͷXPSLqPXઃఆϑΝΠϧɺͦͷϑΝΠϧͷ αϒσΟϨΫτϦͷϑΝΠϧ͔͠ॲཧରʹͰ͖ͳ͍ EC@XPSLqPXEJHɺ EC@MPBEҎԼͷFNCVMLͷ ϩʔυઃఆϑΝΠϧɺ DPOGҎԼͷڞ௨ઃఆΛ ࢀরͰ͖ͳ͍
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w σΟϨΫτϦߏΛΉΈʹਂͨ͘͘͠ͳ͍ w ιϑτϦϯΫͰରԠ
ϫʔΫϑϩʔઃܭͰࠔ͍ͬͯΔ͜ͱ w FNCVMLͷQMVHJOΛCVOEMFͰཧ͢Δ߹ͷઃܭ w ݱঢ়͍͍ํ๏͕ݟ͔ͭΒͣɺXPSLqPXͷઌ಄Ͱ FNCVMLCVOEMFJOTUBMM͢ΔλεΫΛىಈ w ͍͍ํ๏͕͋Εڭ͍͑ͯͩ͘͞
·ͱΊ
·ͱΊ w .ͷσʔλੳج൫ʹ͍ͭͯ w #JH2VFSZͷར༻Λਪਐ͍͖͍ͯͨ͠ w ੳج൫ͷ՝ w Ͳ͜ʹɺͳΜͷσʔλ͕͋Δʁ w
#JH2VFSZʹ͞Ε͍ͯͳ͍σʔλ͕͋Δ w σʔλ࿈ܞδϣϒͷ࣮ w σʔλ࿈ܞδϣϒΛ؆୯ʹॻ͚ΔEJHEBH FNCVMLศར w ਓؒຊདྷ͢Δ͖ࣄʹूத