Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Data Analysis Platform Development @M3, inc.
Search
hsasakawa
September 10, 2018
0
3.4k
Data Analysis Platform Development @M3, inc.
hsasakawa
September 10, 2018
Tweet
Share
More Decks by hsasakawa
See All by hsasakawa
Suffix Trees and Suffix Arrays
hsasakawa
0
120
行動ログ処理基盤の構築
hsasakawa
0
3.3k
冪等性を考慮したデータ連携ジョブの設計
hsasakawa
6
2.1k
Data platform development on M3 USA
hsasakawa
0
1.2k
Featured
See All Featured
Bash Introduction
62gerente
613
210k
Fireside Chat
paigeccino
37
3.5k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
47
9.6k
KATA
mclloyd
30
14k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.4k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Into the Great Unknown - MozCon
thekraken
40
1.9k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Transcript
Data Analysis Platform Development merpay × M3 ػցֶश NIGHT Hirohito
Sasakawa data engineer @ M3, inc.
ࠓ͢͜ͱ w .ͷσʔλੳج൫ʹ͍ͭͯ w σʔλ࿈ܞδϣϒͷ։ൃ w ·ͱΊ
M3ͷσʔλੳج൫ʹ͍ͭͯ
M3ͷσʔλੳج൫ͷݱঢ় 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ
M3ͷσʔλੳج൫ͷݱঢ় 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ ҆Ձ͔ͭߴͳ #JH2VFSZΛ ׆༻͍͖͍ͯͨ͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங Ωʔϫʔυݕࡧ
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங Ωʔϫʔυݕࡧ ࠷ۙӾཡͨ͠ਓ ࠷Α͘Ӿཡͨ͠ਓ Λද͍ࣔ͠߹ΘͤͨΓͰ͖ΔΑ͏ʹ
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλ࿈ܞδϣϒͷ࣮
ڊਓͷݞʹΔ If I have seen further it is by standing
on the shoulders of giants. - Bernardus Carnotensis ԕ͘·ͰݟͤΔͷɺͻͱ͑ʹڊਓͷݞʹ͍ͬͯΔ͔Β γϟϧτϧͷϕϧφʔϧ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ σʔλͷόϧΫϩʔυπʔϧ ༷ʑͳιʔε λʔήοτʹ ରԠ ϓϥάΠϯػߏ 3%#.4
#JH2VFSZ 4 $47 +TPOpMF :BNMϕʔεͷ؆୯ͳهड़Ͱ ઃఆΛॻ͚Δ ςʔϒϧͨΓΧϥϜ ߦ ͘Β͍ͷهड़ྔ ΤϯδχΞͳΒ͋Ε ॻ͚ΔΑ͏ʹ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ ϫʔΫϑϩʔཧπʔϧ ॲཧͷ࿈ ϫʔΫϑϩʔ Λ ίʔυͰཧ͢Δπʔϧ ΠέͯΔ$SPO δϣϒͷґଘؔΛ:BNM
ϕʔε Ͱهड़ ಠཱͳॲཧฒྻ࣮ߦՄೳ Ԇࢹػೳ ֤छ௨ ਖ਼ৗɺҟৗऴྃ࣌ ϦτϥΠػೳ ਓؒॲཧͷલޙͰႈʹͳΔ Α͏ʹॲཧΛॻ͚ͩ͘ͰΑ͍ ॲཧಉ࢜ͷؔ%JHEBH͕ཧɺ ࣮ߦͯ͘͠ΕΔ
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w EJHEBHͷXPSLqPXઃఆϑΝΠϧɺͦͷϑΝΠϧͷ αϒσΟϨΫτϦͷϑΝΠϧ͔͠ॲཧରʹͰ͖ͳ͍
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w EJHEBHͷXPSLqPXઃఆϑΝΠϧɺͦͷϑΝΠϧͷ αϒσΟϨΫτϦͷϑΝΠϧ͔͠ॲཧରʹͰ͖ͳ͍ EC@XPSLqPXEJHɺ EC@MPBEҎԼͷFNCVMLͷ ϩʔυઃఆϑΝΠϧɺ DPOGҎԼͷڞ௨ઃఆΛ ࢀরͰ͖ͳ͍
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w σΟϨΫτϦߏΛΉΈʹਂͨ͘͘͠ͳ͍ w ιϑτϦϯΫͰରԠ
ϫʔΫϑϩʔઃܭͰࠔ͍ͬͯΔ͜ͱ w FNCVMLͷQMVHJOΛCVOEMFͰཧ͢Δ߹ͷઃܭ w ݱঢ়͍͍ํ๏͕ݟ͔ͭΒͣɺXPSLqPXͷઌ಄Ͱ FNCVMLCVOEMFJOTUBMM͢ΔλεΫΛىಈ w ͍͍ํ๏͕͋Εڭ͍͑ͯͩ͘͞
·ͱΊ
·ͱΊ w .ͷσʔλੳج൫ʹ͍ͭͯ w #JH2VFSZͷར༻Λਪਐ͍͖͍ͯͨ͠ w ੳج൫ͷ՝ w Ͳ͜ʹɺͳΜͷσʔλ͕͋Δʁ w
#JH2VFSZʹ͞Ε͍ͯͳ͍σʔλ͕͋Δ w σʔλ࿈ܞδϣϒͷ࣮ w σʔλ࿈ܞδϣϒΛ؆୯ʹॻ͚ΔEJHEBH FNCVMLศར w ਓؒຊདྷ͢Δ͖ࣄʹूத