Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Data Analysis Platform Development @M3, inc.
Search
hsasakawa
September 10, 2018
0
3.4k
Data Analysis Platform Development @M3, inc.
hsasakawa
September 10, 2018
Tweet
Share
More Decks by hsasakawa
See All by hsasakawa
Suffix Trees and Suffix Arrays
hsasakawa
0
120
行動ログ処理基盤の構築
hsasakawa
0
3.4k
冪等性を考慮したデータ連携ジョブの設計
hsasakawa
6
2.1k
Data platform development on M3 USA
hsasakawa
0
1.2k
Featured
See All Featured
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.7k
Reflections from 52 weeks, 52 projects
jeffersonlam
352
21k
Build your cross-platform service in a week with App Engine
jlugia
232
18k
Making the Leap to Tech Lead
cromwellryan
135
9.6k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
The Cost Of JavaScript in 2023
addyosmani
54
9k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Building Flexible Design Systems
yeseniaperezcruz
329
39k
Statistics for Hackers
jakevdp
799
220k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
30
2.9k
Transcript
Data Analysis Platform Development merpay × M3 ػցֶश NIGHT Hirohito
Sasakawa data engineer @ M3, inc.
ࠓ͢͜ͱ w .ͷσʔλੳج൫ʹ͍ͭͯ w σʔλ࿈ܞδϣϒͷ։ൃ w ·ͱΊ
M3ͷσʔλੳج൫ʹ͍ͭͯ
M3ͷσʔλੳج൫ͷݱঢ় 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ
M3ͷσʔλੳج൫ͷݱঢ় 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ ҆Ձ͔ͭߴͳ #JH2VFSZΛ ׆༻͍͖͍ͯͨ͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங Ωʔϫʔυݕࡧ
ϝλσʔλͷݕࡧγεςϜ #JH2VFSZͷ wσʔληοτ໊ɺςʔϒϧ໊ɺΧϥϜ໊ wσʔληοτɺςʔϒϧɺΧϥϜͷEFTDSJQUJPO ͰΩʔϫʔυݕࡧͰ͖ΔγεςϜΛߏங Ωʔϫʔυݕࡧ ࠷ۙӾཡͨ͠ਓ ࠷Α͘Ӿཡͨ͠ਓ Λද͍ࣔ͠߹ΘͤͨΓͰ͖ΔΑ͏ʹ
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλੳج൫ͷ Ͳ͜ʹɺͲΜͳ σʔλ͕͋Δͷ͔ Θ͔Βͳ͍ #JH2VFSZʹ ࿈ܞ͞Ε͍ͯͳ͍ σʔλ͕͋Δ XJLJΛݟΔ ͳ͍߹
ਓʹฉ͘ ੳʹ͔͔Δ࣌ؒͷ େ෦͜͜ ݸਓͷײ ࿈ܞδϣϒͷ࣮ͪ ͳΔ͘؆୯ʹ࣮Ͱ͖Δ ͱخ͍͠
σʔλ࿈ܞδϣϒͷ࣮
ڊਓͷݞʹΔ If I have seen further it is by standing
on the shoulders of giants. - Bernardus Carnotensis ԕ͘·ͰݟͤΔͷɺͻͱ͑ʹڊਓͷݞʹ͍ͬͯΔ͔Β γϟϧτϧͷϕϧφʔϧ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ σʔλͷόϧΫϩʔυπʔϧ ༷ʑͳιʔε λʔήοτʹ ରԠ ϓϥάΠϯػߏ 3%#.4
#JH2VFSZ 4 $47 +TPOpMF :BNMϕʔεͷ؆୯ͳهड़Ͱ ઃఆΛॻ͚Δ ςʔϒϧͨΓΧϥϜ ߦ ͘Β͍ͷهड़ྔ ΤϯδχΞͳΒ͋Ε ॻ͚ΔΑ͏ʹ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ
৽͍͠σʔλ࿈ܞδϣϒͷٕज़ελοΫ ϫʔΫϑϩʔཧπʔϧ ॲཧͷ࿈ ϫʔΫϑϩʔ Λ ίʔυͰཧ͢Δπʔϧ ΠέͯΔ$SPO δϣϒͷґଘؔΛ:BNM
ϕʔε Ͱهड़ ಠཱͳॲཧฒྻ࣮ߦՄೳ Ԇࢹػೳ ֤छ௨ ਖ਼ৗɺҟৗऴྃ࣌ ϦτϥΠػೳ ਓؒॲཧͷલޙͰႈʹͳΔ Α͏ʹॲཧΛॻ͚ͩ͘ͰΑ͍ ॲཧಉ࢜ͷؔ%JHEBH͕ཧɺ ࣮ߦͯ͘͠ΕΔ
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w EJHEBHͷXPSLqPXઃఆϑΝΠϧɺͦͷϑΝΠϧͷ αϒσΟϨΫτϦͷϑΝΠϧ͔͠ॲཧରʹͰ͖ͳ͍
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w EJHEBHͷXPSLqPXઃఆϑΝΠϧɺͦͷϑΝΠϧͷ αϒσΟϨΫτϦͷϑΝΠϧ͔͠ॲཧରʹͰ͖ͳ͍ EC@XPSLqPXEJHɺ EC@MPBEҎԼͷFNCVMLͷ ϩʔυઃఆϑΝΠϧɺ DPOGҎԼͷڞ௨ઃఆΛ ࢀরͰ͖ͳ͍
ϫʔΫϑϩʔઃܭͷϕετϓϥΫςΟε (WIP) w σΟϨΫτϦߏΛΉΈʹਂͨ͘͘͠ͳ͍ w ιϑτϦϯΫͰରԠ
ϫʔΫϑϩʔઃܭͰࠔ͍ͬͯΔ͜ͱ w FNCVMLͷQMVHJOΛCVOEMFͰཧ͢Δ߹ͷઃܭ w ݱঢ়͍͍ํ๏͕ݟ͔ͭΒͣɺXPSLqPXͷઌ಄Ͱ FNCVMLCVOEMFJOTUBMM͢ΔλεΫΛىಈ w ͍͍ํ๏͕͋Εڭ͍͑ͯͩ͘͞
·ͱΊ
·ͱΊ w .ͷσʔλੳج൫ʹ͍ͭͯ w #JH2VFSZͷར༻Λਪਐ͍͖͍ͯͨ͠ w ੳج൫ͷ՝ w Ͳ͜ʹɺͳΜͷσʔλ͕͋Δʁ w
#JH2VFSZʹ͞Ε͍ͯͳ͍σʔλ͕͋Δ w σʔλ࿈ܞδϣϒͷ࣮ w σʔλ࿈ܞδϣϒΛ؆୯ʹॻ͚ΔEJHEBH FNCVMLศར w ਓؒຊདྷ͢Δ͖ࣄʹूத