Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Agoop における AWS をフル活用したビッグデータ収集・分析の仕組み / Agoop b...
Search
Hiroyuki Nakazato / 中里 浩之
April 05, 2018
Technology
0
370
Agoop における AWS をフル活用したビッグデータ収集・分析の仕組み / Agoop big data collection and analysis system fully leveraging AWS
Hiroyuki Nakazato / 中里 浩之
April 05, 2018
Tweet
Share
More Decks by Hiroyuki Nakazato / 中里 浩之
See All by Hiroyuki Nakazato / 中里 浩之
Data Engineering Study 21 - Microsoft Fabric
nakazax
0
800
Azure Databricks Learning Series #3 - Network Design
nakazax
1
1.3k
Azure Databricks Learning Series #2 - Databricks SQL
nakazax
0
640
Microsoft Build 2022 Recap Party!! Azure のデータ & 分析サービス 注目アップデート / microsoft-build-2022-recap-azure-data-and-analytics
nakazax
0
690
Microsoft Build 2022 - Azure のデータ & 分析サービス 最新アップデート / Microsoft Build 2022 Updates on Azure Data and Analytics Services
nakazax
1
920
Architecture patterns of Azure Cosmos DB & Azure Synapse Analytics
nakazax
1
500
CY2021 Updates for ADF & Synapse & Cosmos
nakazax
0
100
Ignite 2020 Update Azure Synapse Analytics
nakazax
0
1.2k
Ignite 2020 Update - Azure DB for MySQL and Postgres - Azure Arc - and more
nakazax
2
420
Other Decks in Technology
See All in Technology
Yahoo!ニュースにおけるソフトウェア開発
lycorptech_jp
PRO
0
370
[OCI Skill Mapping] AWSユーザーのためのOCI(2025年8月20日開催)
oracle4engineer
PRO
2
150
ドキュメントはAIの味方!スタートアップのアジャイルを加速するADR
kawauso
3
390
AIとTDDによるNext.js「隙間ツール」開発の実践
makotot
6
700
Goss: New Production-Ready Go Binding for Faiss #coefl_go_jp
bengo4com
0
1.1k
Webアクセシビリティ入門
recruitengineers
PRO
1
270
見てわかるテスト駆動開発
recruitengineers
PRO
5
380
つくって納得、つかって実感! 大規模言語モデルことはじめ
recruitengineers
PRO
24
6.2k
ABEMAにおける 生成AI活用の現在地 / The Current Status of Generative AI at ABEMA
dekatotoro
0
670
開発と脆弱性と脆弱性診断についての話
su3158
1
1.1k
Understanding Go GC #coefl_go_jp
bengo4com
0
1.1k
第4回 関東Kaggler会 [Training LLMs with Limited VRAM]
tascj
12
1.8k
Featured
See All Featured
Testing 201, or: Great Expectations
jmmastey
45
7.6k
It's Worth the Effort
3n
187
28k
How to Ace a Technical Interview
jacobian
279
23k
A Tale of Four Properties
chriscoyier
160
23k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Unsuck your backbone
ammeep
671
58k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.4k
Producing Creativity
orderedlist
PRO
347
40k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
Become a Pro
speakerdeck
PRO
29
5.5k
Transcript
"84Λϑϧ׆༻ͨ͠ Ϗοάσʔλऩूɾੳ "HPPQ$PSQ
Ґஔใऩू ղੳ ਓͷྲྀΕ
5FDI1BSU
4FMGJOUSPEVDUJPO தཬߒ೭ /BLB[BUP)JSPZVLJ ϩάऩूγεςϜͷҡ࣋ɾվળ ΞϓϦͷόοΫΤϯυ։ൃ ΤϯδχΞνʔϜͷ։ൃɾੳͷͨΊͷ ֤छΈͷඋ גࣜձࣾ"HPPQ ٕज़ຊ෦"*։ൃ෦ όοΫΤϯυνʔϜϦʔμʔ
ϩάऩूɾੳγεςϜͷΞʔΩςΫνϟ "84ΞφϦςΟΫεαʔϏεͷϢʔεέʔε "HFOEB
ϩάऩूɾੳγεςϜ ΞʔΩςΫνϟ
͔Βӡ༻։࢝ ͯࣗ͢લͰ։ൃ ݱࡏ໊ͷόοΫΤϯυνʔϜͰ։ൃ ͯ͢"84্ͰՔಇ ॳ&$ɺ4ɺ3%4ͳͲϕʔγοΫͳαʔϏεͰߏங ৽ن"84αʔϏεΛܧଓతɾੵۃతʹಋೖ ϩάྔ૿͑ଓ͚Δ ίετޮͱ҆ఆੑΛ্ͤ͞ΔͨΊ γεςϜͷجຊใ
ϩάσʔλͷ֓ཁ εϚʔτϑΥϯΞϓϦ͔Βऩू த௨৴ঢ়گͳͲʹؔ͢Δϩά +40/ܗࣜ Ͱऩू͢ΔσʔλϘϦϡʔϜ ݅ԯ݅ σʔλαΠζ5# γεςϜ͕ऩू͢Δϩά
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF ऩूɾੵ
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF &5-
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF ੳ
ऩूɾੵ +BWB൛ͷ,JOFTJT1SPEVDFS-JCSBSZͱ,JOFTJT$POTVNFS-JCSBSZΛར༻ ,JOFTJTϓϩσϡʔαʔड৴ͨ͠+40/Λ΄΅ͦͷ··,JOFTJTʹೖ ,JOFTJTʹೖΕ͓͚ͯ࣌ؒ ࠷ؒ σʔλϩετͷ৺͕ແ͍ͨΊ ,JOFTJTίϯγϡʔϚʔҎԼͷॲཧΛ࣮ࢪ +40/Λύʔεͯ͠ޙଓͷੳͰ༻͢ΔใΛՃ &$ͰՔಇ͢ΔqVFOUEʹόοϑΝͯ͠ҰఆִؒͰ4ʹ([JQܗࣜͰग़ྗ <1PJOU>,JOFTJTಋೖͰ҆ఆੑ͕͔ͳΓ্ˠσʔλऩूܥʹ͓͢͢Ί
"VUP4DBMF&$ ,JOFTJT "VUP4DBMF&$ 4
&5- &.3Ϋϥελʔىಈ 4QBSLPO&.3 +FOLJOTPO&$ -BNCEB ࣌ؒʹճ -BNCEBؔΛίʔϧ &.3ΫϥελʔΛىಈ
&5- &.3Ϋϥελʔىಈ <2">ͳͥ$MPVE8BUDI&WFOUTͰͳ͘+FOLJOTʁ +FOLJOTΛδϣϒεέδϡʔϥʔΘΓʹ͏จԽ ӡ༻νʔϜ͕+FOLJOTͷऔΓѻ͍ʹ׳Ε͓ͯΓϦΧόϦʔͷ͢͠͞Λ༏ઌ <༨ஊ>ෳνʔϜɾෳ༻్Ͱ+FOLJOTΛ͏ͱΧΦεԽ͍͢͠ ᷖᮣʹ04ύοέʔδϓϥάΠϯΛΞοϓσʔτ͢ΔͱΒͳ͍δϣϒ͕ࣦഊ ͍ͭԿ͔ͷδϣϒ͕ಈ͍͍ͯͯϓϥάΠϯͷΠϯετʔϧ͕Ͱ͖ͳ͍FUDʜ +FOLJOT͔Βͷ٫Λࡧதˠ$MPVE8BUDI&WFOUT "84#BUDIΛݕ౼
4QBSLPO&.3 +FOLJOTPO&$ -BNCEB ࣌ؒʹճ -BNCEBؔΛίʔϧ &.3ΫϥελʔΛىಈ
&5- &.3Ϋϥελʔىಈ <2">ͳͥ-BNCEBؔʁ ༷ʑͳνʔϜͰ&.3ΫϥελʔΛؾܰʹར༻ͯ͠Β͏ͨΊ &5-ॲཧҎ֎ʹ༷ʑͳνʔϜɾ༻్Ͱ&.3Λར༻͍ͯ͠Δ &.3Ϋϥελʔͷىಈύϥϝʔλʔ͕ଟ͘Ұ͔Βىಈ͢Δͷେม Ϋϥελʔىಈͷڞ௨తͳॲཧΛ1ZUIPO CPUP Ͱ࣮ ֤νʔϜ͔ΒͷཁΛݩʹΫϥελʔͷεϖοΫΛઃఆϑΝΠϧʹهड़͠1ZUIPO
ϞδϡʔϧʹؚΊͯ-BNCEBؔͱͯ͠Ξοϓϩʔυ ֤νʔϜઃఆϑΝΠϧΛࢦఆ͠-BNCEBؔΛݺͼग़͢͜ͱͰΫϥελʔىಈ 4QBSLPO&.3 +FOLJOTPO&$ -BNCEB ࣌ؒʹճ -BNCEBؔΛίʔϧ &.3ΫϥελʔΛىಈ
&5- 3FETIJGUɾ)JWFϩʔυ ݱࡏ.Z42-3%4Λ)JWFϝλετΞ ςʔϒϧఆٛͷཧઌ ͱͯ͠ར༻ (MVFσʔλΧλϩάͷҠߦɾସΛ࣮ࢪத 3%4͔Β(MVFͷҠߦ(JU)VCͷBXTMBCTͷπʔϧΛར༻ IUUQTHJUIVCDPNBXTMBCTBXTHMVFTBNQMFTUSFFNBTUFS VUJMJUJFT)JWF@NFUBTUPSF@NJHSBUJPO ॳͷέʔε͕͍͔ͭ͋͘Γগ͠ϋϚ͕ͬͨ"84αϙʔτͷ͓ӄͰղܾʂ
4QBSL PO&.3 4 ([JQ 4 1BSRVFU 3FETIJGU 3%4 )JWF.FUBTUPSF ऩूɾੵ 4
&5- 3FETIJGUɾ)JWFϩʔυ <1PJOU>)JWFϝλετΞ3%4ΑΓ(MVF%BUB$BUBMPH͕͓͢͢Ί 3%4͕410' ୯Ұো ʹͳΒͳ͍Α͏ʹؾΛݣ͏ඞཁ͕͋Δ 3%4ಉ࣌ଓͷ੍ݶ͕͋ΔͨΊ&.3Ϋϥελʔͷಉ࣌ىಈʹ੍ݶ͕ੜ͡Δ ࠷େಉ࣌ଓϝϞϦґଘͷઃఆ ࠷େಉ࣌ଓΛ্͛ΔʹΑΓϝϞϦ͕ଟ͍ΠϯελϯελΠϓͷมߋ͕ඞཁ ϝλσʔλ͔͠ཧ͠ͳ͍ͷʹϦονͳεϖοΫମແ͍
(MVF%BUB$BUBMPHϑϧϚωʔδυ͔ͭಉ࣌ଓͷ੍ݶͳ͠ 4QBSL PO&.3 4 ([JQ 4 1BSRVFU 3FETIJGU 3%4 )JWF.FUBTUPSF ऩूɾੵ 4
&5- 3FETIJGUɾ)JWFϩʔυ 4QBSLPO&.3Ͱ࣌ؒͷϩάΛೖྗͱͯ͠ҎԼεςοϓΛ࣮ߦ 4%JTU$QͰ ݸ ,#ڃ ͷϑΝΠϧΛݸ .#d(#ڃ ͷϑΝΠϧʹू <1PJOU>4QBSL
)BEPPQ େྔͷখ͞ͳϑΝΠϧΛѻ͏ͷෆಘҙ 4QBSLͰ([JQQFE+40/Λग़ྗ)JWFςʔϒϧʹ*/4&3507&383*5& εςοϓऴྃޙɺ&.3Ϋϥελʔࣗಈআ ࣌ؒͷॲཧΛಠཱͨ͠ΫϥελʔͰ࣮ߦ͢ΔͷͰϦΧόϦʔͳͲ͕Γ͍͢ 4্ͷ([JQϑΝΠϧΛ3FETIJGUͷ$01:ίϚϯυͰϩʔυ 4QBSL PO&.3 4 ([JQ 4 1BSRVFU 3FETIJGU 3%4 )JWF.FUBTUPSF ऩूɾੵ 4
σʔλऩूܥʹ,JOFTJT4USFBN͕͓͢͢Ί )JWFϝλετΞ(MVF%BUB$BUBMPH͕͓͢͢Ί 3%4͔Β(MVFͷҠߦʹ(JU)VCͷBXTMBCTͷπʔϧ͕͑Δ IUUQTHJUIVCDPNBXTMBCTBXTHMVFTBNQMFTUSFFNBTUFSVUJMJUJFT )JWF@NFUBTUPSF@NJHSBUJPO 4QBSL )BEPPQ େྔͷখ͍͞ϑΝΠϧΛѻ͏ͷෆಘҙͳͨΊɺ 4%JTU$QͳͲΛ༻͍ͯ͋Δఔͷେ͖͞ʹ·ͱΊΔͷ͕͓͢͢Ί &.3ϕετϓϥΫςΟεͰ([JQѹॖͷ߹ɺʙ(#ϑΝΠϧΛਪ
IUUQTNFEJBBNB[POXFCTFSWJDFTDPNKQXQ "84@"NB[PO@&.3@#FTU@1SBDUJDFTQEG લͷ·ͱΊ
"84ΞφϦςΟΫεαʔϏε Ϣʔεέʔε
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF ੳ
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF 3FETIJGU +FOLJOT ఆظఆܕੳ
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF "UIFOB1SFTUPPO&.3 3FEBTI ΞυϗοΫੳ
3FETIJGU ఆظఆܕੳ͕த৺ +FOLJOTΛδϣϒεέδϡʔϥʔͱͯ͠ར༻ σʔλαΠζͱϫʔΫϩʔυʹ߹ͬͨεϖοΫͷ3*Λߪೖ 3*ͷظݶΕ͕͍ۙͨΊ)JWFPO&.3ͷҠߦΛਪਐத ΞφϦςΟΫεαʔϏεͷ͍͚
"UIFOB 1SFTUPPO&.3 ΞυϗοΫੳ͕த৺ 3FEBTI 044ͷμογϡϘʔυπʔϧ Λ(6*ͱͯ͠ར༻ ॳ಄͔Β3FEBTI "UIFOBಋೖ ݩʑ3FETIJGUͰΞυϗοΫੳߦͳ͍ͬͯͨ σʔλྔ૿Ճɺूܭύλʔϯ૿Ճ
ఆظఆܕੳͱΞυϗοΫੳͰͷϦιʔεͷୣ͍߹͍ ΞφϦςΟΫεαʔϏεͷ͍͚
ͳͥ"UIFOBͱ1SFTUPPO&.3ͷ྆ํΛར༻ʁ "UIFOBͷར༻ྉ͕ۚఆΑΓߴ͘ͳͬͯ͠·ͬͨ ྉۚ ݄ εΩϟϯαΠζ1# ݄ ࣍ળͷࡦͱͯ͠1SFTUPPO&.3Λಋೖ ΞφϦςΟΫεαʔϏεͷ͍͚
ΞυϗοΫੳͷද 3FETIJGU ར༻։࢝ ݄ w 3FEBTIར༻։࢝ w σʔλιʔεʹ
3FETIJGUͱ "UIFOBΛՃ ݄ 3FEBTIͷ σʔλιʔε͔Β 3FETIJGUΛআ ݄ ✖3FETIJGUෛՙ૿େ ✔ࣾʹ͘ීٴ ✖"UIFOBͷ ར༻ྉ͕ۚ ݄ 1# εΩϟϯ w 1SFTUPPO&.3 ར༻։࢝ w "UIFOBͷΫΤϦ Λ1SFTUPʹҠߦ ✖ੑೳ໘Ͱෆຬ ✔ར༻ྉ͕ۚ ݄Ͱ҆ఆ ݱࡏ ΞυϗοΫੳͷ සग़ͷूܭॲཧΛ )JWFPO&.3Ͱ όονॲཧԽ ݱࡏਐߦத ݄
ϋʔυΣΞϦιʔεΛҙࣝͤͣʹ5#ΦʔμʔͷεΩϟϯΛݱ࣮తͳ࣌ؒͰॲཧͰ͖Δ ͋ΔఔʮແʯͳΫΤϦ"UIFOBͳΒݱ࣮తͳ࣌ؒͰॲཧͰ͖ͯ͠·͏ "UIFOBͷ͍উखͷྑ͞ ԯ݅ͷϩάΛ࡞ผʹूܭ "UIFOBඵͰྃ εΩϟϯαΠζ(# 1SFTUPPO&.3Ͱྃ NYMBSHF .BTUFS $PSF
5BTL ඵ "UIFOB 1SFTUPPO&.3
"UIFOBͷ͍উखͷྑ͞ ϋʔυΣΞϦιʔεͷ੍ "UIFOB σϑΥϧτͷಉ࣮࣌ߦΫΤϦͷ্ݶ ಉ࣮࣌ߦͨ͠ΫΤϦؒͰϦιʔεͷୣ͍߹͍ൃੜ͠ͳ͍ 1SFTUPPO&.3 ΫϥελʔͷϝϞϦͳͲͷϦιʔεΛෳΫΤϦͰγΣΞ͢Δ ԼهͷϝϞϦઃఆΛ͑ΔΫΤϦ࣮ߦͰ͖ͳ͍ ઃఆ߲ આ໌
RVFSZNBYNFNPSZ ΫΤϦͰ༻͢ΔϝϞϦͷ࠷େ RVFSZNBYNFNPSZQFSOPEF ΫΤϦϊʔυͰ༻͢ΔϝϞϦͷ࠷େ
"UIFOBͷར༻ྉۚߴಅͷݪҼ 3FEBTIͷΫΤϦεέδϡʔϧػೳ 3FEBTIʹอଘͨ͠ΫΤϦΛࢦఆͨ͠සͰ࣮ߦͰ͖Δ "UIFOBͷ͍উखͷྑ͞ ͋ΔఔʮແʯͳΫΤϦݱ࣮తͳ࣌ؒͰॲཧͰ͖Δ 5#ΛεΩϟϯ͢ΔΫΤϦ܈͕සൟʹ࣮ߦ ߹ܭεΩϟϯαΠζ͕1#ΛΦʔόʔ
ݱࡏ1SFTUPPO&.3ΛϝΠϯͰར༻ نͰ"UIFOBʹશવఢΘͳ͍ʜ εΩϟϯαΠζΛখ͘͢͞ΔͨΊɺසग़ͷूܭॲཧΛ )JWFPO&.3ͰόονॲཧԽ͢ΔऔΛਪਐத ্هऔ͕ྃͨ͠Β"UIFOBͷํ͕҆͘ͳΔՄೳੑ͕ ͋ΔͨΊɺ"UIFOBΛϝΠϯʹ͢Δ͜ͱݕ౼ ΞυϗοΫੳͷݱࡏͱࠓޙ
ϏοάσʔλͷΞυϗοΫੳͷจ຺ʹ͓͍ͯ "UIFOB࠷ڧͷબࢶͷҰͭ ಛʹ੍ݶΛઃ͚ͣʹ"UIFOBΛಋೖ͢ΔͱఆΑΓར༻ྉ͕ۚߴ͘ ͳΔέʔε͕͋Δ 3FEBTIͱͷ߹ͤͰ"UIFOBීٴͷϋʔυϧάϯͱԼ͕Δ͕ɺ ΫΤϦεέδϡʔϧػೳͳͲҙ͕ඞཁ "84ར༻ྉۚΛීஈ͋·Γҙࣝ͠ͳ͍Ϣʔβʔ͕"UIFOBΛར༻ ͢Δ߹ɺίετҙࣝͳͲΛఆணͤ͞Δ׆ಈ͕ඞཁ ޙͷ·ͱΊɾڭ܇
ޙͷ·ͱΊɾڭ܇ ίετҙࣝ ΫΤϦ࣮ߦʹΑΓԿ(#εΩϟϯ͢Δͷ͔ɺͦͷ݁Ռྉ͍ۚ͘Βֻ͔Δ͔Λ Ϣʔβʔ͕ҙࣝ͢Δ͜ͱ͕ඇৗʹॏཁ "84ཧऀ͚ Ϣʔβʔ͕ར༻ྉۚΛ֬ೝͰ͖ΔΈΛ༻ҙ͢Δ %3:ݪଇ %PO`U3FQFBU:PVSTFMG εΩϟϯαΠζ͕େ͖͍ΫΤϦΛԿճ࣮ߦ͠ͳ͍ʗͤ͞ͳ͍
සग़ͷΫΤϦճ͚࣮ͩߦͯ݁͠ՌΛอଘ͢Δ ͓࡞๏ͷపఈ εΩϟϯαΠζݮͷͨΊύʔςΟγϣϯΛඞͣࢦఆ͢Δ ඞཁ࠷ݶͷΧϥϜΛ4&-&$5͢Δ ҆қʹ4&-&$5 ͠ͳ͍ FUD "UIFOBΛίετޮྑ͘׆༻͢ΔͨΊͷϚΠϯυηοτ
&0'