Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
実践的データ基盤への処方箋_2-9_2-12
Search
Tomoya Koike
February 03, 2022
Programming
260
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
実践的データ基盤への処方箋_2-9_2-12
Tomoya Koike
February 03, 2022
Transcript
22 slides
࣮ફత デ ʔλج൫ͷॲํᝦ ྠಡձ 2-9 ~ 2-12 খஐ࠸
2−9 ϩάऩूΤʔδΣϯτͷΩϟύγςΟʹҙ
ϩάͱ 3 Α͘ੳ͞ΕΔϩά 1. WebαʔόͷΞΫηεϩά 2. ΞϓϦέʔγϣϯͷϩά WebαʔόͷΞΫηεϩάͷྫ • ΞΫηεͨ࣌ؒ͠ɺURL
• ΞΫηεݩͷIPΞυϨε • ΞΫηεʹ༻͍ͨใ • Web App͕ઃఆͨ͠ɺϢʔβࣝผใ
ϩάϩάऩूΤʔδΣϯτͰऩू͢Δ 4 ϩάऩूΤʔδΣϯτ • όοϑΝʹΑΓϩάऩूϚωʔδϟͷෛՙΛҰఆʹͰ͖Δ • όοϑΝ͕ᷓΕͳ͍Α͏ʹαΠζΛ֬อ͢Δ͜ͱʹҙ͕ඞཁ
ϩάऩू͕Ͱ͖Δ 5 ໊ ఏڙํ๏ ఏڙɾαϙʔτ ͍ͯ͠Δձࣾ fluentd, fluent-bit OSS Treasure
Data Logstash OSS Elastic CloudWatch Cloud AWS Cloud Logging Agent Cloud GCP
2−10 σʔλͷऩूқ͕ߴ͍ͨΊ Ͱ͖Δ͚ͩΛར༻͠ແཧͳΒࣗ࡞͢Δ
σʔλେྔ͕ͩ༗༻ 7 දతͳσʔλ ϒϥβΠϕϯτ εϚϗΞϓϦΠϕϯτ IoTσόΠεσʔλ ը໘ͷεΫϩʔϧϚεͷيɺϢʔβͷϒϥβ্Ͱͷૢ࡞σʔλ εϚϗΞϓϦ্ͰͷϢʔβͷૢ࡞σʔλ ंࡌηϯαʔͷσʔλɺڥηϯαɺΤΞϥϒϧσόΠε
ϒϥβΠϕϯτεϚϗΞϓϦΠϕϯτσʔλऩूΛར༻ 8 ཉ͍͠σʔλΛऩूͰ͖Δ͕ͳ͍͔Λ୳͢ Ϣʔβͷ্ͷߦಈੳπʔϧଟ͘ଘࡏ͢Δ ϒϥβΠϕϯτ εϚϗΞϓϦΠϕϯτ Adobe AnalyticsGoogle Analytics
Google Analytics For Firebase
ࣗ࡞͢Δ߹ࢄϝοηʔδΩϡʔΛ͏ 9 ϩάऩूπʔϧΛࣗ࡞͢Δ໘ • IoTσόΠεͷσʔλͰɺΫϥυαʔϏεʹ͍͍ͷ͕ͳ͍ͱ͖ • ΞΫηεղੳπʔϧͰेͳσʔλ͕ಘΒΕͳ͍߹ • ϩά༰ΛϦΞϧλΠϜʹ׆༻͢Δඞཁ͕͋Δͱ͖I ࢄϝοηʔδΩϡʔʹϩάΛૹ৴͢Δ
ࣗ࡞͢Δ߹ࢄϝοηʔδΩϡʔΛ͏ 10 ࢄϝοηʔδΩϡʔ ΩϡʔͱɺઌೖΕઌग़͠ͷσʔλߏͷ͜ͱɻ ΩϡʔΠϯά͢Δϓϩσϡʔαͱɺpop͢ΔίϯγϡʔϚͷ2ͭͷׂ͕͋Δɻ
ࢄϝοηʔδΩϡʔͷҙ͖͢ಛͱӡ༻ͷίπ 11 ࢄϝοηʔδΩϡʔͷҙ͖͢ಛ 1. ॱংੑอূͷ༗ແ…ॱং͕ඞཁͳ߹λΠϜελϯϓΛೖΕͯฒͼସ͑Δ͕ඞཁ 2. ϝοηʔδͷॏෳ༗ແ…ॲཧΛႈʹ͢Δ͔ɺॲཧ༗ແͷஅϩδοΫΛೖΕΔ 3. ՄࢹੑλΠϜΞτ…ॲཧ࣌ؒΑΓ͍ͱɺॲཧ͕2ճҎ্Δ ӡ༻ͷίπ
• ίϯγϡʔϚ͕ॲཧʹࣦഊ͠ଓ͚ͨσουϨλʔΛઐ༻ͷΩϡʔʹೖΕΔ • ϓϩσϡʔαʔ͕ੜྔΛ੍͢ΔόοΫϓϨογϟʔͱ͍͏ΈΛೖΕΔ
۩ମతͳγεςϜͷ࡞Γํ 12 ࢄϝοηʔδΩϡʔ
2−11 ETLΛબͿϙΠϯτར༻͢Δ ίωΫλͷػೳੑͱσόοάͷ͢͠͞
ETLͱ 14 ఏڙܗଶͷҧ͍ ETL…Extract Transform LoadͷུͰɺσʔλͷநग़ɺՃɺϩʔυΛҙຯ͢Δɻ ෳࡶͳՃ͕Ͱ͖Δ͔Ͳ͏͔ ҟͳΔσʔλιʔε͔ΒͷσʔλΛՃɺ݁߹ͯ͠ϩʔυ͢Δɻ Apache Nifi,
DataSpider, Glue, Cloud Data Fusion OSSɺ༗ঈɺΫϥυͷ3छྨɻ OSS…embulkfluentd, Apache Sqoop, ༗ঈͱͯ͠DataSpider, ASTERIAͳͲɻ ΫϥυͰAWSͷGlueGCPͷCloud Data Fusion, ྆αʔϏεͷDMS, embulkͷϚωʔδυαʔϏεͰ͋ΔtroccoͳͲɻ ※DMS…Database Migration Service
͏ίωΫλͷػೳΛॏࢹ͢Δ 15 MySQLίωΫλͰ͋ΕɺWHERE۟ࠩͷΈऩू͕Ͱ͖Δ͔ɻ ϏοάσʔλͰ͋ΕɺࢄॲཧͰ͖Δ͔ʹҙ͢Δɻ
ιʔείʔυϨϕϧͰσόοά͍͢͠ͷΛར༻͢Δ 16 όά͕ى͖ͨͱ͖ɺσʔλιʔεɾ֨ೲઌɾऩू͠Α͏ͱ͢ΔσʔλͷΈ߹ ΘͤʹΑͬͯ࠶ݱྫ͕ͳ͍͜ͱɻ ίωΫλͷιʔείʔυΛݟʹߦ͚Δͷ͕େࣄɻ ETLͷαϙʔτʹௐࠪͯ͠Β͏ͱ͖ɺσʔλج൫ʹೖͬͯΒ͏Α͏४උɻ σʔλʹґଘͯ͠ى͖Δόά ఆ͍ͯ͠ͳ͍จࣈίʔυ੍ޚจࣈɺվߦίʔυ nullΛظ͢Δͱ͜Ζʹۭจࣈྻ
ΤϯδχΞ͕͍ͳ͚ΕϓϩάϥϛϯάϨεͷETLબࢶͷ1ͭ 17 ઐ༻ͷը໘্Ͱσʔλιʔε֨ ೲઌͷΞΠίϯΛͭͳ͛ͯETLॲ ཧΛఆٛɺσϓϩΠͰ͖Δɻ Apache Nifi, Talend, DataSpider, ASTERIA,
Glue, Cloud Data Fusion ͳͲɻ
2−12 σʔλϨΠΫͰऩूͨ͠σʔλ Λͳ͘͞ͳ͍Α͏ʹ͢Δ
ऩूͨ͠σʔλΛݪଇͦͷ··ੵ͢Δ 19 σʔλϨΠΫʹऩूͨ͠σʔλΛՃͤͣʹ֨ೲ͢Δ σʔλϨΠΫʹԽͰ͖༰ྔ͕֦ுͰ͖ΔΛબͿ ऩूͨ͠σʔλΛͳ͘͞ͳ͍ͨΊʹԽ͢Δ͜ͱɺ σʔλ༰ྔΛ૿ͤΔΑ͏ʹ͓ͯ͘͜͠ͱ͕ॏཁɻ ϑΝΠϧJSONܗࣜɺςʔϒϧߏͳͲΛͦͷ··อଘɻ Ճʹࣦഊͯ͠σʔλଛࣦ͢Δ͜ͱΛ͙ͨΊɻ ػີใݸਓใಗ໊ԽΛߦͬͯੵ͢Δɻ
ϑΝΠϧΦϒδΣΫτετϨʔδʹੵ͢Δ 20 ෳͷσʔληϯλʔͰෳσόΠεʹ Խͯ͠อଘ͢Δ͜ͱͰɺ ΠϨϒϯφΠϯͷݎ࿚ੑͱ 99.99%ͷՄ༻ੑΛ࣮ݱ͍ͯ͠Δɻ σʔλΛʮΦϒδΣΫτʯͱ͍͏୯ҐͰѻ͏هԱஔɻ ΫϥυαʔϏεͱͯ͠S3Cloud StorageͳͲ͕͋Δɻ ΦϒδΣΫτετϨʔδ
ΦϯϓϨͷ߹ࢄετϨʔδΛར༻ɻ OSSͱͯ͠HDFSͳͲɻ
CSVJSONσʔλσʔλϕʔεʹೖΕͯOK 21 CSVJSONσʔλΛDWH༻ੳDBʹೖΕΔ ੳ༻DBͷதͰɺੜͷσʔλ Λ֨ೲ͢ΔσʔλϨΠΫͱ Ճ͞ΕͨσʔλΛ࣋ͭDWH ʹ͚Δ JSONʹ͍ͭͯɺจࣈྻܕ·ͨJSONܕͱͯ֨͠ೲ͢Δ
σʔλ͕ΦϯϓϨϛεʹ͋ͬͯσʔλϨΠΫΫϥυʹ͢Δ 22 3ͭͷཧ༝ ैྔ՝ۚͰར༻Ͱ͖ΔͨΊ ٱੑ͕ߴ͍ͨΊ ӡ༻ਓ݅අ͕͍҆ ج൫ߏங࣌ʹσʔλྔΛਖ਼֬ʹݟੵΔͷࠔͳͨΊ AWS S3ͷٱੑΠϨϒϯφΠϯͰ͋ΓɺΦϯϓϨͰఢΘͳ͍ αʔόͷߏΛؾʹ͠ͳͯ͘Α͍ͷͰɺඞཁͳٕज़ྗ͕Լ͕Δ
σϝϦοτͱͯ͠ɺࡉ҆͘ఆ͠ͳ͍ωοτճઢ͔ɺߴ͍ઐ༻ઢ͔Λ༻͍Δඞཁ͕͋Δ
More Decks by Tomoya Koike
See All by Tomoya Koike
CDLE youth LT会 #1
koikeya
0
110
Other Decks in Programming
See All in Programming
AI駆動TDD ― 2026年10月版
hrmstrsmgs
0
110
AI時代のFlutter品質設計 — アプリ品質を守るQuality Layers
omatty198
0
390
ローカルLLMの基本と近況 2611 / Local LLM basics and recent situation 2611
kishida
3
650
テストだけで見つかる? Rubyでつくるメソッド発掘器
izawa
0
160
LLM 101: Exploring LLMs, Starting with Tokens.
dalinaum
0
120
エラーチャンネルのボールを拾い続けて鍛えるエンジニアの地力 / phpcon-niigata2026
meihei3
1
500
自宅サーバはどこまで再現できるのか - Proxmox VE上のVM環境をIaCとバックアップで戻してみた
youto
0
450
ビジネスロジックを堅牢に~Scala 3のドメインプリミティブ~
j5ik2o
0
220
Python Lambdaにおけるテストしやすい設計とAI支援の活用 / Designing Testable Python Lambda Functions with AI
kitano_yuichi
1
350
Scalaの型でマルチテナントの認可を守る
nextbeatdev
0
180
Engineering Biometric UX Across Android Versions | DroidKnights 2026
ryunen344
1
220
Enforcing Your Architecture with Unit Tests in the AI Era (Swift Connection X)
steliosf
PRO
1
190
Featured
See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
508
140k
The Spectacular Lies of Maps
axbom
PRO
1
1.1k
Side Projects
sachag
456
43k
4 Signs Your Business is Dying
shpigford
187
23k
Applied NLP in the Age of Generative AI
inesmontani
PRO
4
2.6k
Java REST API Framework Comparison - PWX 2021
mraible
34
9.8k
Designing Powerful Visuals for Engaging Learning
tmiket
1
660
Skip the Path - Find Your Career Trail
mkilby
1
300
RailsConf 2023
tenderlove
30
1.6k
エンジニアに許された特別な時間の終わり
watany
109
250k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3.6k
JAMstack: Web Apps at Ludicrous Speed - All Things Open 2022
reverentgeek
1
680