Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Akka streamsを活用したログ集計に優しいデータフローの構築
Search
Shuhei Kimura
December 05, 2018
Technology
2
5.5k
Akka streamsを活用したログ集計に優しいデータフローの構築
Shuhei Kimura
December 05, 2018
Tweet
Share
More Decks by Shuhei Kimura
See All by Shuhei Kimura
pmconf2017
kimurashuhei
0
3.6k
MDH-LT-Thursday-20170126
kimurashuhei
0
1.6k
AWS Summit Tokyo 2015
kimurashuhei
0
110
Other Decks in Technology
See All in Technology
PRDの正しい使い方 ~AI時代にも効く思考・対話・成長ツールとして~
techtekt
PRO
0
310
【 LLMエンジニアがヒューマノイド開発に挑んでみた 】 - 第104回 Machine Learning 15minutes! Hybrid
soneo1127
0
240
退屈なことはDevinにやらせよう〜〜Devin APIを使ったVisual Regression Testの自動追加〜
kawamataryo
4
1.1k
Kubernetes における cgroup driver のしくみ: runwasi の bugfix より
z63d
2
110
ヘブンバーンズレッドにおける、世界観を活かしたミニゲーム企画の作り方
gree_tech
PRO
0
430
トヨタ生産方式(TPS)入門
recruitengineers
PRO
6
1.4k
DDD集約とサービスコンテキスト境界との関係性
pandayumi
2
210
クラウドセキュリティを支える技術と運用の最前線 / Cutting-edge Technologies and Operations Supporting Cloud Security
yuj1osm
2
250
AI時代にPdMとPMMはどう連携すべきか / PdM–PMM-collaboration-in-AI-era
rakus_dev
0
250
AWS環境のリソース調査を Claude Code で効率化 / aws investigate with cc devio2025
masahirokawahara
2
1k
生成AI時代のデータ基盤設計〜ペースレイヤリングで実現する高速開発と持続性〜 / Levtech Meetup_Session_2
sansan_randd
1
110
大「個人開発サービス」時代に僕たちはどう生きるか
sotarok
5
960
Featured
See All Featured
A better future with KSS
kneath
239
17k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
139
34k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
KATA
mclloyd
32
14k
Automating Front-end Workflow
addyosmani
1370
200k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Testing 201, or: Great Expectations
jmmastey
45
7.6k
The Pragmatic Product Professional
lauravandoore
36
6.8k
The Art of Programming - Codeland 2020
erikaheidi
55
13k
GitHub's CSS Performance
jonrohan
1032
460k
Producing Creativity
orderedlist
PRO
347
40k
Transcript
BLLBTUSFBNΛ׆༻ͨ͠ϩάूܭʹ༏͍͠ σʔλϑϩʔͷߏஙʹ͍ͭͯ !#POpSF#BDLFOE
1SPpMF ໊લ ଜऺฏ4IVIFJ,JNVSB IUUQTXXXGBDFCPPLDPNTIVIFJLJNVSB ॴଐ גࣜձࣾαΠόʔΤʔδΣϯτ
ΞυςΫຊ෦"E5FDI4UVEJP%ZOBMZTU 1SPEVDU.BOBHFS4PGUXBSF&OHJOFFS ܦྺ ৽ଔೖࣾ 5SBDLJOH 1$"ET 4NBSU 1IPOF"ET %ZOBNJD 3FUBSHFUJOH "ET (MPCBM &YQBOTJPO
%ZOBMZTUͱ͍͏ࠂ৴αʔϏεͷنײ d,SFRTFD NTGPSFBDI &$*OTUBODFT 'PS#VZJOH*NQSFTTJPOT 5# MPHpMFTEBZ ѻ͏ͷ·͊·͊େมɻɻ ϚϧνϦʔδϣϯͰ
࣮ࡍ͜ͷഒͷྔ
5PEBZ`T5PQJDdϩάσʔλΛ៉ྷʹ4ʹӡͿd d,SFRTFD 5# QFSEBZ ‑‑‑‑‑
"LLB4USFBN͕ొͨ͠എܠ طଘͷ-JCSBSZ͕όά͍ͬͯͨ ,JOFTJT$POOFDUPS-JCSBSZͦͦ044ʹλμΓ͍ͯ͠Δ͜ͱ͕ ΑΖ͘͠ͳ͍͕ɺ ͍ͩͿϝϯς͞Ε͍ͯͳ͍ͷͰɺ CVHpY͢Δͷɻɻ ͍·ͩͱ͍Ζ͍ΖͳιϦϡʔγϣϯ͕͋Δʁ ,JOFTJT'JSFIPTFʁΩϨΠʹ4ʹΦϒδΣΫτΛஔ͢Δʹɺ ͕ඞཁʢ-BNCEB'VODUJPOڬΉͷͪΐͬͱɻʣ
-BNCEB'VODUJPOʁ,JOFTJT%BUB4USFBN͔Βͷ1PMMJOH*OUFSWBMΛ ࢦఆͰ͖ͳ͍ͷͰ4ͷΦϒδΣΫτ͕େʹ ʢ&5-Λ"84(MVFʹͤͯΑ͍͕ɻ8PSL'MPX&OHJOFొɻɻʣ ,$-XJUI"LLB4USFBN͕γϯϓϧͰΑ͍͔ʁ
"LLB4USFBNTͷߏཁૉ 4PVSDFϝοηʔδΛు͖ग़͢ίϯϙʔωϯτ ࢝ 'MPXϝοηʔδΛड͚औͬͯϝοηʔδΛૹΔίϯϙʔωϯτ 4JOLϝοηʔδΛड͚औΔίϯϙʔωϯτ ऴ ࢀߟURL: http://chariotsolutions.com/wp-content/uploads/blog/2016/08/akkastreamstopology.png
͜ΕΒͷίϯϙʔωϯτΛΈ߹Θͤͯ 3VOOBCMF(SBQIΛߏங
৴ϩάΛ4ʹӡͿͨΊͷ(SBQIߏங Kinesis Source Extract Record Flow Upload Log Flow Sink
˞ຊ424NFTTBHJOH'MPXɺ #SPBE$BTUͳͲར༻͍ͯ͠·ׂ͕͢Ѫ
ͲͷΜ͕ूܭʹ༏͍͠ͷ͔ʁ Kinesis Source Extract Record Flow Upload Log Flow Sink
‑‑‑‑‑ TCVDLFUYYYYYYYYZZZZH[ YYYYYY[[[[[H[ ʜ TCVDLFUYYYYZZH[ YY[[H[ ʜ 6QMPBEͨ࣌ؒ͠Ͱͳ͘ɺ ϩά͕ग़ྗ͞Εͨ࣌ؒϕʔεͰ 4ͷ࣌ؒ۠Γͷύʔςʔγϣϯ Ͱ֨ೲ 4QBSL$MVTUFS4ͷࢦఆ࣌ؒͷύεͷϩάΛ ରʹूܭॲཧΛΒͤΔ ूܭɺIPVSMZ EBJMZ ͕ओ
ͲͷΜ͕ूܭʹ༏͍͠ͷ͔ʁ Kinesis Source Extract Record Flow Upload Log Flow Sink
‑‑‑‑‑ TCVDLFUYYYYYYYYZZZZH[ YYYYYY[[[[[H[ ʜ TCVDLFUYYYYZZH[ YY[[H[ ʜ Before After BufferType Memory Local Disc S3 Object Size 5mbલޙ 15mbલޙ ूܭ࣌ؒ 30min 15min
͓·͚ɿ&$4ͷDPNQVUF3FTPVSDFʹ༏͍͠࡞Γʹ Kinesis Source Extract Record Flow Upload Log Flow Sink
‑‑‑‑‑ 'MPXͷத ,JOFTJT3FDPSE͔ΒɺඞཁͳใΛൈ͖ग़ͯ͠ɺΩϨΠʹ͢Δ ύʔςΟγϣϯ͝ͱʹ-PDBMͷετϨʔδʹϩάΛόοϑΝϦϯά ϝϞϦফඅ੍ʢ(#(#ʣ Ұఆ༰ྔҰఆ࣌ؒܦա͢Δͱ4ʹ1VU0CKFDU ‑‑‑‑‑
͓ฉ͖͍͖ͨͩ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ