Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel...
Search
astj
PRO
February 05, 2018
Technology
6
30k
AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo
2018/02/05
https://mackerelio.connpass.com/event/76678/
astj
PRO
February 05, 2018
Tweet
Share
More Decks by astj
See All by astj
ISUCON14公式反省会LT: 社内ISUCONの話
astj
PRO
0
360
Mackerel の時系列データベースにおける Redis Cluster の利用と Amazon ElastiCache への移行について / AWS Purpose-Built Databases Week
astj
PRO
3
5.8k
Mackerel のコンテナ監視の近況 / Mackerel Meetup #13
astj
PRO
1
2.4k
Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8
astj
PRO
2
2.1k
サービス開発と健全なプロダクトメンテナンスを開発チームで持続的に取り組む / Developers Boost KANSAI
astj
PRO
1
1.4k
Mackerel and Stripe / Qiita x Stripe Meetup
astj
PRO
0
5.5k
Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11
astj
PRO
0
1.2k
稼働中の Web サービスの perl のバージョンを上げていく
astj
PRO
0
7.4k
Perl 6 で Web Application Framework をつくる
astj
PRO
0
4.6k
Other Decks in Technology
See All in Technology
意志の力が9割。アニメから学ぶAI時代のこれから。
endohizumi
1
110
プロダクトエンジニアリングで開発の楽しさを拡張する話
barometrica
0
220
Autonomous Database Serverless 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
18
52k
Backboneとしてのtimm2025
yu4u
3
1.2k
ウォンテッドリーのアラート設計と Datadog 移行での知見
donkomura
0
270
コミュニティと計画的偶発性理論 - 出会いが人生を変える / Life-Changing Encounters
soudai
PRO
7
1.2k
[OCI Technical Deep Dive] OCIで生成AIを活用するためのソリューション解説(2025年8月5日開催)
oracle4engineer
PRO
0
130
夢の印税生活 / Life on Royalties
tmtms
0
270
LLM時代の検索とコンテキストエンジニアリング
shibuiwilliam
2
1.1k
AIと描く、未来のBacklog 〜プロジェクト管理の次の10年を想像し、創造するセッション〜
hrm_o25
0
120
Amazon S3 Vectorsは大規模ベクトル検索を低コスト化するサーバーレスなベクトルデータベースだ #jawsugsaga / S3 Vectors As A Serverless Vector Database
quiver
2
1.1k
Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
kzykmyzw
0
300
Featured
See All Featured
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
A designer walks into a library…
pauljervisheath
207
24k
Six Lessons from altMBA
skipperchong
28
4k
Building an army of robots
kneath
306
45k
4 Signs Your Business is Dying
shpigford
184
22k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
30
9.6k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
131
19k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
Typedesign – Prime Four
hannesfritz
42
2.8k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Transcript
"84Ͱ࣮ݱͨ͠.BDLFSFM ࣌ܥྻσʔλཻظอଘͷཪଆ .BDLFSFM.FFUVQ5PLZP JEBTUKʢגࣜձࣾͯͳʣ
w ͯͳΞϓϦέʔγϣϯΤϯδχΞ w ͯͳϒοΫϚʔΫͯͳϒϩάFUDʜ w डୗαʔϏε w .BDLFSFM d
w αʔόαΠυΤʔδΣϯτத৺ JEBTUK
w NBDLFSFMBHFOUTZTUFNEରԠ w NBDLFSFMQMVHJOBXT ͍͔ͭ͘ w -*/&5XJMJP࿈ܞ w "84Ҡߦ࣌ܥྻσʔλϕʔεͷ৽ w
࣌ܥྻσʔλཻظอଘ w ࢹϧʔϧͷૢ࡞Λ௨νϟϯωϧʹ௨ w ʜ JEBTUK
ຊͷ
w .BDLFSFM࣌ܥྻ%#ͷհ w ཻظอଘͷٕज़എܠ w αʔόϨεϛυϧΣΞͷߏஙࣄྫ w %ZOBNP%#4USFBNT -BNCEB w
ߴεϧʔϓοτ4ॻ͖ࠐΈ
લఏ
࣌ܥྻσʔλ
࣌ܥྻσʔλ hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …
࣌ܥྻσʔλϕʔε hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z
44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …
http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud
None
NI NE IE #FGPSF E
NE NE IE E "GUFS
None
.BDLFSFMͷ ࣌ܥྻσʔλϕʔε
Ҡߦલ(SBQIJUF IUUQTHSBQIJUFBQQPSH
http://blog.yuuk.io/entry/high-performance-graphite
ҠߦޙEJBNPOE ίʔυωʔϜ
IUUQCMPHZVVLJPFOUSZUIFSFCVJMEPGUTECPODMPVE ࣌ܥྻσʔλϕʔεͱ͍͏֓೦ΛΫϥυͷٕͰ࠶ߏங͢Δ
IUUQJUDIZOZIBUFOBCMPHDPNFOUSZ 4FSWFSMFTTDPOG5PLZPͰʰαʔόϨεΞʔΩςΫνϟʹ ΑΔ࣌ܥྻσʔλϕʔεͷߏஙͱࢹʱͱ͍͏ൃද͖ͯ͠·ͨ͠
http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud 3FEJT$MVTUFS &$
ίϯηϓτ w εέʔϧ͢ΔΞʔΩςΫνϟ w ϚωʔδυαʔϏεʹΑΔӡ༻লྗԽ w ΞΫηεಛੑʹԠͨ͡֊ܕσʔλετΞ w 3FEJT %ZOBNP%#
4
ετϨʔδ *0ίετ ϨΠςϯγ ༰ྔίετ 3FEJT ˕ ˕ ˚ %ZOBNP%# ˓
˓ ˓ 4 ˚ ˚ ˕ σʔλετϨʔδ
%ZOBNP%# w ϑϧϚωʔδυ/P42- w ҆ఆɾߴύϑΥʔϚϯεɾεέʔϥϏϦςΟ w खࠒͳίετ w σʔλ༰ྔɺεϧʔϓοτ w
ʢͱ͍͑අ༻ΛແࢹͰ͖ͳ͍ʣ
3FEJT $MVTUFS w ΠϯϝϞϦ,74 w *0ίετ͕ %ZOBNP%#ΑΓ खܰ w σʔλྔΛࡹͨ͘Ί3FEJT$MVTUFS
w ࠓͷͱ͜Ζ&$
4 w ,74తʹར༻ w ༰ྔίετඇৗʹ͑ΒΕΔ w ϨΠςϯγɾ*0ίετྼΔ w ॻ͖ࠐΈճΛݮΒ͍ͨ͠
w %ZOBNP%# w ϝΠϯ w 3FEJT w ߋ৽ϦΫΤετΛݮΒͨ͢ΊͷόοϑΝ w 4
w طଘͷอ࣋ظؒͰ ར༻͠ͳ͍ ֊ܕσʔλετΞ
NI NE IE ֊ܕσʔλετΞ %ZOBNP%# 3FEJT E
ཻͷظอଘ
w σʔλྔ͕େ͖͘ΒΉ ഒ w ݹ͍σʔλͷϨΠςϯγཁ݅গ͠؇͍ w ৽͍͠σʔλ΄Ͳසൟʹࢀর͞Εͳ͍ w ݹ͍σʔλͷ্ॻ͖ߋ৽ൃੜ͠ͳ͍
w ݹ͍σʔλΛ4ʹஔ w طଘͷอ࣋ظؒͷύϑΥʔϚϯεΛҡ࣋ w ॻ͖ࠐΈճΛݮΒ͢ ֊ܕσʔλετΞ
NE NE IE ֊ܕσʔλετΞ %ZOBNP%# 3FEJT 4
E
None
55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU
%ZOBNP%#55-
%ZOBNP%#ͷΞΠςϜ͝ͱʹ༗ޮظݶ 55- Λࢦఆͯ͠ɺࣗಈతʹআͤ͞Δ
%ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 2018/02/02 16:00~19:59 ExpireAt:
2018/02/03 21:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user hatena.mackerel.host1.cpu.user 2018/02/02 20:00~23:59 ExpireAt: 2018/02/04 01:00
%ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59
ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user hatena.mackerel.host1.cpu.user 2018/02/02 20:00~23:59 ExpireAt: 2018/02/04 01:00
%ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59
ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user
hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user %ZOBNP%#
%ZOBNP%# hatena.mackerel.host1.cpu.user
%ZOBNP%#
ೳಈతͳૢ࡞ͳ͘%ZOBNP%#্ͷ ϨίʔυΛআͰ͖Δ
%ZOBNP%#4USFBNT
w %ZOBNP%#্ͷΞΠςϜߋ৽͕ Πϕϯτͱͯ͠ྲྀΕΔετϦʔϜ w ςʔϒϧͷσʔλมߋΛτϦΨʹ -BNCEBؔΛىಈͰ͖Δ w ϖΠϩʔυʹมߋલޙͷΞΠςϜ༰Λ ؚΊΔ͜ͱ͕Ͱ͖Δ
%ZOBNP%# 4USFBN -BNCEB ৽نΞΠςϜՃ طଘΞΠςϜߋ৽ ΞΠςϜআ
EJBNPOEFYQPSUFS %ZOBNP%#55- %ZOBNP%#4USFBNT
w 55-ʹ౸ୡͨ͠ΞΠςϜͷআ࣌ʹ ΞΠςϜͷ༰Λͯ͠-BNCEBؔΛ ىಈ w -BNCEBؔͰ4ʹॻ͖ग़͠
͜ͷΞΠςϜ͕55-ʹ౸ୡͯ͠ আ͞Ε·ͨ͠ hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 …
… hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …
hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …
55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU
55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ
w 55-ܦաʹΑΓΞΠςϜআ w ΞΠςϜআΠϕϯτ͕4USFBNʹ௨ w 4USFBN͔Β-BNCEBؔىಈ w -BNCEB͕ؔΞΠςϜΛ4ʹॻࠐ
55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w 55-༩Ҏ֎ͷ ೳಈతͳૢ࡞͕ෆཁ w
-BNCEBͷࣦഊ࣌ϦτϥΠࣗಈ w 4USFBNҎ֎ʹঢ়ଶΛ࣋ͨͳ͍ w %ZOBNP%#ΞΠςϜ4Φϒ δΣΫτ
4 EJBNPOE"11 NBDLFSFM"11 (FU0CKFDU )551 )551
γϯϓϧ
"84ͷΞοϓσʔτʹΑ࣮ͬͯݱ
ͷ%ZOBNP%#Ξοϓσʔτ w 55-d w "VUP4DBMF %"9d w 71$&OEQPJOUd w (MPCBM5BCMFT
ΦϯσϚϯυόοΫΞοϓd
4ͷߴසॻ͖ࠐΈ
w 4ͷߴසॻ͖ࠐΈΛߦ͏߹ରԠ͕ඞཁ w ෦γϟʔσΟϯάରԠͷͨΊͷQSFpY༩ w ٸܹͳ૿Ճͷ߹ "84αϙʔτܦ༝Ͱͷ ࣄલରԠґཔ
Amazon S3 όέοτͷϫʔΫϩʔυ͕ຖඵ 100 ճͷ PUT/LIST/DELETE ϦΫΤετ·ͨຖඵ 300 ճͷ GET
ϦΫΤετΛසൟʹ͑Δ߹ ɺ࠷ߴͷύϑΥʔϚϯεͱεέʔϥϏϦςΟ Λ֬อ͢ΔͨΊɺ͜ͷτϐοΫͷΨΠυϥΠϯ ʹै͍ͬͯͩ͘͞ɻ ϦΫΤετ͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲ https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request- rate-perf-considerations.html
EJBNPOEͰ4ͷॻ͖ࠐΈίϯελϯτʹൃੜ SQT༏ʹ͑Δ
55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ
w 55-ܦաʹΑΓΞΠςϜআ w ΞΠςϜআΠϕϯτ͕4USFBNʹ௨ w 4USFBN͔Β-BNCEBؔىಈ w -BNCEB͕ؔΞΠςϜΛ4ʹॻࠐ ࠶ܝ
55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w 55-༩Ҏ֎ͷ ೳಈతͳૢ࡞͕ෆཁ w
-BNCEBͷࣦഊ࣌ϦτϥΠࣗಈ w %ZOBNP%#ͷΞΠςϜ4Φϒ δΣΫτ ࠶ܝ
ϦΫΤετ͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲ https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request- rate-perf-considerations.html
w ΦϒδΣΫτ໊ઌ಄ʹ.%ϋογϡͷ༩ w ॻ͖ࠐΈ։࢝લʹαϙʔτʹରԠΛґཔ w Φʔμʔͷ͕ͪ࣌ؒൃੜ͢Δ
աͳϖʔεͰͷߋ৽)551YYΤϥʔ͕ ൃੜ͢ΔʢΒ͍͠ʣ ϞχλϦϯάʂ
"844ͷϞχλϦϯά
w 4ͷඪ४$MPVE8BUDI.FUSJDTݶఆత w όέοτͷ࣍ͷετϨʔδϝτϦΫε w ϦΫΤετͷੳͰ͖ͳ͍ w ΦϓγϣϯͰϦΫΤετͷϞχλϦϯάՄೳ w 4όέοτͷઃఆͰ༗ޮʹͰ͖Δ
https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/ cloudwatch-monitoring.html
None
$MPVE8BUDI.BDLFSFM
mackerel-plugin-aws-s3-requests [plugin.metrics.diamond-s3] command = "/path/to/mackerel-plugin-aws-s3-requests --bucket- name=XXX --region=ap-northeast-1 --filter-id YYY"
ެࣜϓϥάΠϯͱͯ͠ϦϦʔε༧ఆ ͖ͬͱࠓिதʹʜ
None
w (FU 1VUͦΕͧΕͷϦΫΤετ w ظతͳόʔετɺظతͳτϨϯυ w YYΤϥʔͷ ࢹϧʔϧ
͓ΘΓʹ
w ࣌ܥྻσʔλཻظอଘͷ෦࣮ w ߴස4ॻ͖ࠐΈ w ཹҙࣄ߲ϞχλϦϯά
༗Γ͏͍͟͝·ͨ͠ʂ