Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel...

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for astj astj PRO
February 05, 2018
Technology
31k
6

AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo

2018/02/05 https://mackerelio.connpass.com/event/76678/

Avatar for astj

astj PRO

February 05, 2018

More Decks by astj

See All by astj
ISUCON14公式反省会LT: 社内ISUCONの話
Avatar for astj astj
PRO
0
590
Mackerel の時系列データベースにおける Redis Cluster の利用と Amazon ElastiCache への移行について / AWS Purpose-Built Databases Week
Avatar for astj astj
PRO
3
6.1k
Mackerel のコンテナ監視の近況 / Mackerel Meetup #13
Avatar for astj astj
PRO
1
2.7k
Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8
Avatar for astj astj
PRO
2
2.4k
サービス開発と健全なプロダクトメンテナンスを開発チームで持続的に取り組む / Developers Boost KANSAI
Avatar for astj astj
PRO
1
1.6k
Mackerel and Stripe / Qiita x Stripe Meetup
Avatar for astj astj
PRO
0
5.8k
Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11
Avatar for astj astj
PRO
0
1.3k
稼働中の Web サービスの perl のバージョンを上げていく
Avatar for astj astj
PRO
0
7.6k
Perl 6 で Web Application Framework をつくる
Avatar for astj astj
PRO
0
4.9k

Other Decks in Technology

See All in Technology
AI時代のAPI品質を支えるガードレール / API Guardrails for API quality in the AI ​​era
Avatar for Yoichi Kawasaki yokawasa
1
250
アプリをもっと"iOSアプリっぽく"する小さな工夫 / Small Touches That Make Your App Feel More Like an iOS App
Avatar for matsuji matsuji
1
800
Snowflakeで実現する全社横断の顧客の声（VOC）分析・活用基盤@Snowflake World Tour Tokyo 2026
Avatar for yuto16 yuto16
0
200
俺の仕事は AIに奪われないし、たぶんその BIも要らない
Avatar for Hikaru Sawahata hikaruri
0
370
株式会社シーエーシー　エンジニア向け会社紹介資料
Avatar for 株式会社シーエーシー cac
0
57k
Gitは怖い？共有ワークスペースから始めるSnowflakeチーム開発
Avatar for あべ coco_se
0
200
2026-09-09 【sigma_ucj#1】Sigma を IaC 管理したい！ / IaC for Sigma
Avatar for civitaspo civitaspo
0
130
Adaptive Warehouse を今すぐ導入すべき理由と迷ったときの判断基準
Avatar for あれ __allllllllez__
0
210
Multica × 長期記憶：40個のミニプロジェクト管理
Avatar for eiei114 eiei114
1
270
AI 時代のスタートアップエコシステ厶から考究する技術的負債との向き合い方
Avatar for memory m3m0r7
PRO
2
680
omasushiというライブラリを作った
Avatar for polidog polidog
PRO
0
220
消えない　動かない　効かない
Avatar for Yuuki Yamashita yama3133
1
120

Featured

See All Featured
End of SEO as We Know It (SMX Advanced Version)
Avatar for Michael King ipullrank
3
4.4k
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
350
KATA
Avatar for Megan Lloyd mclloyd
PRO
35
15k
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
2
270
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
9
1.5k
Lessons Learnt from Crawling 1000+ Websites
Avatar for Charles Meaden charlesmeaden
PRO
1
1.6k
Prompt Engineering for Job Search
Avatar for Mfonobong Umondia mfonobong
0
450
The untapped power of vector embeddings
Avatar for Frank van Dijk frankvandijk
2
1.9k
We Analyzed 250 Million AI Search Results: Here's What I Found
Avatar for Josh Blyskal joshbly
1
1.9k
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
16
2.2k
技術選定の審美眼（2025年版） / Understanding the Spiral of Technologies 2025 edition
Avatar for Takuto WADA twada
PRO
120
120k
Exploring the relationship between traditional SERPs and Gen AI search
Avatar for Ray Grieselhuber raygrieselhuber
PRO
2
4.3k

Transcript

  1. "84Ͱ࣮ݱͨ͠.BDLFSFM ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷཪଆ  .BDLFSFM.FFUVQ5PLZP JEBTUKʢגࣜձࣾ͸ͯͳʣ

  2. w ͸ͯͳΞϓϦέʔγϣϯΤϯδχΞ w ͸ͯͳϒοΫϚʔΫ͸ͯͳϒϩάFUDʜ w डୗαʔϏε w .BDLFSFM d 

    w αʔόαΠυΤʔδΣϯτத৺ JEBTUK

  3. w NBDLFSFMBHFOUTZTUFNEରԠ w NBDLFSFMQMVHJOBXT ͍͔ͭ͘ w -*/&5XJMJP࿈ܞ w "84Ҡߦ࣌ܥྻσʔλϕʔεͷ࡮৽ w

    ࣌ܥྻσʔλ෼ཻ౓௕ظอଘ w ؂ࢹϧʔϧͷૢ࡞Λ௨஌νϟϯωϧʹ௨஌ w ʜ JEBTUK

  4. ຊ೔ͷ࿩୊

  5. w .BDLFSFM࣌ܥྻ%#ͷ঺հ w ෼ཻ౓௕ظอଘͷٕज़എܠ w αʔόϨεϛυϧ΢ΣΞͷߏஙࣄྫ w %ZOBNP%#4USFBNT -BNCEB w

    ߴεϧʔϓοτ4ॻ͖ࠐΈ

  6. લఏ

  7. ࣌ܥྻσʔλ

  8. ࣌ܥྻσʔλ hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …

  9. ࣌ܥྻσʔλϕʔε hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z

    44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …

  10. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud

  11. None

  12. NI NE IE   #FGPSF E

  13. NE NE IE E   "GUFS

  14. None

  15. .BDLFSFMͷ ࣌ܥྻσʔλϕʔε

  16. Ҡߦલ(SBQIJUF IUUQTHSBQIJUFBQQPSH

  17. http://blog.yuuk.io/entry/high-performance-graphite

  18. ҠߦޙEJBNPOE ಺੡ίʔυωʔϜ

  19. IUUQCMPHZVVLJPFOUSZUIFSFCVJMEPGUTECPODMPVE ࣌ܥྻσʔλϕʔεͱ͍͏֓೦ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ

  20. IUUQJUDIZOZIBUFOBCMPHDPNFOUSZ 4FSWFSMFTTDPOG5PLZPͰʰαʔόϨεΞʔΩςΫνϟʹ ΑΔ࣌ܥྻσʔλϕʔεͷߏஙͱ؂ࢹʱͱ͍͏ൃද͖ͯ͠·ͨ͠

  21. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud 3FEJT$MVTUFS &$

  22. ίϯηϓτ w εέʔϧ͢ΔΞʔΩςΫνϟ w ϚωʔδυαʔϏεʹΑΔӡ༻লྗԽ w ΞΫηεಛੑʹԠͨ͡֊૚ܕσʔλετΞ w 3FEJT %ZOBNP%#

    4

  23. ετϨʔδ *0ίετ ϨΠςϯγ ༰ྔίετ 3FEJT ˕ ˕ ˚ %ZOBNP%# ˓

    ˓ ˓ 4 ˚ ˚ ˕ σʔλετϨʔδ

  24. %ZOBNP%# w ϑϧϚωʔδυ/P42- w ҆ఆɾߴύϑΥʔϚϯεɾεέʔϥϏϦςΟ w खࠒͳίετ w σʔλ༰ྔɺεϧʔϓοτ w

    ʢͱ͸͍͑අ༻Λແࢹ͸Ͱ͖ͳ͍ʣ

  25. 3FEJT $MVTUFS w ΠϯϝϞϦ,74 w *0ίετ͕ %ZOBNP%#ΑΓ खܰ w σʔλྔΛࡹͨ͘Ί3FEJT$MVTUFS

    w ࠓͷͱ͜Ζ&$

  26. 4 w ,74తʹར༻ w ༰ྔίετ͸ඇৗʹ཈͑ΒΕΔ w ϨΠςϯγɾ*0ίετ͸ྼΔ w ॻ͖ࠐΈճ਺ΛݮΒ͍ͨ͠

  27. w %ZOBNP%# w ϝΠϯ w 3FEJT w ߋ৽ϦΫΤετΛݮΒͨ͢ΊͷόοϑΝ w 4

    w طଘͷอ࣋ظؒͰ͸ ར༻͠ͳ͍ ֊૚ܕσʔλετΞ 

  28. NI NE IE   ֊૚ܕσʔλετΞ  %ZOBNP%# 3FEJT E

  29. ෼ཻ౓ͷ௕ظอଘ

  30. w σʔλྔ͕େ͖͘๲ΒΉ ഒ  w ݹ͍σʔλͷϨΠςϯγཁ݅͸গ͠؇͍ w ৽͍͠σʔλ΄Ͳසൟʹ͸ࢀর͞Εͳ͍ w ݹ͍σʔλͷ্ॻ͖ߋ৽͸ൃੜ͠ͳ͍

  31. w ݹ͍σʔλΛ4ʹ഑ஔ w طଘͷอ࣋ظؒͷύϑΥʔϚϯεΛҡ࣋ w ॻ͖ࠐΈճ਺ΛݮΒ͢ ֊૚ܕσʔλετΞ 

  32. NE NE IE   ֊૚ܕσʔλετΞ  %ZOBNP%# 3FEJT 4

    E
  33. None

  34. 55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU

  35. %ZOBNP%#55-

  36. %ZOBNP%#ͷΞΠςϜ͝ͱʹ༗ޮظݶ 55- Λࢦఆͯ͠ɺࣗಈతʹ࡟আͤ͞Δ

  37. %ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 2018/02/02 16:00~19:59 ExpireAt:

    2018/02/03 21:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user hatena.mackerel.host1.cpu.user 2018/02/02 20:00~23:59 ExpireAt: 2018/02/04 01:00

  38. %ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59

    ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user hatena.mackerel.host1.cpu.user 2018/02/02 20:00~23:59 ExpireAt: 2018/02/04 01:00

  39. %ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59

    ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user

  40. hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user %ZOBNP%#

  41. %ZOBNP%# hatena.mackerel.host1.cpu.user

  42. %ZOBNP%#

  43. ೳಈతͳૢ࡞ͳ͘%ZOBNP%#্ͷ ϨίʔυΛ࡟আͰ͖Δ

  44. %ZOBNP%#4USFBNT

  45. w %ZOBNP%#্ͷΞΠςϜߋ৽͕
 Πϕϯτͱͯ͠ྲྀΕΔετϦʔϜ w ςʔϒϧͷσʔλมߋΛτϦΨʹ
 -BNCEBؔ਺ΛىಈͰ͖Δ w ϖΠϩʔυʹมߋલޙͷΞΠςϜ಺༰Λ
 ؚΊΔ͜ͱ͕Ͱ͖Δ

  46. %ZOBNP%# 4USFBN -BNCEB ৽نΞΠςϜ௥Ճ طଘΞΠςϜߋ৽ ΞΠςϜ࡟আ

  47. EJBNPOEFYQPSUFS %ZOBNP%#55-  %ZOBNP%#4USFBNT

  48. w 55-ʹ౸ୡͨ͠ΞΠςϜͷ࡟আ࣌ʹ
 ΞΠςϜͷ಺༰Λ౉ͯ͠-BNCEBؔ਺Λ ىಈ w -BNCEBؔ਺Ͱ4ʹॻ͖ग़͠

  49. ͜ͷΞΠςϜ͕55-ʹ౸ୡͯ͠
 ࡟আ͞Ε·ͨ͠ hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 …

    … hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …

  50. hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …

  51. 55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU

  52. 55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ

    w 55-ܦաʹΑΓΞΠςϜ࡟আ w ΞΠςϜ࡟আΠϕϯτ͕4USFBNʹ௨஌ w 4USFBN͔Β-BNCEBؔ਺ىಈ w -BNCEBؔ਺͕ΞΠςϜΛ4ʹॻࠐ

  53. 55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w 55-෇༩Ҏ֎ͷ ೳಈతͳૢ࡞͕ෆཁ w

    -BNCEBͷࣦഊ࣌ϦτϥΠ΋ࣗಈ w 4USFBNҎ֎ʹঢ়ଶΛ࣋ͨͳ͍ w %ZOBNP%#ΞΠςϜ4Φϒ δΣΫτ

  54. 4 EJBNPOE"11 NBDLFSFM"11 (FU0CKFDU )551 )551

  55. γϯϓϧ

  56. "84ͷΞοϓσʔτʹΑ࣮ͬͯݱ

  57. ೥ͷ%ZOBNP%#Ξοϓσʔτ w 55-d w "VUP4DBMF %"9d w 71$&OEQPJOUd w (MPCBM5BCMFT

    ΦϯσϚϯυόοΫΞοϓd

  58. 4΁ͷߴස౓ॻ͖ࠐΈ

  59. w 4΁ͷߴස౓ॻ͖ࠐΈΛߦ͏৔߹͸ରԠ͕ඞཁ w ಺෦γϟʔσΟϯάରԠͷͨΊͷQSFpY෇༩ w ٸܹͳ૿Ճͷ৔߹͸ "84αϙʔτܦ༝Ͱͷ
 ࣄલରԠґཔ

  60. Amazon S3 όέοτͷϫʔΫϩʔυ͕ຖඵ 100 ճͷ PUT/LIST/DELETE ϦΫΤετ·ͨ͸ຖඵ 300 ճͷ GET

    ϦΫΤετΛසൟʹ௒͑Δ৔߹ ͸ɺ࠷ߴͷύϑΥʔϚϯεͱεέʔϥϏϦςΟ Λ֬อ͢ΔͨΊɺ͜ͷτϐοΫͷΨΠυϥΠϯ ʹै͍ͬͯͩ͘͞ɻ ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲ https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request- rate-perf-considerations.html

  61. EJBNPOEͰ4΁ͷॻ͖ࠐΈ͸ίϯελϯτʹൃੜ SQT͸༏ʹ௒͑Δ

  62. 55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ

    w 55-ܦաʹΑΓΞΠςϜ࡟আ w ΞΠςϜ࡟আΠϕϯτ͕4USFBNʹ௨஌ w 4USFBN͔Β-BNCEBؔ਺ىಈ w -BNCEBؔ਺͕ΞΠςϜΛ4ʹॻࠐ ࠶ܝ

  63. 55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w 55-෇༩Ҏ֎ͷ ೳಈతͳૢ࡞͕ෆཁ w

    -BNCEBͷࣦഊ࣌ϦτϥΠ΋ࣗಈ w %ZOBNP%#ͷΞΠςϜ4Φϒ δΣΫτ ࠶ܝ

  64. ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲ https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request- rate-perf-considerations.html

  65. w ΦϒδΣΫτ໊ઌ಄ʹ.%ϋογϡͷ෇༩ w ॻ͖ࠐΈ։࢝લʹαϙʔτʹରԠΛґཔ w ਺೔Φʔμʔͷ଴͕ͪ࣌ؒൃੜ͢Δ

  66. ա৒ͳϖʔεͰͷߋ৽͸)551YYΤϥʔ͕ ൃੜ͢ΔʢΒ͍͠ʣ ϞχλϦϯάʂ

  67. "844ͷϞχλϦϯά

  68. w 4ͷඪ४$MPVE8BUDI.FUSJDT͸ݶఆత w όέοτͷ೔࣍ͷετϨʔδϝτϦΫε w ϦΫΤετͷ෼ੳ͸Ͱ͖ͳ͍ w ΦϓγϣϯͰϦΫΤετͷϞχλϦϯά΋Մೳ w 4όέοτͷઃఆͰ༗ޮʹͰ͖Δ

    https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/ cloudwatch-monitoring.html
  69. None

  70. $MPVE8BUDI.BDLFSFM

  71. mackerel-plugin-aws-s3-requests [plugin.metrics.diamond-s3] command = "/path/to/mackerel-plugin-aws-s3-requests --bucket- name=XXX --region=ap-northeast-1 --filter-id YYY"

    ެࣜϓϥάΠϯͱͯ͠ϦϦʔε༧ఆ ͖ͬͱࠓिதʹ͸ʜ
  72. None

  73. w (FU 1VUͦΕͧΕͷϦΫΤετ਺ w ୹ظతͳόʔετɺ௕ظతͳτϨϯυ w YYΤϥʔͷ਺ ؂ࢹϧʔϧ

  74. ͓ΘΓʹ

  75. w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷ಺෦࣮૷ w ߴස౓4ॻ͖ࠐΈ w ཹҙࣄ߲ϞχλϦϯά

  76. ༗Γ೉͏͍͟͝·ͨ͠ʂ