AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo

2018/02/05 https://mackerelio.connpass.com/event/76678/

Transcript

1. "84
   Ͱ࣮ݱͨ͠
   .BDLFSFM
   ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷཪଆ 
   .BDLFSFM
   .FFUVQ
   
   5PLZP JEBTUKʢגࣜձࣾ͸ͯͳʣ
2. w ͸ͯͳ
   ΞϓϦέʔγϣϯΤϯδχΞ
   w ͸ͯͳϒοΫϚʔΫ
   ͸ͯͳϒϩά
   FUDʜ
   w डୗαʔϏε
   w .BDLFSFM
   d
   w αʔόαΠυ
   
   ΤʔδΣϯτத৺ JEBTUK
3. w NBDLFSFMBHFOU
   TZTUFNE
   ରԠ
   w NBDLFSFMQMVHJOBXT
   ͍͔ͭ͘
   w -*/&
   
   5XJMJP
   ࿈ܞ
   w "84
   Ҡߦ
   
   ࣌ܥྻσʔλϕʔεͷ࡮৽
   w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘ
   w ؂ࢹϧʔϧͷૢ࡞Λ௨஌νϟϯωϧʹ௨஌
   w ʜ JEBTUK
4. ຊ೔ͷ࿩୊
5. w .BDLFSFM
   ࣌ܥྻ
   %#
   ͷ঺հ
   w ෼ཻ౓௕ظอଘͷٕज़എܠ
   w αʔόϨεϛυϧ΢ΣΞͷߏஙࣄྫ
   w %ZOBNP%#
   4USFBNT
   
   -BNCEB
   w ߴεϧʔϓοτ
   4
   ॻ͖ࠐΈ
6. લఏ
7. ࣌ܥྻσʔλ
8. ࣌ܥྻσʔλ hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …
9. ࣌ܥྻσʔλϕʔε hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …
10. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud
12. NI NE IE   #FGPSF E
13. NE NE IE E   "GUFS
15. .BDLFSFM
    ͷ
    ࣌ܥྻσʔλϕʔε
16. Ҡߦલ
    (SBQIJUF IUUQTHSBQIJUFBQQPSH
17. http://blog.yuuk.io/entry/high-performance-graphite
18. Ҡߦޙ
    EJBNPOE ಺੡ίʔυωʔϜ
19. IUUQCMPHZVVLJPFOUSZUIFSFCVJMEPGUTECPODMPVE ࣌ܥྻσʔλϕʔεͱ͍͏֓೦ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ
20. IUUQJUDIZOZIBUFOBCMPHDPNFOUSZ 4FSWFSMFTTDPOG
    5PLZP
    ͰʰαʔόϨεΞʔΩςΫνϟʹ ΑΔ࣌ܥྻσʔλϕʔεͷߏஙͱ؂ࢹʱͱ͍͏ൃද͖ͯ͠·ͨ͠
21. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud 3FEJT
    $MVTUFS
    &$
22. ίϯηϓτ w εέʔϧ͢ΔΞʔΩςΫνϟ
    w ϚωʔδυαʔϏεʹΑΔӡ༻লྗԽ
    w ΞΫηεಛੑʹԠͨ͡֊૚ܕσʔλετΞ
    w 3FEJT
    %ZOBNP%#
    4
23. ετϨʔδ *0ίετ ϨΠςϯγ ༰ྔίετ 3FEJT ˕ ˕ ˚ %ZOBNP%# ˓ ˓ ˓ 4 ˚ ˚ ˕ σʔλετϨʔδ
24. %ZOBNP%# w ϑϧϚωʔδυ
    /P42-
    w ҆ఆɾߴύϑΥʔϚϯεɾεέʔϥϏϦςΟ
    w खࠒͳίετ
    w σʔλ༰ྔɺεϧʔϓοτ
    w ʢͱ͸͍͑අ༻Λແࢹ͸Ͱ͖ͳ͍ʣ
25. 3FEJT
    $MVTUFS w ΠϯϝϞϦ
    ,74
    w *0ίετ͕ %ZOBNP%#ΑΓ खܰ
    w σʔλྔΛࡹͨ͘Ί
    3FEJT
    $MVTUFS
    w ࠓͷͱ͜Ζ
    &$
26. 4 w ,74తʹར༻
    w ༰ྔίετ͸ඇৗʹ཈͑ΒΕΔ
    w ϨΠςϯγɾ*0ίετ͸ྼΔ
    w ॻ͖ࠐΈճ਺ΛݮΒ͍ͨ͠
27. w %ZOBNP%#
    w ϝΠϯ
    w 3FEJT
    w ߋ৽ϦΫΤετΛݮΒͨ͢ΊͷόοϑΝ
    w 4
    w طଘͷอ࣋ظؒͰ͸ ར༻͠ͳ͍ ֊૚ܕσʔλετΞ 
28. NI NE IE   ֊૚ܕσʔλετΞ  %ZOBNP%# 3FEJT E
29. ෼ཻ౓ͷ௕ظอଘ
30. w σʔλྔ͕େ͖͘๲ΒΉ ഒ
    w ݹ͍σʔλͷϨΠςϯγཁ݅͸গ͠؇͍
    w ৽͍͠σʔλ΄Ͳසൟʹ͸ࢀর͞Εͳ͍
    w ݹ͍σʔλͷ্ॻ͖ߋ৽͸ൃੜ͠ͳ͍
31. w ݹ͍σʔλΛ4ʹ഑ஔ
    w طଘͷอ࣋ظؒͷύϑΥʔϚϯεΛҡ࣋
    w ॻ͖ࠐΈճ਺ΛݮΒ͢ ֊૚ܕσʔλετΞ 
32. NE NE IE   ֊૚ܕσʔλετΞ  %ZOBNP%# 3FEJT 4 E
34. 55-
    &YQJSF %ZOBNP%# %%#
    4USFBN -BNCEB 4 1VU0CKFDU
35. %ZOBNP%#
    55-
36. %ZOBNP%#
    ͷΞΠςϜ͝ͱʹ༗ޮظݶ 55- Λࢦఆͯ͠ɺࣗಈతʹ࡟আͤ͞Δ
37. %ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 2018/02/02 16:00~19:59 ExpireAt: 2018/02/03 21:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user hatena.mackerel.host1.cpu.user 2018/02/02 20:00~23:59 ExpireAt: 2018/02/04 01:00
38. %ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user hatena.mackerel.host1.cpu.user 2018/02/02 20:00~23:59 ExpireAt: 2018/02/04 01:00
39. %ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user
40. hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user %ZOBNP%#
41. %ZOBNP%# hatena.mackerel.host1.cpu.user
42. %ZOBNP%#
43. ೳಈతͳૢ࡞ͳ͘
    %ZOBNP%#
    ্ͷ ϨίʔυΛ࡟আͰ͖Δ
44. %ZOBNP%#
    4USFBNT
45. w %ZOBNP%#
    ্ͷΞΠςϜߋ৽͕
 Πϕϯτͱͯ͠ྲྀΕΔετϦʔϜ
    w ςʔϒϧͷσʔλมߋΛτϦΨʹ
 -BNCEB
    ؔ਺ΛىಈͰ͖Δ
    w ϖΠϩʔυʹมߋલޙͷΞΠςϜ಺༰Λ
 ؚΊΔ͜ͱ͕Ͱ͖Δ
46. %ZOBNP%# 4USFBN -BNCEB ৽نΞΠςϜ௥Ճ طଘΞΠςϜߋ৽ ΞΠςϜ࡟আ
47. EJBNPOEFYQPSUFS
    %ZOBNP%#
    55-
    
    %ZOBNP%#
    4USFBNT
48. w 55-
    ʹ౸ୡͨ͠ΞΠςϜͷ࡟আ࣌ʹ
 ΞΠςϜͷ಺༰Λ౉ͯ͠
    -BNCEB
    ؔ਺Λ ىಈ
    w -BNCEB
    ؔ਺Ͱ
    4
    ʹॻ͖ग़͠
49. ͜ͷΞΠςϜ͕55-ʹ౸ୡͯ͠
 ࡟আ͞Ε·ͨ͠
    hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … … hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …
50. hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …
51. 55-
    &YQJSF %ZOBNP%# %%#
    4USFBN -BNCEB 4 1VU0CKFDU
52. 55-
    &YQJSF %ZOBNP%# %%#
    4USFBN -BNCEB 4 1VU0CKFDU w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ
    w %ZOBNP%#
    ͷΞΠςϜʹ
    55-
    Λઃఆ
    w 55-
    ܦաʹΑΓΞΠςϜ࡟আ
    w ΞΠςϜ࡟আΠϕϯτ͕
    4USFBN
    ʹ௨஌
    w 4USFBN
    ͔Β
    -BNCEB
    ؔ਺ىಈ
    w -BNCEB
    ؔ਺͕ΞΠςϜΛ
    4
    ʹॻࠐ
53. 55-
    &YQJSF %ZOBNP%# %%#
    4USFBN -BNCEB 4 1VU0CKFDU w 55-෇༩Ҏ֎ͷ ೳಈతͳૢ࡞͕ෆཁ
    w -BNCEB
    ͷࣦഊ࣌ϦτϥΠ΋ࣗಈ
    w 4USFBN
    Ҏ֎ʹঢ়ଶΛ࣋ͨͳ͍
    w %ZOBNP%#
    ΞΠςϜ
    
    4
    Φϒ δΣΫτ
54. 4 EJBNPOE
    "11 NBDLFSFM
    "11 (FU0CKFDU )551 )551
55. γϯϓϧ
56. "84ͷΞοϓσʔτʹΑ࣮ͬͯݱ
57. ೥ͷ
    %ZOBNP%#
    Ξοϓσʔτ w 55-
    d
    w "VUP4DBMF
    %"9
    d
    w 71$
    &OEQPJOU
    d
    w (MPCBM
    5BCMFT
    ΦϯσϚϯυόοΫΞοϓ
    d
58. 4΁ͷߴස౓ॻ͖ࠐΈ
59. w 4
    ΁ͷߴස౓ॻ͖ࠐΈΛߦ͏৔߹͸ରԠ͕ඞཁ
    w ಺෦γϟʔσΟϯάରԠͷͨΊͷ
    QSFpY
    ෇༩
    w ٸܹͳ૿Ճͷ৔߹͸ "84
    αϙʔτܦ༝Ͱͷ
 ࣄલରԠґཔ
60. Amazon S3 όέοτͷϫʔΫϩʔυ͕ຖඵ 100 ճͷ PUT/LIST/DELETE ϦΫΤετ·ͨ͸ຖඵ 300 ճͷ GET ϦΫΤετΛසൟʹ௒͑Δ৔߹ ͸ɺ࠷ߴͷύϑΥʔϚϯεͱεέʔϥϏϦςΟ Λ֬อ͢ΔͨΊɺ͜ͷτϐοΫͷΨΠυϥΠϯ ʹै͍ͬͯͩ͘͞ɻ ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲ https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request- rate-perf-considerations.html
61. EJBNPOE
    Ͱ
    4
    ΁ͷॻ͖ࠐΈ͸ίϯελϯτʹൃੜ
    
    SQT͸༏ʹ௒͑Δ
62. 55-
    &YQJSF %ZOBNP%# %%#
    4USFBN -BNCEB 4 1VU0CKFDU w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ
    w %ZOBNP%#
    ͷΞΠςϜʹ
    55-
    Λઃఆ
    w 55-
    ܦաʹΑΓΞΠςϜ࡟আ
    w ΞΠςϜ࡟আΠϕϯτ͕
    4USFBN
    ʹ௨஌
    w 4USFBN
    ͔Β
    -BNCEB
    ؔ਺ىಈ
    w -BNCEB
    ؔ਺͕ΞΠςϜΛ
    4
    ʹॻࠐ ࠶ܝ
63. 55-
    &YQJSF %ZOBNP%# %%#
    4USFBN -BNCEB 4 1VU0CKFDU w 55-෇༩Ҏ֎ͷ ೳಈతͳૢ࡞͕ෆཁ
    w -BNCEB
    ͷࣦഊ࣌ϦτϥΠ΋ࣗಈ
    w %ZOBNP%#
    ͷΞΠςϜ
    
    4
    Φϒ δΣΫτ ࠶ܝ
64. ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲ https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request- rate-perf-considerations.html
65. w ΦϒδΣΫτ໊ઌ಄ʹ.%ϋογϡͷ෇༩
    w ॻ͖ࠐΈ։࢝લʹαϙʔτʹରԠΛґཔ
    w ਺೔Φʔμʔͷ଴͕ͪ࣌ؒൃੜ͢Δ
66. ա৒ͳϖʔεͰͷߋ৽͸)551
    YYΤϥʔ͕ ൃੜ͢ΔʢΒ͍͠ʣ
    ϞχλϦϯάʂ
67. "84
    4
    ͷϞχλϦϯά
68. w 4
    ͷඪ४
    $MPVE8BUDI
    .FUSJDT
    ͸ݶఆత
    w όέοτͷ೔࣍ͷετϨʔδϝτϦΫε
    w ϦΫΤετͷ෼ੳ͸Ͱ͖ͳ͍
    w ΦϓγϣϯͰϦΫΤετͷϞχλϦϯά΋Մೳ
    w 4
    όέοτͷઃఆͰ༗ޮʹͰ͖Δ https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/ cloudwatch-monitoring.html
70. $MPVE8BUDI
    
    .BDLFSFM
71. mackerel-plugin-aws-s3-requests [plugin.metrics.diamond-s3] command = "/path/to/mackerel-plugin-aws-s3-requests --bucket- name=XXX --region=ap-northeast-1 --filter-id YYY" ެࣜϓϥάΠϯͱͯ͠ϦϦʔε༧ఆ
    ͖ͬͱࠓिதʹ͸ʜ
73. w (FU
    1VU
    ͦΕͧΕͷϦΫΤετ਺
    w ୹ظతͳόʔετɺ௕ظతͳτϨϯυ
    w YY
    Τϥʔͷ਺ ؂ࢹϧʔϧ
74. ͓ΘΓʹ
75. w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷ಺෦࣮૷
    w ߴස౓4ॻ͖ࠐΈ
    w ཹҙࣄ߲
    
    ϞχλϦϯά
76. ༗Γ೉͏͍͟͝·ͨ͠ʂ