Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo

astj
PRO
February 05, 2018

AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo

astj
PRO

February 05, 2018
Tweet

More Decks by astj

Other Decks in Technology

Transcript

  1. "84Ͱ࣮ݱͨ͠.BDLFSFM
    ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷཪଆ

    .BDLFSFM.FFUVQ5PLZP
    JEBTUKʢגࣜձࣾ͸ͯͳʣ

    View Slide

  2. w ͸ͯͳΞϓϦέʔγϣϯΤϯδχΞ
    w ͸ͯͳϒοΫϚʔΫ͸ͯͳϒϩάFUDʜ
    w डୗαʔϏε
    w .BDLFSFM d

    w αʔόαΠυΤʔδΣϯτத৺
    JEBTUK

    View Slide

  3. w NBDLFSFMBHFOUTZTUFNEରԠ
    w NBDLFSFMQMVHJOBXT͍͔ͭ͘
    w -*/&5XJMJP࿈ܞ
    w "84Ҡߦ࣌ܥྻσʔλϕʔεͷ࡮৽
    w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘ
    w ؂ࢹϧʔϧͷૢ࡞Λ௨஌νϟϯωϧʹ௨஌
    w ʜ
    JEBTUK

    View Slide

  4. ຊ೔ͷ࿩୊

    View Slide

  5. w .BDLFSFM࣌ܥྻ%#ͷ঺հ
    w ෼ཻ౓௕ظอଘͷٕज़എܠ
    w αʔόϨεϛυϧ΢ΣΞͷߏஙࣄྫ
    w %ZOBNP%#4USFBNT-BNCEB
    w ߴεϧʔϓοτ4ॻ͖ࠐΈ

    View Slide

  6. લఏ

    View Slide

  7. ࣌ܥྻσʔλ

    View Slide

  8. ࣌ܥྻσʔλ
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …

    View Slide

  9. ࣌ܥྻσʔλϕʔε
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …

    View Slide

  10. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud

    View Slide

  11. View Slide

  12. NI
    NE
    IE


    #FGPSF
    E

    View Slide

  13. NE
    NE
    IE
    E

    "GUFS

    View Slide

  14. View Slide

  15. .BDLFSFMͷ
    ࣌ܥྻσʔλϕʔε

    View Slide

  16. Ҡߦલ(SBQIJUF
    IUUQTHSBQIJUFBQQPSH

    View Slide

  17. http://blog.yuuk.io/entry/high-performance-graphite

    View Slide

  18. ҠߦޙEJBNPOE
    ಺੡ίʔυωʔϜ

    View Slide

  19. IUUQCMPHZVVLJPFOUSZUIFSFCVJMEPGUTECPODMPVE
    ࣌ܥྻσʔλϕʔεͱ͍͏֓೦ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ

    View Slide

  20. IUUQJUDIZOZIBUFOBCMPHDPNFOUSZ
    4FSWFSMFTTDPOG5PLZPͰʰαʔόϨεΞʔΩςΫνϟʹ
    ΑΔ࣌ܥྻσʔλϕʔεͷߏஙͱ؂ࢹʱͱ͍͏ൃද͖ͯ͠·ͨ͠

    View Slide

  21. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud
    3FEJT$MVTUFS &$

    View Slide

  22. ίϯηϓτ
    w εέʔϧ͢ΔΞʔΩςΫνϟ
    w ϚωʔδυαʔϏεʹΑΔӡ༻লྗԽ
    w ΞΫηεಛੑʹԠͨ͡֊૚ܕσʔλετΞ
    w 3FEJT %ZOBNP%# 4

    View Slide

  23. ετϨʔδ *0ίετ ϨΠςϯγ ༰ྔίετ
    3FEJT ˕ ˕ ˚
    %ZOBNP%# ˓ ˓ ˓
    4 ˚ ˚ ˕
    σʔλετϨʔδ

    View Slide

  24. %ZOBNP%#
    w ϑϧϚωʔδυ/P42-
    w ҆ఆɾߴύϑΥʔϚϯεɾεέʔϥϏϦςΟ
    w खࠒͳίετ
    w σʔλ༰ྔɺεϧʔϓοτ
    w ʢͱ͸͍͑අ༻Λແࢹ͸Ͱ͖ͳ͍ʣ

    View Slide

  25. 3FEJT $MVTUFS

    w ΠϯϝϞϦ,74
    w *0ίετ͕ %ZOBNP%#ΑΓ
    खܰ
    w σʔλྔΛࡹͨ͘Ί3FEJT$MVTUFS
    w ࠓͷͱ͜Ζ&$

    View Slide

  26. 4
    w ,74తʹར༻
    w ༰ྔίετ͸ඇৗʹ཈͑ΒΕΔ
    w ϨΠςϯγɾ*0ίετ͸ྼΔ
    w ॻ͖ࠐΈճ਺ΛݮΒ͍ͨ͠

    View Slide

  27. w %ZOBNP%#
    w ϝΠϯ
    w 3FEJT
    w ߋ৽ϦΫΤετΛݮΒͨ͢ΊͷόοϑΝ
    w 4
    w طଘͷอ࣋ظؒͰ͸
    ར༻͠ͳ͍
    ֊૚ܕσʔλετΞ

    View Slide

  28. NI
    NE
    IE


    ֊૚ܕσʔλετΞ

    %ZOBNP%#
    3FEJT
    E

    View Slide

  29. ෼ཻ౓ͷ௕ظอଘ

    View Slide

  30. w σʔλྔ͕େ͖͘๲ΒΉ ഒ

    w ݹ͍σʔλͷϨΠςϯγཁ݅͸গ͠؇͍
    w ৽͍͠σʔλ΄Ͳසൟʹ͸ࢀর͞Εͳ͍
    w ݹ͍σʔλͷ্ॻ͖ߋ৽͸ൃੜ͠ͳ͍

    View Slide

  31. w ݹ͍σʔλΛ4ʹ഑ஔ
    w طଘͷอ࣋ظؒͷύϑΥʔϚϯεΛҡ࣋
    w ॻ͖ࠐΈճ਺ΛݮΒ͢
    ֊૚ܕσʔλετΞ

    View Slide

  32. NE
    NE
    IE


    ֊૚ܕσʔλετΞ

    %ZOBNP%#
    3FEJT
    4
    E

    View Slide

  33. View Slide

  34. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU

    View Slide

  35. %ZOBNP%#55-

    View Slide

  36. %ZOBNP%#ͷΞΠςϜ͝ͱʹ༗ޮظݶ
    55-
    Λࢦఆͯ͠ɺࣗಈతʹ࡟আͤ͞Δ

    View Slide

  37. %ZOBNP%#
    hatena.mackerel.host1.cpu.user
    2018/02/03 00:00~03:59
    ExpireAt: 2018/02/04 05:00
    2018/02/02 16:00~19:59
    ExpireAt: 2018/02/03 21:00
    hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user

    hatena.mackerel.host1.cpu.user
    2018/02/02 20:00~23:59
    ExpireAt: 2018/02/04 01:00

    View Slide

  38. %ZOBNP%#
    hatena.mackerel.host1.cpu.user
    2018/02/03 00:00~03:59
    ExpireAt: 2018/02/04 05:00
    hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user

    hatena.mackerel.host1.cpu.user
    2018/02/02 20:00~23:59
    ExpireAt: 2018/02/04 01:00

    View Slide

  39. %ZOBNP%#
    hatena.mackerel.host1.cpu.user
    2018/02/03 00:00~03:59
    ExpireAt: 2018/02/04 05:00
    hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user

    View Slide

  40. hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user

    %ZOBNP%#

    View Slide

  41. %ZOBNP%#
    hatena.mackerel.host1.cpu.user

    View Slide

  42. %ZOBNP%#

    View Slide

  43. ೳಈతͳૢ࡞ͳ͘%ZOBNP%#্ͷ
    ϨίʔυΛ࡟আͰ͖Δ

    View Slide

  44. %ZOBNP%#4USFBNT

    View Slide

  45. w %ZOBNP%#্ͷΞΠςϜߋ৽͕

    Πϕϯτͱͯ͠ྲྀΕΔετϦʔϜ
    w ςʔϒϧͷσʔλมߋΛτϦΨʹ

    -BNCEBؔ਺ΛىಈͰ͖Δ
    w ϖΠϩʔυʹมߋલޙͷΞΠςϜ಺༰Λ

    ؚΊΔ͜ͱ͕Ͱ͖Δ

    View Slide

  46. %ZOBNP%#
    4USFBN
    -BNCEB
    ৽نΞΠςϜ௥Ճ
    طଘΞΠςϜߋ৽ ΞΠςϜ࡟আ

    View Slide

  47. EJBNPOEFYQPSUFS
    %ZOBNP%#55-

    %ZOBNP%#4USFBNT

    View Slide

  48. w 55-ʹ౸ୡͨ͠ΞΠςϜͷ࡟আ࣌ʹ

    ΞΠςϜͷ಺༰Λ౉ͯ͠-BNCEBؔ਺Λ
    ىಈ
    w -BNCEBؔ਺Ͱ4ʹॻ͖ग़͠

    View Slide

  49. ͜ͷΞΠςϜ͕55-ʹ౸ୡͯ͠

    ࡟আ͞Ε·ͨ͠
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …

    View Slide

  50. hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …

    View Slide

  51. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU

    View Slide

  52. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ
    w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ
    w 55-ܦաʹΑΓΞΠςϜ࡟আ
    w ΞΠςϜ࡟আΠϕϯτ͕4USFBNʹ௨஌
    w 4USFBN͔Β-BNCEBؔ਺ىಈ
    w -BNCEBؔ਺͕ΞΠςϜΛ4ʹॻࠐ

    View Slide

  53. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w 55-෇༩Ҏ֎ͷ
    ೳಈతͳૢ࡞͕ෆཁ
    w -BNCEBͷࣦഊ࣌ϦτϥΠ΋ࣗಈ
    w 4USFBNҎ֎ʹঢ়ଶΛ࣋ͨͳ͍
    w %ZOBNP%#ΞΠςϜ4Φϒ
    δΣΫτ

    View Slide

  54. 4
    EJBNPOE"11
    NBDLFSFM"11
    (FU0CKFDU
    )551
    )551

    View Slide

  55. γϯϓϧ

    View Slide

  56. "84ͷΞοϓσʔτʹΑ࣮ͬͯݱ

    View Slide

  57. ೥ͷ%ZOBNP%#Ξοϓσʔτ
    w 55-d
    w "VUP4DBMF %"9d
    w 71$&OEQPJOUd
    w (MPCBM5BCMFT ΦϯσϚϯυόοΫΞοϓd

    View Slide

  58. 4΁ͷߴස౓ॻ͖ࠐΈ

    View Slide

  59. w 4΁ͷߴස౓ॻ͖ࠐΈΛߦ͏৔߹͸ରԠ͕ඞཁ
    w ಺෦γϟʔσΟϯάରԠͷͨΊͷQSFpY෇༩
    w ٸܹͳ૿Ճͷ৔߹͸
    "84αϙʔτܦ༝Ͱͷ

    ࣄલରԠґཔ

    View Slide

  60. Amazon S3 όέοτͷϫʔΫϩʔυ͕ຖඵ 100
    ճͷ PUT/LIST/DELETE ϦΫΤετ·ͨ͸ຖඵ
    300 ճͷ GET ϦΫΤετΛසൟʹ௒͑Δ৔߹
    ͸ɺ࠷ߴͷύϑΥʔϚϯεͱεέʔϥϏϦςΟ
    Λ֬อ͢ΔͨΊɺ͜ͷτϐοΫͷΨΠυϥΠϯ
    ʹै͍ͬͯͩ͘͞ɻ
    ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲
    https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request-
    rate-perf-considerations.html

    View Slide

  61. EJBNPOEͰ4΁ͷॻ͖ࠐΈ͸ίϯελϯτʹൃੜ
    SQT͸༏ʹ௒͑Δ

    View Slide

  62. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ
    w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ
    w 55-ܦաʹΑΓΞΠςϜ࡟আ
    w ΞΠςϜ࡟আΠϕϯτ͕4USFBNʹ௨஌
    w 4USFBN͔Β-BNCEBؔ਺ىಈ
    w -BNCEBؔ਺͕ΞΠςϜΛ4ʹॻࠐ
    ࠶ܝ

    View Slide

  63. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w 55-෇༩Ҏ֎ͷ
    ೳಈతͳૢ࡞͕ෆཁ
    w -BNCEBͷࣦഊ࣌ϦτϥΠ΋ࣗಈ
    w %ZOBNP%#ͷΞΠςϜ4Φϒ
    δΣΫτ
    ࠶ܝ

    View Slide

  64. ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲
    https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request-
    rate-perf-considerations.html

    View Slide

  65. w ΦϒδΣΫτ໊ઌ಄ʹ.%ϋογϡͷ෇༩
    w ॻ͖ࠐΈ։࢝લʹαϙʔτʹରԠΛґཔ
    w ਺೔Φʔμʔͷ଴͕ͪ࣌ؒൃੜ͢Δ

    View Slide

  66. ա৒ͳϖʔεͰͷߋ৽͸)551YYΤϥʔ͕
    ൃੜ͢ΔʢΒ͍͠ʣ
    ϞχλϦϯάʂ

    View Slide

  67. "844ͷϞχλϦϯά

    View Slide

  68. w 4ͷඪ४$MPVE8BUDI.FUSJDT͸ݶఆత
    w όέοτͷ೔࣍ͷετϨʔδϝτϦΫε
    w ϦΫΤετͷ෼ੳ͸Ͱ͖ͳ͍
    w ΦϓγϣϯͰϦΫΤετͷϞχλϦϯά΋Մೳ
    w 4όέοτͷઃఆͰ༗ޮʹͰ͖Δ
    https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/
    cloudwatch-monitoring.html

    View Slide

  69. View Slide

  70. $MPVE8BUDI.BDLFSFM

    View Slide

  71. mackerel-plugin-aws-s3-requests
    [plugin.metrics.diamond-s3]
    command = "/path/to/mackerel-plugin-aws-s3-requests --bucket-
    name=XXX --region=ap-northeast-1 --filter-id YYY"
    ެࣜϓϥάΠϯͱͯ͠ϦϦʔε༧ఆ
    ͖ͬͱࠓिதʹ͸ʜ

    View Slide

  72. View Slide

  73. w (FU 1VUͦΕͧΕͷϦΫΤετ਺
    w ୹ظతͳόʔετɺ௕ظతͳτϨϯυ
    w YYΤϥʔͷ਺
    ؂ࢹϧʔϧ

    View Slide

  74. ͓ΘΓʹ

    View Slide

  75. w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷ಺෦࣮૷
    w ߴස౓4ॻ͖ࠐΈ
    w ཹҙࣄ߲ϞχλϦϯά

    View Slide

  76. ༗Γ೉͏͍͟͝·ͨ͠ʂ

    View Slide