Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo

astj
February 05, 2018

AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo

astj

February 05, 2018
Tweet

More Decks by astj

Other Decks in Technology

Transcript

  1. "84Ͱ࣮ݱͨ͠.BDLFSFM
    ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷཪଆ

    .BDLFSFM.FFUVQ5PLZP
    JEBTUKʢגࣜձࣾ͸ͯͳʣ

    View full-size slide

  2. w ͸ͯͳΞϓϦέʔγϣϯΤϯδχΞ
    w ͸ͯͳϒοΫϚʔΫ͸ͯͳϒϩάFUDʜ
    w डୗαʔϏε
    w .BDLFSFM d

    w αʔόαΠυΤʔδΣϯτத৺
    JEBTUK

    View full-size slide

  3. w NBDLFSFMBHFOUTZTUFNEରԠ
    w NBDLFSFMQMVHJOBXT͍͔ͭ͘
    w -*/&5XJMJP࿈ܞ
    w "84Ҡߦ࣌ܥྻσʔλϕʔεͷ࡮৽
    w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘ
    w ؂ࢹϧʔϧͷૢ࡞Λ௨஌νϟϯωϧʹ௨஌
    w ʜ
    JEBTUK

    View full-size slide

  4. ຊ೔ͷ࿩୊

    View full-size slide

  5. w .BDLFSFM࣌ܥྻ%#ͷ঺հ
    w ෼ཻ౓௕ظอଘͷٕज़എܠ
    w αʔόϨεϛυϧ΢ΣΞͷߏஙࣄྫ
    w %ZOBNP%#4USFBNT-BNCEB
    w ߴεϧʔϓοτ4ॻ͖ࠐΈ

    View full-size slide

  6. ࣌ܥྻσʔλ

    View full-size slide

  7. ࣌ܥྻσʔλ
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …

    View full-size slide

  8. ࣌ܥྻσʔλϕʔε
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …

    View full-size slide

  9. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud

    View full-size slide

  10. NI
    NE
    IE


    #FGPSF
    E

    View full-size slide

  11. NE
    NE
    IE
    E

    "GUFS

    View full-size slide

  12. .BDLFSFMͷ
    ࣌ܥྻσʔλϕʔε

    View full-size slide

  13. Ҡߦલ(SBQIJUF
    IUUQTHSBQIJUFBQQPSH

    View full-size slide

  14. http://blog.yuuk.io/entry/high-performance-graphite

    View full-size slide

  15. ҠߦޙEJBNPOE
    ಺੡ίʔυωʔϜ

    View full-size slide

  16. IUUQCMPHZVVLJPFOUSZUIFSFCVJMEPGUTECPODMPVE
    ࣌ܥྻσʔλϕʔεͱ͍͏֓೦ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ

    View full-size slide

  17. IUUQJUDIZOZIBUFOBCMPHDPNFOUSZ
    4FSWFSMFTTDPOG5PLZPͰʰαʔόϨεΞʔΩςΫνϟʹ
    ΑΔ࣌ܥྻσʔλϕʔεͷߏஙͱ؂ࢹʱͱ͍͏ൃද͖ͯ͠·ͨ͠

    View full-size slide

  18. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud
    3FEJT$MVTUFS &$

    View full-size slide

  19. ίϯηϓτ
    w εέʔϧ͢ΔΞʔΩςΫνϟ
    w ϚωʔδυαʔϏεʹΑΔӡ༻লྗԽ
    w ΞΫηεಛੑʹԠͨ͡֊૚ܕσʔλετΞ
    w 3FEJT %ZOBNP%# 4

    View full-size slide

  20. ετϨʔδ *0ίετ ϨΠςϯγ ༰ྔίετ
    3FEJT ˕ ˕ ˚
    %ZOBNP%# ˓ ˓ ˓
    4 ˚ ˚ ˕
    σʔλετϨʔδ

    View full-size slide

  21. %ZOBNP%#
    w ϑϧϚωʔδυ/P42-
    w ҆ఆɾߴύϑΥʔϚϯεɾεέʔϥϏϦςΟ
    w खࠒͳίετ
    w σʔλ༰ྔɺεϧʔϓοτ
    w ʢͱ͸͍͑අ༻Λແࢹ͸Ͱ͖ͳ͍ʣ

    View full-size slide

  22. 3FEJT $MVTUFS

    w ΠϯϝϞϦ,74
    w *0ίετ͕ %ZOBNP%#ΑΓ
    खܰ
    w σʔλྔΛࡹͨ͘Ί3FEJT$MVTUFS
    w ࠓͷͱ͜Ζ&$

    View full-size slide

  23. 4
    w ,74తʹར༻
    w ༰ྔίετ͸ඇৗʹ཈͑ΒΕΔ
    w ϨΠςϯγɾ*0ίετ͸ྼΔ
    w ॻ͖ࠐΈճ਺ΛݮΒ͍ͨ͠

    View full-size slide

  24. w %ZOBNP%#
    w ϝΠϯ
    w 3FEJT
    w ߋ৽ϦΫΤετΛݮΒͨ͢ΊͷόοϑΝ
    w 4
    w طଘͷอ࣋ظؒͰ͸
    ར༻͠ͳ͍
    ֊૚ܕσʔλετΞ

    View full-size slide

  25. NI
    NE
    IE


    ֊૚ܕσʔλετΞ

    %ZOBNP%#
    3FEJT
    E

    View full-size slide

  26. ෼ཻ౓ͷ௕ظอଘ

    View full-size slide

  27. w σʔλྔ͕େ͖͘๲ΒΉ ഒ

    w ݹ͍σʔλͷϨΠςϯγཁ݅͸গ͠؇͍
    w ৽͍͠σʔλ΄Ͳසൟʹ͸ࢀর͞Εͳ͍
    w ݹ͍σʔλͷ্ॻ͖ߋ৽͸ൃੜ͠ͳ͍

    View full-size slide

  28. w ݹ͍σʔλΛ4ʹ഑ஔ
    w طଘͷอ࣋ظؒͷύϑΥʔϚϯεΛҡ࣋
    w ॻ͖ࠐΈճ਺ΛݮΒ͢
    ֊૚ܕσʔλετΞ

    View full-size slide

  29. NE
    NE
    IE


    ֊૚ܕσʔλετΞ

    %ZOBNP%#
    3FEJT
    4
    E

    View full-size slide

  30. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU

    View full-size slide

  31. %ZOBNP%#ͷΞΠςϜ͝ͱʹ༗ޮظݶ
    55-
    Λࢦఆͯ͠ɺࣗಈతʹ࡟আͤ͞Δ

    View full-size slide

  32. %ZOBNP%#
    hatena.mackerel.host1.cpu.user
    2018/02/03 00:00~03:59
    ExpireAt: 2018/02/04 05:00
    2018/02/02 16:00~19:59
    ExpireAt: 2018/02/03 21:00
    hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user

    hatena.mackerel.host1.cpu.user
    2018/02/02 20:00~23:59
    ExpireAt: 2018/02/04 01:00

    View full-size slide

  33. %ZOBNP%#
    hatena.mackerel.host1.cpu.user
    2018/02/03 00:00~03:59
    ExpireAt: 2018/02/04 05:00
    hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user

    hatena.mackerel.host1.cpu.user
    2018/02/02 20:00~23:59
    ExpireAt: 2018/02/04 01:00

    View full-size slide

  34. %ZOBNP%#
    hatena.mackerel.host1.cpu.user
    2018/02/03 00:00~03:59
    ExpireAt: 2018/02/04 05:00
    hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user

    View full-size slide

  35. hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user

    %ZOBNP%#

    View full-size slide

  36. %ZOBNP%#
    hatena.mackerel.host1.cpu.user

    View full-size slide

  37. ೳಈతͳૢ࡞ͳ͘%ZOBNP%#্ͷ
    ϨίʔυΛ࡟আͰ͖Δ

    View full-size slide

  38. %ZOBNP%#4USFBNT

    View full-size slide

  39. w %ZOBNP%#্ͷΞΠςϜߋ৽͕

    Πϕϯτͱͯ͠ྲྀΕΔετϦʔϜ
    w ςʔϒϧͷσʔλมߋΛτϦΨʹ

    -BNCEBؔ਺ΛىಈͰ͖Δ
    w ϖΠϩʔυʹมߋલޙͷΞΠςϜ಺༰Λ

    ؚΊΔ͜ͱ͕Ͱ͖Δ

    View full-size slide

  40. %ZOBNP%#
    4USFBN
    -BNCEB
    ৽نΞΠςϜ௥Ճ
    طଘΞΠςϜߋ৽ ΞΠςϜ࡟আ

    View full-size slide

  41. EJBNPOEFYQPSUFS
    %ZOBNP%#55-

    %ZOBNP%#4USFBNT

    View full-size slide

  42. w 55-ʹ౸ୡͨ͠ΞΠςϜͷ࡟আ࣌ʹ

    ΞΠςϜͷ಺༰Λ౉ͯ͠-BNCEBؔ਺Λ
    ىಈ
    w -BNCEBؔ਺Ͱ4ʹॻ͖ग़͠

    View full-size slide

  43. ͜ͷΞΠςϜ͕55-ʹ౸ୡͯ͠

    ࡟আ͞Ε·ͨ͠
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …

    View full-size slide

  44. hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …

    View full-size slide

  45. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU

    View full-size slide

  46. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ
    w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ
    w 55-ܦաʹΑΓΞΠςϜ࡟আ
    w ΞΠςϜ࡟আΠϕϯτ͕4USFBNʹ௨஌
    w 4USFBN͔Β-BNCEBؔ਺ىಈ
    w -BNCEBؔ਺͕ΞΠςϜΛ4ʹॻࠐ

    View full-size slide

  47. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w 55-෇༩Ҏ֎ͷ
    ೳಈతͳૢ࡞͕ෆཁ
    w -BNCEBͷࣦഊ࣌ϦτϥΠ΋ࣗಈ
    w 4USFBNҎ֎ʹঢ়ଶΛ࣋ͨͳ͍
    w %ZOBNP%#ΞΠςϜ4Φϒ
    δΣΫτ

    View full-size slide

  48. 4
    EJBNPOE"11
    NBDLFSFM"11
    (FU0CKFDU
    )551
    )551

    View full-size slide

  49. "84ͷΞοϓσʔτʹΑ࣮ͬͯݱ

    View full-size slide

  50. ೥ͷ%ZOBNP%#Ξοϓσʔτ
    w 55-d
    w "VUP4DBMF %"9d
    w 71$&OEQPJOUd
    w (MPCBM5BCMFT ΦϯσϚϯυόοΫΞοϓd

    View full-size slide

  51. 4΁ͷߴස౓ॻ͖ࠐΈ

    View full-size slide

  52. w 4΁ͷߴස౓ॻ͖ࠐΈΛߦ͏৔߹͸ରԠ͕ඞཁ
    w ಺෦γϟʔσΟϯάରԠͷͨΊͷQSFpY෇༩
    w ٸܹͳ૿Ճͷ৔߹͸
    "84αϙʔτܦ༝Ͱͷ

    ࣄલରԠґཔ

    View full-size slide

  53. Amazon S3 όέοτͷϫʔΫϩʔυ͕ຖඵ 100
    ճͷ PUT/LIST/DELETE ϦΫΤετ·ͨ͸ຖඵ
    300 ճͷ GET ϦΫΤετΛසൟʹ௒͑Δ৔߹
    ͸ɺ࠷ߴͷύϑΥʔϚϯεͱεέʔϥϏϦςΟ
    Λ֬อ͢ΔͨΊɺ͜ͷτϐοΫͷΨΠυϥΠϯ
    ʹै͍ͬͯͩ͘͞ɻ
    ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲
    https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request-
    rate-perf-considerations.html

    View full-size slide

  54. EJBNPOEͰ4΁ͷॻ͖ࠐΈ͸ίϯελϯτʹൃੜ
    SQT͸༏ʹ௒͑Δ

    View full-size slide

  55. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ
    w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ
    w 55-ܦաʹΑΓΞΠςϜ࡟আ
    w ΞΠςϜ࡟আΠϕϯτ͕4USFBNʹ௨஌
    w 4USFBN͔Β-BNCEBؔ਺ىಈ
    w -BNCEBؔ਺͕ΞΠςϜΛ4ʹॻࠐ
    ࠶ܝ

    View full-size slide

  56. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w 55-෇༩Ҏ֎ͷ
    ೳಈతͳૢ࡞͕ෆཁ
    w -BNCEBͷࣦഊ࣌ϦτϥΠ΋ࣗಈ
    w %ZOBNP%#ͷΞΠςϜ4Φϒ
    δΣΫτ
    ࠶ܝ

    View full-size slide

  57. ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲
    https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request-
    rate-perf-considerations.html

    View full-size slide

  58. w ΦϒδΣΫτ໊ઌ಄ʹ.%ϋογϡͷ෇༩
    w ॻ͖ࠐΈ։࢝લʹαϙʔτʹରԠΛґཔ
    w ਺೔Φʔμʔͷ଴͕ͪ࣌ؒൃੜ͢Δ

    View full-size slide

  59. ա৒ͳϖʔεͰͷߋ৽͸)551YYΤϥʔ͕
    ൃੜ͢ΔʢΒ͍͠ʣ
    ϞχλϦϯάʂ

    View full-size slide

  60. "844ͷϞχλϦϯά

    View full-size slide

  61. w 4ͷඪ४$MPVE8BUDI.FUSJDT͸ݶఆత
    w όέοτͷ೔࣍ͷετϨʔδϝτϦΫε
    w ϦΫΤετͷ෼ੳ͸Ͱ͖ͳ͍
    w ΦϓγϣϯͰϦΫΤετͷϞχλϦϯά΋Մೳ
    w 4όέοτͷઃఆͰ༗ޮʹͰ͖Δ
    https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/
    cloudwatch-monitoring.html

    View full-size slide

  62. $MPVE8BUDI.BDLFSFM

    View full-size slide

  63. mackerel-plugin-aws-s3-requests
    [plugin.metrics.diamond-s3]
    command = "/path/to/mackerel-plugin-aws-s3-requests --bucket-
    name=XXX --region=ap-northeast-1 --filter-id YYY"
    ެࣜϓϥάΠϯͱͯ͠ϦϦʔε༧ఆ
    ͖ͬͱࠓिதʹ͸ʜ

    View full-size slide

  64. w (FU 1VUͦΕͧΕͷϦΫΤετ਺
    w ୹ظతͳόʔετɺ௕ظతͳτϨϯυ
    w YYΤϥʔͷ਺
    ؂ࢹϧʔϧ

    View full-size slide

  65. w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷ಺෦࣮૷
    w ߴස౓4ॻ͖ࠐΈ
    w ཹҙࣄ߲ϞχλϦϯά

    View full-size slide

  66. ༗Γ೉͏͍͟͝·ͨ͠ʂ

    View full-size slide