AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo

2018/02/05 https://mackerelio.connpass.com/event/76678/

Transcript

1. "84Ͱ࣮ݱͨ͠.BDLFSFM
   ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷཪଆ
   
   .BDLFSFM.FFUVQ5PLZP
   JEBTUKʢגࣜձࣾ͸ͯͳʣ
   

   View Slide

2. w ͸ͯͳΞϓϦέʔγϣϯΤϯδχΞ
   w ͸ͯͳϒοΫϚʔΫ͸ͯͳϒϩάFUDʜ
   w डୗαʔϏε
   w .BDLFSFM d
   
   w αʔόαΠυΤʔδΣϯτத৺
   JEBTUK
   

   View Slide

3. w NBDLFSFMBHFOUTZTUFNEରԠ
   w NBDLFSFMQMVHJOBXT͍͔ͭ͘
   w -*/&5XJMJP࿈ܞ
   w "84Ҡߦ࣌ܥྻσʔλϕʔεͷ࡮৽
   w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘ
   w ؂ࢹϧʔϧͷૢ࡞Λ௨஌νϟϯωϧʹ௨஌
   w ʜ
   JEBTUK
   

   View Slide

4. ຊ೔ͷ࿩୊
   

   View Slide

5. w .BDLFSFM࣌ܥྻ%#ͷ঺հ
   w ෼ཻ౓௕ظอଘͷٕज़എܠ
   w αʔόϨεϛυϧ΢ΣΞͷߏஙࣄྫ
   w %ZOBNP%#4USFBNT-BNCEB
   w ߴεϧʔϓοτ4ॻ͖ࠐΈ
   

   View Slide

6. લఏ
   

   View Slide

7. ࣌ܥྻσʔλ
   

   View Slide

8. ࣌ܥྻσʔλ
   hatena.mackerel.host1.cpu.user
   2018/02/01T21:15:00Z 44.00
   2018/02/01T21:16:00Z 6.00
   2018/02/05T21:17:00Z 8.00
   … …
   

   View Slide

9. ࣌ܥྻσʔλϕʔε
   hatena.mackerel.host1.cpu.user
   2018/02/01T21:15:00Z 44.00
   2018/02/01T21:16:00Z 6.00
   2018/02/05T21:17:00Z 8.00
   hatena.mackerel.host1.cpu.user
   2018/02/01T21:15:00Z 44.00
   2018/02/01T21:16:00Z 6.00
   2018/02/05T21:17:00Z 8.00
   hatena.mackerel.host1.cpu.user
   2018/02/01T21:15:00Z 44.00
   2018/02/01T21:16:00Z 6.00
   2018/02/05T21:17:00Z 8.00
   hatena.mackerel.host1.cpu.user
   2018/02/01T21:15:00Z 44.00
   2018/02/01T21:16:00Z 6.00
   2018/02/05T21:17:00Z 8.00
   … …
   

   View Slide

10. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud
    

    View Slide

11. View Slide

12. NI
    NE
    IE
    
    
    #FGPSF
    E
    

    View Slide

13. NE
    NE
    IE
    E
    
    "GUFS
    

    View Slide

14. View Slide

15. .BDLFSFMͷ
    ࣌ܥྻσʔλϕʔε
    

    View Slide

16. Ҡߦલ(SBQIJUF
    IUUQTHSBQIJUFBQQPSH
    

    View Slide

17. http://blog.yuuk.io/entry/high-performance-graphite
    

    View Slide

18. ҠߦޙEJBNPOE
    ಺੡ίʔυωʔϜ
    
    

    View Slide

19. IUUQCMPHZVVLJPFOUSZUIFSFCVJMEPGUTECPODMPVE
    ࣌ܥྻσʔλϕʔεͱ͍͏֓೦ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ
    

    View Slide

20. IUUQJUDIZOZIBUFOBCMPHDPNFOUSZ
    4FSWFSMFTTDPOG5PLZPͰʰαʔόϨεΞʔΩςΫνϟʹ
    ΑΔ࣌ܥྻσʔλϕʔεͷߏஙͱ؂ࢹʱͱ͍͏ൃද͖ͯ͠·ͨ͠
    

    View Slide

21. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud
    3FEJT$MVTUFS &$
    
    

    View Slide

22. ίϯηϓτ
    w εέʔϧ͢ΔΞʔΩςΫνϟ
    w ϚωʔδυαʔϏεʹΑΔӡ༻লྗԽ
    w ΞΫηεಛੑʹԠͨ͡֊૚ܕσʔλετΞ
    w 3FEJT %ZOBNP%# 4
    

    View Slide

23. ετϨʔδ *0ίετ ϨΠςϯγ ༰ྔίετ
    3FEJT ˕ ˕ ˚
    %ZOBNP%# ˓ ˓ ˓
    4 ˚ ˚ ˕
    σʔλετϨʔδ
    

    View Slide

24. %ZOBNP%#
    w ϑϧϚωʔδυ/P42-
    w ҆ఆɾߴύϑΥʔϚϯεɾεέʔϥϏϦςΟ
    w खࠒͳίετ
    w σʔλ༰ྔɺεϧʔϓοτ
    w ʢͱ͸͍͑අ༻Λແࢹ͸Ͱ͖ͳ͍ʣ
    

    View Slide

25. 3FEJT $MVTUFS
    
    w ΠϯϝϞϦ,74
    w *0ίετ͕ %ZOBNP%#ΑΓ
    खܰ
    w σʔλྔΛࡹͨ͘Ί3FEJT$MVTUFS
    w ࠓͷͱ͜Ζ&$
    

    View Slide

26. 4
    w ,74తʹར༻
    w ༰ྔίετ͸ඇৗʹ཈͑ΒΕΔ
    w ϨΠςϯγɾ*0ίετ͸ྼΔ
    w ॻ͖ࠐΈճ਺ΛݮΒ͍ͨ͠
    

    View Slide

27. w %ZOBNP%#
    w ϝΠϯ
    w 3FEJT
    w ߋ৽ϦΫΤετΛݮΒͨ͢ΊͷόοϑΝ
    w 4
    w طଘͷอ࣋ظؒͰ͸
    ར༻͠ͳ͍
    ֊૚ܕσʔλετΞ
    
    

    View Slide

28. NI
    NE
    IE
    
    
    ֊૚ܕσʔλετΞ
    
    %ZOBNP%#
    3FEJT
    E
    

    View Slide

29. ෼ཻ౓ͷ௕ظอଘ
    

    View Slide

30. w σʔλྔ͕େ͖͘๲ΒΉ ഒ
    
    w ݹ͍σʔλͷϨΠςϯγཁ݅͸গ͠؇͍
    w ৽͍͠σʔλ΄Ͳසൟʹ͸ࢀর͞Εͳ͍
    w ݹ͍σʔλͷ্ॻ͖ߋ৽͸ൃੜ͠ͳ͍
    

    View Slide

31. w ݹ͍σʔλΛ4ʹ഑ஔ
    w طଘͷอ࣋ظؒͷύϑΥʔϚϯεΛҡ࣋
    w ॻ͖ࠐΈճ਺ΛݮΒ͢
    ֊૚ܕσʔλετΞ
    
    

    View Slide

32. NE
    NE
    IE
    
    
    ֊૚ܕσʔλετΞ
    
    %ZOBNP%#
    3FEJT
    4
    E
    

    View Slide

33. View Slide

34. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    

    View Slide

35. %ZOBNP%#55-
    

    View Slide

36. %ZOBNP%#ͷΞΠςϜ͝ͱʹ༗ޮظݶ
    55-
    Λࢦఆͯ͠ɺࣗಈతʹ࡟আͤ͞Δ
    

    View Slide

37. %ZOBNP%#
    hatena.mackerel.host1.cpu.user
    2018/02/03 00:00~03:59
    ExpireAt: 2018/02/04 05:00
    2018/02/02 16:00~19:59
    ExpireAt: 2018/02/03 21:00
    hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user
    
    hatena.mackerel.host1.cpu.user
    2018/02/02 20:00~23:59
    ExpireAt: 2018/02/04 01:00
    

    View Slide

38. %ZOBNP%#
    hatena.mackerel.host1.cpu.user
    2018/02/03 00:00~03:59
    ExpireAt: 2018/02/04 05:00
    hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user
    
    hatena.mackerel.host1.cpu.user
    2018/02/02 20:00~23:59
    ExpireAt: 2018/02/04 01:00
    

    View Slide

39. %ZOBNP%#
    hatena.mackerel.host1.cpu.user
    2018/02/03 00:00~03:59
    ExpireAt: 2018/02/04 05:00
    hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user
    
    

    View Slide

40. hatena.mackerel.host1.cpu.user
    2018/02/03 04:00~07:59
    ExpireAt: 2018/02/04 09:00
    hatena.mackerel.host1.cpu.user
    
    %ZOBNP%#
    

    View Slide

41. %ZOBNP%#
    hatena.mackerel.host1.cpu.user
    
    

    View Slide

42. %ZOBNP%#
    
    

    View Slide

43. ೳಈతͳૢ࡞ͳ͘%ZOBNP%#্ͷ
    ϨίʔυΛ࡟আͰ͖Δ
    

    View Slide

44. %ZOBNP%#4USFBNT
    

    View Slide

45. w %ZOBNP%#্ͷΞΠςϜߋ৽͕

    Πϕϯτͱͯ͠ྲྀΕΔετϦʔϜ
    w ςʔϒϧͷσʔλมߋΛτϦΨʹ

    -BNCEBؔ਺ΛىಈͰ͖Δ
    w ϖΠϩʔυʹมߋલޙͷΞΠςϜ಺༰Λ

    ؚΊΔ͜ͱ͕Ͱ͖Δ
    

    View Slide

46. %ZOBNP%#
    4USFBN
    -BNCEB
    ৽نΞΠςϜ௥Ճ
    طଘΞΠςϜߋ৽ ΞΠςϜ࡟আ
    

    View Slide

47. EJBNPOEFYQPSUFS
    %ZOBNP%#55-
    
    %ZOBNP%#4USFBNT
    

    View Slide

48. w 55-ʹ౸ୡͨ͠ΞΠςϜͷ࡟আ࣌ʹ

    ΞΠςϜͷ಺༰Λ౉ͯ͠-BNCEBؔ਺Λ
    ىಈ
    w -BNCEBؔ਺Ͱ4ʹॻ͖ग़͠
    

    View Slide

49. ͜ͷΞΠςϜ͕55-ʹ౸ୡͯ͠

    ࡟আ͞Ε·ͨ͠
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …
    

    View Slide

50. hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …
    

    View Slide

51. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    

    View Slide

52. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ
    w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ
    w 55-ܦաʹΑΓΞΠςϜ࡟আ
    w ΞΠςϜ࡟আΠϕϯτ͕4USFBNʹ௨஌
    w 4USFBN͔Β-BNCEBؔ਺ىಈ
    w -BNCEBؔ਺͕ΞΠςϜΛ4ʹॻࠐ
    

    View Slide

53. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w 55-෇༩Ҏ֎ͷ
    ೳಈతͳૢ࡞͕ෆཁ
    w -BNCEBͷࣦഊ࣌ϦτϥΠ΋ࣗಈ
    w 4USFBNҎ֎ʹঢ়ଶΛ࣋ͨͳ͍
    w %ZOBNP%#ΞΠςϜ4Φϒ
    δΣΫτ
    

    View Slide

54. 4
    EJBNPOE"11
    NBDLFSFM"11
    (FU0CKFDU
    )551
    )551
    

    View Slide

55. γϯϓϧ
    

    View Slide

56. "84ͷΞοϓσʔτʹΑ࣮ͬͯݱ
    

    View Slide

57. ೥ͷ%ZOBNP%#Ξοϓσʔτ
    w 55-d
    w "VUP4DBMF %"9d
    w 71$&OEQPJOUd
    w (MPCBM5BCMFT ΦϯσϚϯυόοΫΞοϓd
    

    View Slide

58. 4΁ͷߴස౓ॻ͖ࠐΈ
    

    View Slide

59. w 4΁ͷߴස౓ॻ͖ࠐΈΛߦ͏৔߹͸ରԠ͕ඞཁ
    w ಺෦γϟʔσΟϯάରԠͷͨΊͷQSFpY෇༩
    w ٸܹͳ૿Ճͷ৔߹͸
    "84αϙʔτܦ༝Ͱͷ

    ࣄલରԠґཔ
    

    View Slide

60. Amazon S3 όέοτͷϫʔΫϩʔυ͕ຖඵ 100
    ճͷ PUT/LIST/DELETE ϦΫΤετ·ͨ͸ຖඵ
    300 ճͷ GET ϦΫΤετΛසൟʹ௒͑Δ৔߹
    ͸ɺ࠷ߴͷύϑΥʔϚϯεͱεέʔϥϏϦςΟ
    Λ֬อ͢ΔͨΊɺ͜ͷτϐοΫͷΨΠυϥΠϯ
    ʹै͍ͬͯͩ͘͞ɻ
    ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲
    https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request-
    rate-perf-considerations.html
    

    View Slide

61. EJBNPOEͰ4΁ͷॻ͖ࠐΈ͸ίϯελϯτʹൃੜ
    SQT͸༏ʹ௒͑Δ
    
    

    View Slide

62. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ
    w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ
    w 55-ܦաʹΑΓΞΠςϜ࡟আ
    w ΞΠςϜ࡟আΠϕϯτ͕4USFBNʹ௨஌
    w 4USFBN͔Β-BNCEBؔ਺ىಈ
    w -BNCEBؔ਺͕ΞΠςϜΛ4ʹॻࠐ
    ࠶ܝ
    

    View Slide

63. 55-&YQJSF
    %ZOBNP%# %%#4USFBN -BNCEB
    4
    1VU0CKFDU
    w 55-෇༩Ҏ֎ͷ
    ೳಈతͳૢ࡞͕ෆཁ
    w -BNCEBͷࣦഊ࣌ϦτϥΠ΋ࣗಈ
    w %ZOBNP%#ͷΞΠςϜ4Φϒ
    δΣΫτ
    ࠶ܝ
    

    View Slide

64. ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲
    https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request-
    rate-perf-considerations.html
    

    View Slide

65. w ΦϒδΣΫτ໊ઌ಄ʹ.%ϋογϡͷ෇༩
    w ॻ͖ࠐΈ։࢝લʹαϙʔτʹରԠΛґཔ
    w ਺೔Φʔμʔͷ଴͕ͪ࣌ؒൃੜ͢Δ
    

    View Slide

66. ա৒ͳϖʔεͰͷߋ৽͸)551YYΤϥʔ͕
    ൃੜ͢ΔʢΒ͍͠ʣ
    ϞχλϦϯάʂ
    

    View Slide

67. "844ͷϞχλϦϯά
    

    View Slide

68. w 4ͷඪ४$MPVE8BUDI.FUSJDT͸ݶఆత
    w όέοτͷ೔࣍ͷετϨʔδϝτϦΫε
    w ϦΫΤετͷ෼ੳ͸Ͱ͖ͳ͍
    w ΦϓγϣϯͰϦΫΤετͷϞχλϦϯά΋Մೳ
    w 4όέοτͷઃఆͰ༗ޮʹͰ͖Δ
    https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/
    cloudwatch-monitoring.html
    

    View Slide

69. View Slide

70. $MPVE8BUDI.BDLFSFM
    

    View Slide

71. mackerel-plugin-aws-s3-requests
    [plugin.metrics.diamond-s3]
    command = "/path/to/mackerel-plugin-aws-s3-requests --bucket-
    name=XXX --region=ap-northeast-1 --filter-id YYY"
    ެࣜϓϥάΠϯͱͯ͠ϦϦʔε༧ఆ
    ͖ͬͱࠓिதʹ͸ʜ
    
    

    View Slide

72. View Slide

73. w (FU 1VUͦΕͧΕͷϦΫΤετ਺
    w ୹ظతͳόʔετɺ௕ظతͳτϨϯυ
    w YYΤϥʔͷ਺
    ؂ࢹϧʔϧ
    

    View Slide

74. ͓ΘΓʹ
    

    View Slide

75. w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷ಺෦࣮૷
    w ߴස౓4ॻ͖ࠐΈ
    w ཹҙࣄ߲ϞχλϦϯά
    

    View Slide

76. ༗Γ೉͏͍͟͝·ͨ͠ʂ
    

    View Slide