Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11

1/23 Tokyo: https://hatena.connpass.com/event/115458/
1/30 Kyoto : https://hatena.connpass.com/event/115460/

Reference:
Graphite: http://graphite.readthedocs.org/
Diamond:
- https://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud
- https://itchyny.hatenablog.com/entry/2017/11/06/090000
- https://astj.hatenablog.com/entry/2018/02/06/175902
Postmortem for Data Lost: https://mackerel.io/ja/blog/entry/2017/08/15/113803

Transcript

1. MackerelΛΦϯϓϨϛε͔Β
   AWSʹҠ͔ͯ͠Βͷ
   1೥൒ΛৼΓฦΔ
   Hatena Engineer Seminar #11
   id:astj
   

   View Slide

2. id:astjʢ͋͞ͱ͐͡ʔʣ
   ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ
   ৽ଔ (2014/04)
   Mackerel ։ൃνʔϜ (2016/08~)
   Mackerel ςοΫϦʔυ (2018/05~)
   

   View Slide

3. Mackerel
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. • 2014೥ϦϦʔεͷʮαʔόʔ؂ࢹαʔϏεʯ
   • αʔόʔαΠυ: Scala / Go
   • ػցֶशύʔτʢ։ൃதʂʣ: Python
   • AWS Lambda : NodeJS
   

   View Slide

8. ຊ೔ͷத਎
   • ΦϯϓϨϛε͔ΒAWSʹҠߦͨ͜͠ͱ
   • Ҡߦͯ͠ಘΒΕͨ͜ͱɾมΘͬͨ͜ͱ
   

   View Slide

9. ༧උ஌ࣝ / ͝঺հ
   

   View Slide

10. On-Premises
    DC
    Office
    AWS
    Account A
    (Main)
    AWS
    Account B
    AWS
    Account C
    Dedicated
    Hosting
    ͸ͯͳͷΠϯϑϥߏ੒
    …
    

    View Slide

11. ͸ͯͳͷΠϯϑϥߏ੒
    On-Premises
    DC
    Office
    AWS
    Account A
    AWS
    Account B
    AWS
    Account C
    Dedicated
    Hosting
    …
    ΦϯϓϨϛεͷDCͱAWSʢͳͲʣΛซ༻
    ྺ࢙తܦҢͰڞ௨ͷAWSΞΧ΢ϯτ
    αʔϏεʹΑͬͯ͸ಠཱΞΧ΢ϯτ
    

    View Slide

12. Mackerelͷߏ੒
    RDBMS
    (Postgres) Redis
    Tsdb
    (nginx)
    (nginx)
    reverse
    proxy
    (nginx) (nginx)
    (nginx)
    app
    (Scala)
    (nginx)
    (nginx)
    Sub-
    systems
    

    View Slide

13. hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …
    ࣌ܥྻσʔλ
    

    View Slide

14. ࣌ܥྻσʔλϕʔε
    • ΦϯϓϨϛε࣌୅: Graphite
    • http://graphite.readthedocs.org/
    • AWS Ҡߦޙ: ಺੡ϓϩμΫτ
    • ίʔυωʔϜ "diamond"
    

    View Slide

15. ΦϯϓϨϛε͔ΒAWS΁
    

    View Slide

16. ॳظͷ Mackerel ͷߏ੒
    • ϩʔϯν౰ॳ͸ΦϯϓϨϛε
    • جຊతʹ͸XenͰԾ૝Խͨ͠VMӡ༻
    • Tsdb΍RDBͰ͸ϋΠεϖοΫͳ෺ཧαʔόΛ

    Ծ૝Խͤͣར༻
    • ioDrive (~380k IOPS Ͱେ༰ྔͰߴ͍)
    

    View Slide

17. ՝୊ײ
    • αʔϏε੒௕ʹ൐͍՝୊͕ग़͖ͯͨ
    • Graphiteͷӡ༻͕͍ͨ΁Μ
    • σΟεΫ͕΋Γ΋Γຒ·Δɾ৑௕Խߏ੒ͷӡ༻
    • εέʔϧΞ΢τʹ෺ཧαʔόͷௐୡ
    

    View Slide

18. => AWS
    

    View Slide

19. ʮ࣍ੈ୅MackerelϓϩδΣΫτʯ
    

    View Slide

20. AWSҠߦ
    • TsdbͷΩϟύγςΟ֬อͷॊೈԽ
    • ηΩϡϦςΟରࡦج൫ͷॆ࣮
    • ܭը => Ҡߦ·Ͱ͓Αͦ1೥
    • Tsdbͷݕূɾ։ൃʹ࣌ؒͱ޻਺Λେׂ͖͍ͨ͘
    

    View Slide

21. TsdbͷAWSҠߦ
    • ߴεϖοΫ෺ཧαʔόಉ༷ͷӡ༻͸ݫ͍͠
    • EBSͰٻΊΒΕͨiops͸ग़ͤͳ͍
    • ΦϯϓϨϛεͷ࣌఺Ͱӡ༻ෛՙ͸͋ͬͨ
    

    View Slide

22. ࣍ੈ୅Tsdb: diamond
    DynamoDBΛ࣠ʹͨ͠ϚωʔδυαʔϏε

    த৺ͷ࣌ܥྻσʔλϕʔεΛ։ൃ
    • https://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud
    • https://itchyny.hatenablog.com/entry/2017/11/06/090000
    • https://astj.hatenablog.com/entry/2018/02/06/175902
    

    View Slide

23. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud
    

    View Slide

24. Ҡߦϓϩηε
    

    View Slide

25. "࣍ੈ୅Ͱ΍Γ͍ͨ͜ͱ"
    ແݶʹͰͯ͘Δ
    

    View Slide

26. ·ͣ͸EC2ʹશ෦৐ͤΔ
    • είʔϓΛߜΔ
    • ʢͱ͸͍͑Tsdb͸diamondʹҠߦ͢Δʣ
    • ੾Γ෼͚Մೳͳ෦෼͸͞Βʹ੾Γ෼͚Δ
    • ϦϦʔεཻ౓Λۃྗখ͘͢͞Δ
    • ιϑτ΢ΣΞ։ൃͱಉ༷ʢܧଓతσϦόϦʔʁʣɹ
    

    View Slide

27. ࡞ઓ
    

    View Slide

28. On-Premises DC AWS
    nginx
    app
    db
    redis tsdb (Graphite)
    subsystem
    Πϝʔδਤ
    STEP0 (ॳظঢ়ଶ)
    

    View Slide

29. On-Premises DC AWS
    nginx
    app
    db
    redis tsdb (Graphite)
    subsystem
    tsdb (diamond)
    Πϝʔδਤ
    STEP1 (tsdb)
    

    View Slide

30. STEP1
    • ࣌ܥྻσʔλϕʔεͷ৽چ྆ํ΁ͷॻ͖ࠐΈ
    • ৽چͷࠩҟΛࣄલʹ֬ೝ͢Δ
    • ͜ͷ͋ͱ΋͠͹Β͘͸৽چ྆ํʹॻ͖ଓ͚Δ
    • ສ͕Ұͷ࣌ͷ੾Γ໭͠ʹඋ͑Δ
    

    View Slide

31. On-Premises DC AWS
    nginx
    app
    db
    redis tsdb (Graphite)
    subsystem
    tsdb (diamond)
    Πϝʔδਤ
    STEP1 (tsdb)
    

    View Slide

32. On-Premises DC AWS
    tsdb (Graphite)
    subsystem app
    db
    redis tsdb (diamond)
    Πϝʔδਤ
    HTTP
    STEP2-1 (app etc)
    nginx
    

    View Slide

33. On-Premises DC AWS
    tsdb (Graphite)
    subsystem
    nginx
    app
    db
    redis tsdb (diamond)
    Πϝʔδਤ
    HTTP
    STEP2-2 (app etc)
    

    View Slide

34. STEP2
    • ΞϓϦέʔγϣϯͱಉظతʹ௨৴͢ΔՕॴʢେ൒ʣ
    • subsystem͸ΦϯϓϨϛεͷ··
    • HTTPܦ༝ͷඇಉظ௨৴ͳͷͰ༨༟͕͋Δ
    • diamondͷ͓൸࿐໨΋͜ͷλΠϛϯά
    

    View Slide

35. On-Premises DC AWS
    tsdb (Graphite)
    subsystem
    nginx
    app
    db
    redis tsdb (diamond)
    Πϝʔδਤ
    HTTP
    STEP2 (app etc)
    

    View Slide

36. On-Premises DC AWS
    tsdb (Graphite)
    nginx
    app
    db
    redis tsdb (diamond)
    subsystem
    Πϝʔδਤ
    STEP3 (subsystem)
    

    View Slide

37. STEP3
    • HTTPܦ༝Ͱ௨৴͢ΔαϒγεςϜͷҠઃ
    • ֎ܗ؂ࢹͷΫϩʔϥΛؚΉͷͰग़ޱIP͕มߋ
    • ࣄલࠂ஌ɻ͝ڠྗ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide

38. On-Premises DC AWS
    tsdb (Graphite)
    nginx
    app
    db
    redis tsdb (diamond)
    subsystem
    Πϝʔδਤ
    STEP3 (subsystem)
    

    View Slide

39. On-Premises DC AWS
    nginx
    app
    db
    redis tsdb (diamond)
    subsystem
    Πϝʔδਤ
    STEP4 (DONE!)
    

    View Slide

40. Ҡߦ׬ྃ
    • ύϑΥʔϚϯεɺඅ༻ͳͲ੾Γ໭͠ͳ͠ͷ൑அ
    • STEP2͔Β1ϲ݄ଓ͚ͨฒߦՔಇΛऴྃ
    

    View Slide

41. ࣮ࡍͷҠߦ
    

    View Slide

42. STEP1 (tsdb)
    • ෛՙࢼݧͯ͠ϦϦʔεՄೳͷ൑அ
    • astj͸STEP2͕৺഑ͩͬͨ
    • ৽ن։ൃ = ౰વաڈͷՔಇ࣮੷͸ͳ͍
    • ࢥͬͨΑ͏ʹύϑΥʔϚϯεग़Δͷ͔ʁ
    

    View Slide

43. STEP2 (app etc)
    • ແఀࢭͷ੾Γସ͑Λ࣮ࢪ
    • ੾ସ࣌ʹΠϯελϯεো֐ͰҰ෦σʔλফࣦ
    • ޙ೔ௐࠪ಺༰ͱ࠶ൃ๷ࢭࡦΛެ։
    • https://mackerel.io/ja/blog/entry/2017/08/15/113803
    

    View Slide

44. STEP2 (app etc)
    • ୹࣌ؒ෼ͷΈͱ͸͍͑σʔλফࣦ
    • Ϣʔβ͞Μʹ͝໎࿭Λ͓͔͚ͯ͠͠·ͬͨ
    • Ұํɺdiamond ʹؔͯ͠͸໰୊ͳ͘Քಇ
    • ؼ୐͔ͯ͠Βײ֒ʹ;͚͍ͬͯͨ
    

    View Slide

45. STEP3 (subsystem)
    • ͕ͭͭͳ͘੾Γସ͑
    • ʢastj͸͜ͷ೔ٳՋͰͨ͠…ʣ
    

    View Slide

46. Ҡߦ׬ྃ
    

    View Slide

47. Ҡߦ׬ྃʁ
    

    View Slide

48. Ҡߦ׬ྃʁ
    • ʮ·ͣ͸EC2ʹ৐ͤͨʯஈ֊
    • ϚωʔδυαʔϏε (ex. RDS) ΁ͷҠߦ༨஍
    • “࣍ੈ୅Mackerel”ͷεςοϓΞοϓɹ
    

    View Slide

49. RDSͷ࣮ྫ࿩࣌ؒ͢͸
    ଟ෼ͳ͍Ͱ͢
    

    View Slide

50. RDSҠߦ
    • EC2Ҡߦ౰ॳ͸ PostgreSQL 9.3 (EC2)
    • ϚωʔδυαʔϏεԽͰ DB ӡ༻ίετΛԼ͛Δ
    • EC2Ҡߦ࣌ͷΑ͏ͳো֐Λආ͚Δ͜ͱʹ΋ܨ͕Δ
    • ݕ౼ࣄ߲: Aurora for PostgreSQL
    

    View Slide

51. RDSҠߦ
    • 9.3 (EC2) => 9.3 (RDS) => 9.6 (RDS)
    • 2ճʹ෼͚ͯҠߦ
    • ͦΕͧΕఀࢭϝϯςφϯεΛ࣮ࢪ
    • ͝ཧղ/͝ڠྗ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide

52. ҠߦશମΛৼΓฦΔ
    • Ҡߦखॱͷ෼ׂ͸ਖ਼ղ
    • ΦϖϨʔγϣϯͱͯ͠͸DB੾ସ͕ϔϏʔ
    • σʔλফࣦͰ͸͝໎࿭Λ͓͔͚͠·ͨ͠
    

    View Slide

53. Ҡߦ͔ͯ͠ΒͷมԽ
    

    View Slide

54. ΩϟύγςΟ֬อͷॊೈੑ
    • tsdbͷεέʔϥϏϦςΟͷݒ೦ͷղফ
    • େن໛ͳεέʔϧΞ΢τ/Πϯ
    • ؾܰʹԾ૝Ϛγϯͷ࡞੒/ഇغ͕ߦ͑Δ
    • ΠϯϑϥΤϯδχΞ΁ͷʮ͓ئ͍ʯ͕ෆཁʹ
    • ΠϯϑϥͷݕূɾߏஙͷεϐʔυΞοϓ
    

    View Slide

55. ӡ༻ͷखؒͷݮগ
    • ෺ཧαʔόΛҙࣝͨ͠ӡ༻͔Βͷղ์
    • ϚωʔδυαʔϏεԽ
    • ετϨʔδ૚ͷऔΓճ͠ͷྑ͞
    • EBS snapshot
    

    View Slide

56. ΞʔΩςΫνϟબఆͷࣗ༝౓޲্
    • ӡ༻ମྗͷඞཁʢͰ͋Ζ͏ʣίϯϙʔωϯτΛ

    ϚωʔδυͰར༻ग़དྷΔ
    • Kinesis Data Streams, Lambda, 

    DynamoDB, ...
    • ৽نߏஙՕॴͰࣗવͱ࠾༻ग़དྷΔ
    • ػցֶशػೳͰͷ AWS Batch, ...
    

    View Slide

57. ศར
    

    View Slide

58. ͱ͸͍͑
    

    View Slide

59. ཧ૝ڷʹདྷͨΘ͚Ͱ͸ͳ͍
    

    View Slide

60. • αʔόɾωοτϫʔΫো֐ࣗମ͸ଘࡏ
    • Ҿ͖ଓ͖ؤு͍͖ͬͯ·͠ΐ͏
    • Ծ૝Ϛγϯ͸Ծ૝Ϛγϯ
    • Ҿ͖ଓ͖ؤு͍͖ͬͯ·͠ΐ͏
    • ΑΓந৅౓ͷߴ͍Πϯϑϥʁ
    

    View Slide

61. • ϚωʔδυαʔϏεʹ͋Θͤͨઃܭɾӡ༻
    • ٕज़બఆ
    • ʮ࢖͑ͦ͏͚ͩͲϋϚΒͳ͍ʯ
    • AWS APIΛར༻ͨ͠ӡ༻ઃܭ
    

    View Slide

62. View Slide

63. ·ͱΊ
    

    View Slide

64. AWSʹҠߦͨ͠ɹ
    • ΩϟύγςΟΛओͳཁ݅ͱͯ͠AWSʹҠߦ
    • ஈ֊Λ౿ΜͰҠߦͨ͠
    • Tsdb͸DynamoDBத৺ʹ಺੡
    • ͦͷଞ͸Ұ୴EC2্ʹ
    • ͦͷޙঃʑʹϚωʔδυԽ
    

    View Slide

65. AWSʹҠߦͯ͠
    • TsdbͷΩϟύγςΟʹک͑Δੜ׆ʹผΕ
    • ϚωʔδυαʔϏεΛੵۃతʹऔΓೖΕΒΕΔΑ͏ʹ
    • ӡ༻ͷखؒΛݮΒ͢
    • ։ൃΞʔΩςΫνϟબ୒ࢶΛ૿΍͢
    • ϓϩμΫτͷมߋ଎౓ΛߴΊΒΕ͍ͯΔ
    

    View Slide

66. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide