Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11

astj
January 23, 2019

Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11

astj

January 23, 2019
Tweet

More Decks by astj

Other Decks in Technology

Transcript

  1. MackerelΛΦϯϓϨϛε͔Β
    AWSʹҠ͔ͯ͠Βͷ
    1೥൒ΛৼΓฦΔ
    Hatena Engineer Seminar #11
    id:astj

    View full-size slide

  2. id:astjʢ͋͞ͱ͐͡ʔʣ
    ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ
    ৽ଔ (2014/04)
    Mackerel ։ൃνʔϜ (2016/08~)
    Mackerel ςοΫϦʔυ (2018/05~)

    View full-size slide

  3. • 2014೥ϦϦʔεͷʮαʔόʔ؂ࢹαʔϏεʯ
    • αʔόʔαΠυ: Scala / Go
    • ػցֶशύʔτʢ։ൃதʂʣ: Python
    • AWS Lambda : NodeJS

    View full-size slide

  4. ຊ೔ͷத਎
    • ΦϯϓϨϛε͔ΒAWSʹҠߦͨ͜͠ͱ
    • Ҡߦͯ͠ಘΒΕͨ͜ͱɾมΘͬͨ͜ͱ

    View full-size slide

  5. ༧උ஌ࣝ / ͝঺հ

    View full-size slide

  6. On-Premises
    DC
    Office
    AWS
    Account A
    (Main)
    AWS
    Account B
    AWS
    Account C
    Dedicated
    Hosting
    ͸ͯͳͷΠϯϑϥߏ੒

    View full-size slide

  7. ͸ͯͳͷΠϯϑϥߏ੒
    On-Premises
    DC
    Office
    AWS
    Account A
    AWS
    Account B
    AWS
    Account C
    Dedicated
    Hosting

    ΦϯϓϨϛεͷDCͱAWSʢͳͲʣΛซ༻
    ྺ࢙తܦҢͰڞ௨ͷAWSΞΧ΢ϯτ
    αʔϏεʹΑͬͯ͸ಠཱΞΧ΢ϯτ

    View full-size slide

  8. Mackerelͷߏ੒
    RDBMS
    (Postgres) Redis
    Tsdb
    (nginx)
    (nginx)
    reverse
    proxy
    (nginx) (nginx)
    (nginx)
    app
    (Scala)
    (nginx)
    (nginx)
    Sub-
    systems

    View full-size slide

  9. hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    hatena.mackerel.host1.cpu.user
    2018/02/01T21:15:00Z 44.00
    2018/02/01T21:16:00Z 6.00
    2018/02/05T21:17:00Z 8.00
    … …
    ࣌ܥྻσʔλ

    View full-size slide

  10. ࣌ܥྻσʔλϕʔε
    • ΦϯϓϨϛε࣌୅: Graphite
    • http://graphite.readthedocs.org/
    • AWS Ҡߦޙ: ಺੡ϓϩμΫτ
    • ίʔυωʔϜ "diamond"

    View full-size slide

  11. ΦϯϓϨϛε͔ΒAWS΁

    View full-size slide

  12. ॳظͷ Mackerel ͷߏ੒
    • ϩʔϯν౰ॳ͸ΦϯϓϨϛε
    • جຊతʹ͸XenͰԾ૝Խͨ͠VMӡ༻
    • Tsdb΍RDBͰ͸ϋΠεϖοΫͳ෺ཧαʔόΛ

    Ծ૝Խͤͣར༻
    • ioDrive (~380k IOPS Ͱେ༰ྔͰߴ͍)

    View full-size slide

  13. ՝୊ײ
    • αʔϏε੒௕ʹ൐͍՝୊͕ग़͖ͯͨ
    • Graphiteͷӡ༻͕͍ͨ΁Μ
    • σΟεΫ͕΋Γ΋Γຒ·Δɾ৑௕Խߏ੒ͷӡ༻
    • εέʔϧΞ΢τʹ෺ཧαʔόͷௐୡ

    View full-size slide

  14. ʮ࣍ੈ୅MackerelϓϩδΣΫτʯ

    View full-size slide

  15. AWSҠߦ
    • TsdbͷΩϟύγςΟ֬อͷॊೈԽ
    • ηΩϡϦςΟରࡦج൫ͷॆ࣮
    • ܭը => Ҡߦ·Ͱ͓Αͦ1೥
    • Tsdbͷݕূɾ։ൃʹ࣌ؒͱ޻਺Λେׂ͖͍ͨ͘

    View full-size slide

  16. TsdbͷAWSҠߦ
    • ߴεϖοΫ෺ཧαʔόಉ༷ͷӡ༻͸ݫ͍͠
    • EBSͰٻΊΒΕͨiops͸ग़ͤͳ͍
    • ΦϯϓϨϛεͷ࣌఺Ͱӡ༻ෛՙ͸͋ͬͨ

    View full-size slide

  17. ࣍ੈ୅Tsdb: diamond
    DynamoDBΛ࣠ʹͨ͠ϚωʔδυαʔϏε

    த৺ͷ࣌ܥྻσʔλϕʔεΛ։ൃ
    • https://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud
    • https://itchyny.hatenablog.com/entry/2017/11/06/090000
    • https://astj.hatenablog.com/entry/2018/02/06/175902

    View full-size slide

  18. http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud

    View full-size slide

  19. Ҡߦϓϩηε

    View full-size slide

  20. "࣍ੈ୅Ͱ΍Γ͍ͨ͜ͱ"
    ແݶʹͰͯ͘Δ

    View full-size slide

  21. ·ͣ͸EC2ʹશ෦৐ͤΔ
    • είʔϓΛߜΔ
    • ʢͱ͸͍͑Tsdb͸diamondʹҠߦ͢Δʣ
    • ੾Γ෼͚Մೳͳ෦෼͸͞Βʹ੾Γ෼͚Δ
    • ϦϦʔεཻ౓Λۃྗখ͘͢͞Δ
    • ιϑτ΢ΣΞ։ൃͱಉ༷ʢܧଓతσϦόϦʔʁʣɹ

    View full-size slide

  22. On-Premises DC AWS
    nginx
    app
    db
    redis tsdb (Graphite)
    subsystem
    Πϝʔδਤ
    STEP0 (ॳظঢ়ଶ)

    View full-size slide

  23. On-Premises DC AWS
    nginx
    app
    db
    redis tsdb (Graphite)
    subsystem
    tsdb (diamond)
    Πϝʔδਤ
    STEP1 (tsdb)

    View full-size slide

  24. STEP1
    • ࣌ܥྻσʔλϕʔεͷ৽چ྆ํ΁ͷॻ͖ࠐΈ
    • ৽چͷࠩҟΛࣄલʹ֬ೝ͢Δ
    • ͜ͷ͋ͱ΋͠͹Β͘͸৽چ྆ํʹॻ͖ଓ͚Δ
    • ສ͕Ұͷ࣌ͷ੾Γ໭͠ʹඋ͑Δ

    View full-size slide

  25. On-Premises DC AWS
    nginx
    app
    db
    redis tsdb (Graphite)
    subsystem
    tsdb (diamond)
    Πϝʔδਤ
    STEP1 (tsdb)

    View full-size slide

  26. On-Premises DC AWS
    tsdb (Graphite)
    subsystem app
    db
    redis tsdb (diamond)
    Πϝʔδਤ
    HTTP
    STEP2-1 (app etc)
    nginx

    View full-size slide

  27. On-Premises DC AWS
    tsdb (Graphite)
    subsystem
    nginx
    app
    db
    redis tsdb (diamond)
    Πϝʔδਤ
    HTTP
    STEP2-2 (app etc)

    View full-size slide

  28. STEP2
    • ΞϓϦέʔγϣϯͱಉظతʹ௨৴͢ΔՕॴʢେ൒ʣ
    • subsystem͸ΦϯϓϨϛεͷ··
    • HTTPܦ༝ͷඇಉظ௨৴ͳͷͰ༨༟͕͋Δ
    • diamondͷ͓൸࿐໨΋͜ͷλΠϛϯά

    View full-size slide

  29. On-Premises DC AWS
    tsdb (Graphite)
    subsystem
    nginx
    app
    db
    redis tsdb (diamond)
    Πϝʔδਤ
    HTTP
    STEP2 (app etc)

    View full-size slide

  30. On-Premises DC AWS
    tsdb (Graphite)
    nginx
    app
    db
    redis tsdb (diamond)
    subsystem
    Πϝʔδਤ
    STEP3 (subsystem)

    View full-size slide

  31. STEP3
    • HTTPܦ༝Ͱ௨৴͢ΔαϒγεςϜͷҠઃ
    • ֎ܗ؂ࢹͷΫϩʔϥΛؚΉͷͰग़ޱIP͕มߋ
    • ࣄલࠂ஌ɻ͝ڠྗ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View full-size slide

  32. On-Premises DC AWS
    tsdb (Graphite)
    nginx
    app
    db
    redis tsdb (diamond)
    subsystem
    Πϝʔδਤ
    STEP3 (subsystem)

    View full-size slide

  33. On-Premises DC AWS
    nginx
    app
    db
    redis tsdb (diamond)
    subsystem
    Πϝʔδਤ
    STEP4 (DONE!)

    View full-size slide

  34. Ҡߦ׬ྃ
    • ύϑΥʔϚϯεɺඅ༻ͳͲ੾Γ໭͠ͳ͠ͷ൑அ
    • STEP2͔Β1ϲ݄ଓ͚ͨฒߦՔಇΛऴྃ

    View full-size slide

  35. ࣮ࡍͷҠߦ

    View full-size slide

  36. STEP1 (tsdb)
    • ෛՙࢼݧͯ͠ϦϦʔεՄೳͷ൑அ
    • astj͸STEP2͕৺഑ͩͬͨ
    • ৽ن։ൃ = ౰વաڈͷՔಇ࣮੷͸ͳ͍
    • ࢥͬͨΑ͏ʹύϑΥʔϚϯεग़Δͷ͔ʁ

    View full-size slide

  37. STEP2 (app etc)
    • ແఀࢭͷ੾Γସ͑Λ࣮ࢪ
    • ੾ସ࣌ʹΠϯελϯεো֐ͰҰ෦σʔλফࣦ
    • ޙ೔ௐࠪ಺༰ͱ࠶ൃ๷ࢭࡦΛެ։
    • https://mackerel.io/ja/blog/entry/2017/08/15/113803

    View full-size slide

  38. STEP2 (app etc)
    • ୹࣌ؒ෼ͷΈͱ͸͍͑σʔλফࣦ
    • Ϣʔβ͞Μʹ͝໎࿭Λ͓͔͚ͯ͠͠·ͬͨ
    • Ұํɺdiamond ʹؔͯ͠͸໰୊ͳ͘Քಇ
    • ؼ୐͔ͯ͠Βײ֒ʹ;͚͍ͬͯͨ

    View full-size slide

  39. STEP3 (subsystem)
    • ͕ͭͭͳ͘੾Γସ͑
    • ʢastj͸͜ͷ೔ٳՋͰͨ͠…ʣ

    View full-size slide

  40. Ҡߦ׬ྃʁ
    • ʮ·ͣ͸EC2ʹ৐ͤͨʯஈ֊
    • ϚωʔδυαʔϏε (ex. RDS) ΁ͷҠߦ༨஍
    • “࣍ੈ୅Mackerel”ͷεςοϓΞοϓɹ

    View full-size slide

  41. RDSͷ࣮ྫ࿩࣌ؒ͢͸
    ଟ෼ͳ͍Ͱ͢

    View full-size slide

  42. RDSҠߦ
    • EC2Ҡߦ౰ॳ͸ PostgreSQL 9.3 (EC2)
    • ϚωʔδυαʔϏεԽͰ DB ӡ༻ίετΛԼ͛Δ
    • EC2Ҡߦ࣌ͷΑ͏ͳো֐Λආ͚Δ͜ͱʹ΋ܨ͕Δ
    • ݕ౼ࣄ߲: Aurora for PostgreSQL

    View full-size slide

  43. RDSҠߦ
    • 9.3 (EC2) => 9.3 (RDS) => 9.6 (RDS)
    • 2ճʹ෼͚ͯҠߦ
    • ͦΕͧΕఀࢭϝϯςφϯεΛ࣮ࢪ
    • ͝ཧղ/͝ڠྗ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View full-size slide

  44. ҠߦશମΛৼΓฦΔ
    • Ҡߦखॱͷ෼ׂ͸ਖ਼ղ
    • ΦϖϨʔγϣϯͱͯ͠͸DB੾ସ͕ϔϏʔ
    • σʔλফࣦͰ͸͝໎࿭Λ͓͔͚͠·ͨ͠

    View full-size slide

  45. Ҡߦ͔ͯ͠ΒͷมԽ

    View full-size slide

  46. ΩϟύγςΟ֬อͷॊೈੑ
    • tsdbͷεέʔϥϏϦςΟͷݒ೦ͷղফ
    • େن໛ͳεέʔϧΞ΢τ/Πϯ
    • ؾܰʹԾ૝Ϛγϯͷ࡞੒/ഇغ͕ߦ͑Δ
    • ΠϯϑϥΤϯδχΞ΁ͷʮ͓ئ͍ʯ͕ෆཁʹ
    • ΠϯϑϥͷݕূɾߏஙͷεϐʔυΞοϓ

    View full-size slide

  47. ӡ༻ͷखؒͷݮগ
    • ෺ཧαʔόΛҙࣝͨ͠ӡ༻͔Βͷղ์
    • ϚωʔδυαʔϏεԽ
    • ετϨʔδ૚ͷऔΓճ͠ͷྑ͞
    • EBS snapshot

    View full-size slide

  48. ΞʔΩςΫνϟબఆͷࣗ༝౓޲্
    • ӡ༻ମྗͷඞཁʢͰ͋Ζ͏ʣίϯϙʔωϯτΛ

    ϚωʔδυͰར༻ग़དྷΔ
    • Kinesis Data Streams, Lambda, 

    DynamoDB, ...
    • ৽نߏஙՕॴͰࣗવͱ࠾༻ग़དྷΔ
    • ػցֶशػೳͰͷ AWS Batch, ...

    View full-size slide

  49. ཧ૝ڷʹདྷͨΘ͚Ͱ͸ͳ͍

    View full-size slide

  50. • αʔόɾωοτϫʔΫো֐ࣗମ͸ଘࡏ
    • Ҿ͖ଓ͖ؤு͍͖ͬͯ·͠ΐ͏
    • Ծ૝Ϛγϯ͸Ծ૝Ϛγϯ
    • Ҿ͖ଓ͖ؤு͍͖ͬͯ·͠ΐ͏
    • ΑΓந৅౓ͷߴ͍Πϯϑϥʁ

    View full-size slide

  51. • ϚωʔδυαʔϏεʹ͋Θͤͨઃܭɾӡ༻
    • ٕज़બఆ
    • ʮ࢖͑ͦ͏͚ͩͲϋϚΒͳ͍ʯ
    • AWS APIΛར༻ͨ͠ӡ༻ઃܭ

    View full-size slide

  52. AWSʹҠߦͨ͠ɹ
    • ΩϟύγςΟΛओͳཁ݅ͱͯ͠AWSʹҠߦ
    • ஈ֊Λ౿ΜͰҠߦͨ͠
    • Tsdb͸DynamoDBத৺ʹ಺੡
    • ͦͷଞ͸Ұ୴EC2্ʹ
    • ͦͷޙঃʑʹϚωʔδυԽ

    View full-size slide

  53. AWSʹҠߦͯ͠
    • TsdbͷΩϟύγςΟʹک͑Δੜ׆ʹผΕ
    • ϚωʔδυαʔϏεΛੵۃతʹऔΓೖΕΒΕΔΑ͏ʹ
    • ӡ༻ͷखؒΛݮΒ͢
    • ։ൃΞʔΩςΫνϟબ୒ࢶΛ૿΍͢
    • ϓϩμΫτͷมߋ଎౓ΛߴΊΒΕ͍ͯΔ

    View full-size slide

  54. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View full-size slide