Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11

by astj

Slide 1

Slide 1 text

MackerelΛΦϯϓϨϛε͔Β AWSʹҠ͔ͯ͠Βͷ 1೥൒ΛৼΓฦΔ Hatena Engineer Seminar #11 id:astj

Slide 2

Slide 2 text

id:astjʢ͋͞ͱ͐͡ʔʣ ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ ৽ଔ (2014/04) Mackerel ։ൃνʔϜ (2016/08~) Mackerel ςοΫϦʔυ (2018/05~)

Slide 3

Slide 3 text

Mackerel

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

• 2014೥ϦϦʔεͷʮαʔόʔ؂ࢹαʔϏεʯ • αʔόʔαΠυ: Scala / Go • ػցֶशύʔτʢ։ൃதʂʣ: Python • AWS Lambda : NodeJS

Slide 8

Slide 8 text

ຊ೔ͷத਎ • ΦϯϓϨϛε͔ΒAWSʹҠߦͨ͜͠ͱ • Ҡߦͯ͠ಘΒΕͨ͜ͱɾมΘͬͨ͜ͱ

Slide 9

Slide 9 text

༧උ஌ࣝ / ͝঺հ

Slide 10

Slide 10 text

On-Premises DC Ofﬁce AWS Account A (Main) AWS Account B AWS Account C Dedicated Hosting ͸ͯͳͷΠϯϑϥߏ੒ …

Slide 11

Slide 11 text

͸ͯͳͷΠϯϑϥߏ੒ On-Premises DC Ofﬁce AWS Account A AWS Account B AWS Account C Dedicated Hosting … ΦϯϓϨϛεͷDCͱAWSʢͳͲʣΛซ༻ ྺ࢙తܦҢͰڞ௨ͷAWSΞΧ΢ϯτ αʔϏεʹΑͬͯ͸ಠཱΞΧ΢ϯτ

Slide 12

Slide 12 text

Mackerelͷߏ੒ RDBMS (Postgres) Redis Tsdb (nginx) (nginx) reverse proxy (nginx) (nginx) (nginx) app (Scala) (nginx) (nginx) Sub- systems

Slide 13

Slide 13 text

hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … … ࣌ܥྻσʔλ

Slide 14

Slide 14 text

࣌ܥྻσʔλϕʔε • ΦϯϓϨϛε࣌୅: Graphite • http://graphite.readthedocs.org/ • AWS Ҡߦޙ: ಺੡ϓϩμΫτ • ίʔυωʔϜ "diamond"

Slide 15

Slide 15 text

ΦϯϓϨϛε͔ΒAWS΁

Slide 16

Slide 16 text

ॳظͷ Mackerel ͷߏ੒ • ϩʔϯν౰ॳ͸ΦϯϓϨϛε • جຊతʹ͸XenͰԾ૝Խͨ͠VMӡ༻ • Tsdb΍RDBͰ͸ϋΠεϖοΫͳ෺ཧαʔόΛ  Ծ૝Խͤͣར༻ • ioDrive (~380k IOPS Ͱେ༰ྔͰߴ͍)

Slide 17

Slide 17 text

՝୊ײ • αʔϏε੒௕ʹ൐͍՝୊͕ग़͖ͯͨ • Graphiteͷӡ༻͕͍ͨ΁Μ • σΟεΫ͕΋Γ΋Γຒ·Δɾ৑௕Խߏ੒ͷӡ༻ • εέʔϧΞ΢τʹ෺ཧαʔόͷௐୡ

Slide 18

Slide 18 text

=> AWS

Slide 19

Slide 19 text

ʮ࣍ੈ୅MackerelϓϩδΣΫτʯ

Slide 20

Slide 20 text

AWSҠߦ • TsdbͷΩϟύγςΟ֬อͷॊೈԽ • ηΩϡϦςΟରࡦج൫ͷॆ࣮ • ܭը => Ҡߦ·Ͱ͓Αͦ1೥ • Tsdbͷݕূɾ։ൃʹ࣌ؒͱ޻਺Λେׂ͖͍ͨ͘

Slide 21

Slide 21 text

TsdbͷAWSҠߦ • ߴεϖοΫ෺ཧαʔόಉ༷ͷӡ༻͸ݫ͍͠ • EBSͰٻΊΒΕͨiops͸ग़ͤͳ͍ • ΦϯϓϨϛεͷ࣌఺Ͱӡ༻ෛՙ͸͋ͬͨ

Slide 22

Slide 22 text

࣍ੈ୅Tsdb: diamond DynamoDBΛ࣠ʹͨ͠ϚωʔδυαʔϏε  த৺ͷ࣌ܥྻσʔλϕʔεΛ։ൃ • https://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud • https://itchyny.hatenablog.com/entry/2017/11/06/090000 • https://astj.hatenablog.com/entry/2018/02/06/175902

Slide 23

Slide 23 text

http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud

Slide 24

Slide 24 text

Ҡߦϓϩηε

Slide 25

Slide 25 text

"࣍ੈ୅Ͱ΍Γ͍ͨ͜ͱ" ແݶʹͰͯ͘Δ

Slide 26

Slide 26 text

·ͣ͸EC2ʹશ෦৐ͤΔ • είʔϓΛߜΔ • ʢͱ͸͍͑Tsdb͸diamondʹҠߦ͢Δʣ • ੾Γ෼͚Մೳͳ෦෼͸͞Βʹ੾Γ෼͚Δ • ϦϦʔεཻ౓Λۃྗখ͘͢͞Δ • ιϑτ΢ΣΞ։ൃͱಉ༷ʢܧଓతσϦόϦʔʁʣɹ

Slide 27

Slide 27 text

࡞ઓ

Slide 28

Slide 28 text

On-Premises DC AWS nginx app db redis tsdb (Graphite) subsystem Πϝʔδਤ STEP0 (ॳظঢ়ଶ)

Slide 29

Slide 29 text

On-Premises DC AWS nginx app db redis tsdb (Graphite) subsystem tsdb (diamond) Πϝʔδਤ STEP1 (tsdb)

Slide 30

Slide 30 text

STEP1 • ࣌ܥྻσʔλϕʔεͷ৽چ྆ํ΁ͷॻ͖ࠐΈ • ৽چͷࠩҟΛࣄલʹ֬ೝ͢Δ • ͜ͷ͋ͱ΋͠͹Β͘͸৽چ྆ํʹॻ͖ଓ͚Δ • ສ͕Ұͷ࣌ͷ੾Γ໭͠ʹඋ͑Δ

Slide 31

Slide 31 text

On-Premises DC AWS nginx app db redis tsdb (Graphite) subsystem tsdb (diamond) Πϝʔδਤ STEP1 (tsdb)

Slide 32

Slide 32 text

On-Premises DC AWS tsdb (Graphite) subsystem app db redis tsdb (diamond) Πϝʔδਤ HTTP STEP2-1 (app etc) nginx

Slide 33

Slide 33 text

On-Premises DC AWS tsdb (Graphite) subsystem nginx app db redis tsdb (diamond) Πϝʔδਤ HTTP STEP2-2 (app etc)

Slide 34

Slide 34 text

STEP2 • ΞϓϦέʔγϣϯͱಉظతʹ௨৴͢ΔՕॴʢେ൒ʣ • subsystem͸ΦϯϓϨϛεͷ·· • HTTPܦ༝ͷඇಉظ௨৴ͳͷͰ༨༟͕͋Δ • diamondͷ͓൸࿐໨΋͜ͷλΠϛϯά

Slide 35

Slide 35 text

On-Premises DC AWS tsdb (Graphite) subsystem nginx app db redis tsdb (diamond) Πϝʔδਤ HTTP STEP2 (app etc)

Slide 36

Slide 36 text

On-Premises DC AWS tsdb (Graphite) nginx app db redis tsdb (diamond) subsystem Πϝʔδਤ STEP3 (subsystem)

Slide 37

Slide 37 text

STEP3 • HTTPܦ༝Ͱ௨৴͢ΔαϒγεςϜͷҠઃ • ֎ܗ؂ࢹͷΫϩʔϥΛؚΉͷͰग़ޱIP͕มߋ • ࣄલࠂ஌ɻ͝ڠྗ͋Γ͕ͱ͏͍͟͝·ͨ͠

Slide 38

Slide 38 text

On-Premises DC AWS tsdb (Graphite) nginx app db redis tsdb (diamond) subsystem Πϝʔδਤ STEP3 (subsystem)

Slide 39

Slide 39 text

On-Premises DC AWS nginx app db redis tsdb (diamond) subsystem Πϝʔδਤ STEP4 (DONE!)

Slide 40

Slide 40 text

Ҡߦ׬ྃ • ύϑΥʔϚϯεɺඅ༻ͳͲ੾Γ໭͠ͳ͠ͷ൑அ • STEP2͔Β1ϲ݄ଓ͚ͨฒߦՔಇΛऴྃ

Slide 41

Slide 41 text

࣮ࡍͷҠߦ

Slide 42

Slide 42 text

STEP1 (tsdb) • ෛՙࢼݧͯ͠ϦϦʔεՄೳͷ൑அ • astj͸STEP2͕৺഑ͩͬͨ • ৽ن։ൃ = ౰વաڈͷՔಇ࣮੷͸ͳ͍ • ࢥͬͨΑ͏ʹύϑΥʔϚϯεग़Δͷ͔ʁ

Slide 43

Slide 43 text

STEP2 (app etc) • ແఀࢭͷ੾Γସ͑Λ࣮ࢪ • ੾ସ࣌ʹΠϯελϯεো֐ͰҰ෦σʔλফࣦ • ޙ೔ௐࠪ಺༰ͱ࠶ൃ๷ࢭࡦΛެ։ • https://mackerel.io/ja/blog/entry/2017/08/15/113803

Slide 44

Slide 44 text

STEP2 (app etc) • ୹࣌ؒ෼ͷΈͱ͸͍͑σʔλফࣦ • Ϣʔβ͞Μʹ͝໎࿭Λ͓͔͚ͯ͠͠·ͬͨ • Ұํɺdiamond ʹؔͯ͠͸໰୊ͳ͘Քಇ • ؼ୐͔ͯ͠Βײ֒ʹ;͚͍ͬͯͨ

Slide 45

Slide 45 text

STEP3 (subsystem) • ͕ͭͭͳ͘੾Γସ͑ • ʢastj͸͜ͷ೔ٳՋͰͨ͠…ʣ

Slide 46

Slide 46 text

Ҡߦ׬ྃ

Slide 47

Slide 47 text

Ҡߦ׬ྃʁ

Slide 48

Slide 48 text

Ҡߦ׬ྃʁ • ʮ·ͣ͸EC2ʹ৐ͤͨʯஈ֊ • ϚωʔδυαʔϏε (ex. RDS) ΁ͷҠߦ༨஍ • “࣍ੈ୅Mackerel”ͷεςοϓΞοϓɹ

Slide 49

Slide 49 text

RDSͷ࣮ྫ࿩࣌ؒ͢͸ ଟ෼ͳ͍Ͱ͢

Slide 50

Slide 50 text

RDSҠߦ • EC2Ҡߦ౰ॳ͸ PostgreSQL 9.3 (EC2) • ϚωʔδυαʔϏεԽͰ DB ӡ༻ίετΛԼ͛Δ • EC2Ҡߦ࣌ͷΑ͏ͳো֐Λආ͚Δ͜ͱʹ΋ܨ͕Δ • ݕ౼ࣄ߲: Aurora for PostgreSQL

Slide 51

Slide 51 text

RDSҠߦ • 9.3 (EC2) => 9.3 (RDS) => 9.6 (RDS) • 2ճʹ෼͚ͯҠߦ • ͦΕͧΕఀࢭϝϯςφϯεΛ࣮ࢪ • ͝ཧղ/͝ڠྗ͋Γ͕ͱ͏͍͟͝·ͨ͠

Slide 52

Slide 52 text

ҠߦશମΛৼΓฦΔ • Ҡߦखॱͷ෼ׂ͸ਖ਼ղ • ΦϖϨʔγϣϯͱͯ͠͸DB੾ସ͕ϔϏʔ • σʔλফࣦͰ͸͝໎࿭Λ͓͔͚͠·ͨ͠

Slide 53

Slide 53 text

Ҡߦ͔ͯ͠ΒͷมԽ

Slide 54

Slide 54 text

ΩϟύγςΟ֬อͷॊೈੑ • tsdbͷεέʔϥϏϦςΟͷݒ೦ͷղফ • େن໛ͳεέʔϧΞ΢τ/Πϯ • ؾܰʹԾ૝Ϛγϯͷ࡞੒/ഇغ͕ߦ͑Δ • ΠϯϑϥΤϯδχΞ΁ͷʮ͓ئ͍ʯ͕ෆཁʹ • ΠϯϑϥͷݕূɾߏஙͷεϐʔυΞοϓ

Slide 55

Slide 55 text

ӡ༻ͷखؒͷݮগ • ෺ཧαʔόΛҙࣝͨ͠ӡ༻͔Βͷղ์ • ϚωʔδυαʔϏεԽ • ετϨʔδ૚ͷऔΓճ͠ͷྑ͞ • EBS snapshot

Slide 56

Slide 56 text

ΞʔΩςΫνϟબఆͷࣗ༝౓޲্ • ӡ༻ମྗͷඞཁʢͰ͋Ζ͏ʣίϯϙʔωϯτΛ  ϚωʔδυͰར༻ग़དྷΔ • Kinesis Data Streams, Lambda,   DynamoDB, ... • ৽نߏஙՕॴͰࣗવͱ࠾༻ग़དྷΔ • ػցֶशػೳͰͷ AWS Batch, ...

Slide 57

Slide 57 text

ศར

Slide 58

Slide 58 text

ͱ͸͍͑

Slide 59

Slide 59 text

ཧ૝ڷʹདྷͨΘ͚Ͱ͸ͳ͍

Slide 60

Slide 60 text

• αʔόɾωοτϫʔΫো֐ࣗମ͸ଘࡏ • Ҿ͖ଓ͖ؤு͍͖ͬͯ·͠ΐ͏ • Ծ૝Ϛγϯ͸Ծ૝Ϛγϯ • Ҿ͖ଓ͖ؤு͍͖ͬͯ·͠ΐ͏ • ΑΓந৅౓ͷߴ͍Πϯϑϥʁ

Slide 61

Slide 61 text

• ϚωʔδυαʔϏεʹ͋Θͤͨઃܭɾӡ༻ • ٕज़બఆ • ʮ࢖͑ͦ͏͚ͩͲϋϚΒͳ͍ʯ • AWS APIΛར༻ͨ͠ӡ༻ઃܭ

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

·ͱΊ

Slide 64

Slide 64 text

AWSʹҠߦͨ͠ɹ • ΩϟύγςΟΛओͳཁ݅ͱͯ͠AWSʹҠߦ • ஈ֊Λ౿ΜͰҠߦͨ͠ • Tsdb͸DynamoDBத৺ʹ಺੡ • ͦͷଞ͸Ұ୴EC2্ʹ • ͦͷޙঃʑʹϚωʔδυԽ

Slide 65

Slide 65 text

AWSʹҠߦͯ͠ • TsdbͷΩϟύγςΟʹک͑Δੜ׆ʹผΕ • ϚωʔδυαʔϏεΛੵۃతʹऔΓೖΕΒΕΔΑ͏ʹ • ӡ༻ͷखؒΛݮΒ͢ • ։ൃΞʔΩςΫνϟબ୒ࢶΛ૿΍͢ • ϓϩμΫτͷมߋ଎౓ΛߴΊΒΕ͍ͯΔ

Slide 66

Slide 66 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠