AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo

by astj

Slide 1

Slide 1 text

"84Ͱ࣮ݱͨ͠.BDLFSFM ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷཪଆ .BDLFSFM.FFUVQ5PLZP JEBTUKʢגࣜձࣾ͸ͯͳʣ

Slide 2

Slide 2 text

w ͸ͯͳΞϓϦέʔγϣϯΤϯδχΞ w ͸ͯͳϒοΫϚʔΫ͸ͯͳϒϩάFUDʜ w डୗαʔϏε w .BDLFSFM d w αʔόαΠυΤʔδΣϯτத৺ JEBTUK

Slide 3

Slide 3 text

w NBDLFSFMBHFOUTZTUFNEରԠ w NBDLFSFMQMVHJOBXT͍͔ͭ͘ w -*/&5XJMJP࿈ܞ w "84Ҡߦ࣌ܥྻσʔλϕʔεͷ࡮৽ w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘ w ؂ࢹϧʔϧͷૢ࡞Λ௨஌νϟϯωϧʹ௨஌ w ʜ JEBTUK

Slide 4

Slide 4 text

ຊ೔ͷ࿩୊

Slide 5

Slide 5 text

w .BDLFSFM࣌ܥྻ%#ͷ঺հ w ෼ཻ౓௕ظอଘͷٕज़എܠ w αʔόϨεϛυϧ΢ΣΞͷߏஙࣄྫ w %ZOBNP%#4USFBNT-BNCEB w ߴεϧʔϓοτ4ॻ͖ࠐΈ

Slide 6

Slide 6 text

લఏ

Slide 7

Slide 7 text

࣌ܥྻσʔλ

Slide 8

Slide 8 text

࣌ܥྻσʔλ hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …

Slide 9

Slide 9 text

࣌ܥྻσʔλϕʔε hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …

Slide 10

Slide 10 text

http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

NI NE IE #FGPSF E

Slide 13

Slide 13 text

NE NE IE E "GUFS

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

.BDLFSFMͷ ࣌ܥྻσʔλϕʔε

Slide 16

Slide 16 text

Ҡߦલ(SBQIJUF IUUQTHSBQIJUFBQQPSH

Slide 17

Slide 17 text

http://blog.yuuk.io/entry/high-performance-graphite

Slide 18

Slide 18 text

ҠߦޙEJBNPOE ಺੡ίʔυωʔϜ

Slide 19

Slide 19 text

IUUQCMPHZVVLJPFOUSZUIFSFCVJMEPGUTECPODMPVE ࣌ܥྻσʔλϕʔεͱ͍͏֓೦ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ

Slide 20

Slide 20 text

IUUQJUDIZOZIBUFOBCMPHDPNFOUSZ 4FSWFSMFTTDPOG5PLZPͰʰαʔόϨεΞʔΩςΫνϟʹ ΑΔ࣌ܥྻσʔλϕʔεͷߏஙͱ؂ࢹʱͱ͍͏ൃද͖ͯ͠·ͨ͠

Slide 21

Slide 21 text

http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud 3FEJT$MVTUFS &$

Slide 22

Slide 22 text

ίϯηϓτ w εέʔϧ͢ΔΞʔΩςΫνϟ w ϚωʔδυαʔϏεʹΑΔӡ༻লྗԽ w ΞΫηεಛੑʹԠͨ͡֊૚ܕσʔλετΞ w 3FEJT %ZOBNP%# 4

Slide 23

Slide 23 text

ετϨʔδ *0ίετ ϨΠςϯγ ༰ྔίετ 3FEJT ˕ ˕ ˚ %ZOBNP%# ˓ ˓ ˓ 4 ˚ ˚ ˕ σʔλετϨʔδ

Slide 24

Slide 24 text

%ZOBNP%# w ϑϧϚωʔδυ/P42- w ҆ఆɾߴύϑΥʔϚϯεɾεέʔϥϏϦςΟ w खࠒͳίετ w σʔλ༰ྔɺεϧʔϓοτ w ʢͱ͸͍͑අ༻Λແࢹ͸Ͱ͖ͳ͍ʣ

Slide 25

Slide 25 text

3FEJT $MVTUFS w ΠϯϝϞϦ,74 w *0ίετ͕ %ZOBNP%#ΑΓ खܰ w σʔλྔΛࡹͨ͘Ί3FEJT$MVTUFS w ࠓͷͱ͜Ζ&$

Slide 26

Slide 26 text

4 w ,74తʹར༻ w ༰ྔίετ͸ඇৗʹ཈͑ΒΕΔ w ϨΠςϯγɾ*0ίετ͸ྼΔ w ॻ͖ࠐΈճ਺ΛݮΒ͍ͨ͠

Slide 27

Slide 27 text

w %ZOBNP%# w ϝΠϯ w 3FEJT w ߋ৽ϦΫΤετΛݮΒͨ͢ΊͷόοϑΝ w 4 w طଘͷอ࣋ظؒͰ͸ ར༻͠ͳ͍ ֊૚ܕσʔλετΞ

Slide 28

Slide 28 text

NI NE IE ֊૚ܕσʔλετΞ %ZOBNP%# 3FEJT E

Slide 29

Slide 29 text

෼ཻ౓ͷ௕ظอଘ

Slide 30

Slide 30 text

w σʔλྔ͕େ͖͘๲ΒΉ ഒ w ݹ͍σʔλͷϨΠςϯγཁ݅͸গ͠؇͍ w ৽͍͠σʔλ΄Ͳසൟʹ͸ࢀর͞Εͳ͍ w ݹ͍σʔλͷ্ॻ͖ߋ৽͸ൃੜ͠ͳ͍

Slide 31

Slide 31 text

w ݹ͍σʔλΛ4ʹ഑ஔ w طଘͷอ࣋ظؒͷύϑΥʔϚϯεΛҡ࣋ w ॻ͖ࠐΈճ਺ΛݮΒ͢ ֊૚ܕσʔλετΞ

Slide 32

Slide 32 text

NE NE IE ֊૚ܕσʔλετΞ %ZOBNP%# 3FEJT 4 E

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU

Slide 35

Slide 35 text

%ZOBNP%#55-

Slide 36

Slide 36 text

%ZOBNP%#ͷΞΠςϜ͝ͱʹ༗ޮظݶ 55- Λࢦఆͯ͠ɺࣗಈతʹ࡟আͤ͞Δ

Slide 37

Slide 37 text

%ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 2018/02/02 16:00~19:59 ExpireAt: 2018/02/03 21:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user hatena.mackerel.host1.cpu.user 2018/02/02 20:00~23:59 ExpireAt: 2018/02/04 01:00

Slide 38

Slide 38 text

%ZOBNP%# hatena.mackerel.host1.cpu.user 2018/02/03 00:00~03:59 ExpireAt: 2018/02/04 05:00 hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user hatena.mackerel.host1.cpu.user 2018/02/02 20:00~23:59 ExpireAt: 2018/02/04 01:00

Slide 39

Slide 39 text

Slide 40

Slide 40 text

hatena.mackerel.host1.cpu.user 2018/02/03 04:00~07:59 ExpireAt: 2018/02/04 09:00 hatena.mackerel.host1.cpu.user %ZOBNP%#

Slide 41

Slide 41 text

%ZOBNP%# hatena.mackerel.host1.cpu.user

Slide 42

Slide 42 text

%ZOBNP%#

Slide 43

Slide 43 text

ೳಈతͳૢ࡞ͳ͘%ZOBNP%#্ͷ ϨίʔυΛ࡟আͰ͖Δ

Slide 44

Slide 44 text

%ZOBNP%#4USFBNT

Slide 45

Slide 45 text

w %ZOBNP%#্ͷΞΠςϜߋ৽͕  Πϕϯτͱͯ͠ྲྀΕΔετϦʔϜ w ςʔϒϧͷσʔλมߋΛτϦΨʹ  -BNCEBؔ਺ΛىಈͰ͖Δ w ϖΠϩʔυʹมߋલޙͷΞΠςϜ಺༰Λ  ؚΊΔ͜ͱ͕Ͱ͖Δ

Slide 46

Slide 46 text

%ZOBNP%# 4USFBN -BNCEB ৽نΞΠςϜ௥Ճ طଘΞΠςϜߋ৽ ΞΠςϜ࡟আ

Slide 47

Slide 47 text

EJBNPOEFYQPSUFS %ZOBNP%#55- %ZOBNP%#4USFBNT

Slide 48

Slide 48 text

w 55-ʹ౸ୡͨ͠ΞΠςϜͷ࡟আ࣌ʹ  ΞΠςϜͷ಺༰Λ౉ͯ͠-BNCEBؔ਺Λ ىಈ w -BNCEBؔ਺Ͱ4ʹॻ͖ग़͠

Slide 49

Slide 49 text

͜ͷΞΠςϜ͕55-ʹ౸ୡͯ͠  ࡟আ͞Ε·ͨ͠ hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … … hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …

Slide 50

Slide 50 text

hatena.mackerel.host1.cpu.user 2018/02/01T21:15:00Z 44.00 2018/02/01T21:16:00Z 6.00 2018/02/05T21:17:00Z 8.00 … …

Slide 51

Slide 51 text

55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU

Slide 52

Slide 52 text

55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w ϝτϦοΫ͝ͱO࣌ؒ͝ͱʹΞΠςϜ w %ZOBNP%#ͷΞΠςϜʹ55-Λઃఆ w 55-ܦաʹΑΓΞΠςϜ࡟আ w ΞΠςϜ࡟আΠϕϯτ͕4USFBNʹ௨஌ w 4USFBN͔Β-BNCEBؔ਺ىಈ w -BNCEBؔ਺͕ΞΠςϜΛ4ʹॻࠐ

Slide 53

Slide 53 text

55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w 55-෇༩Ҏ֎ͷ ೳಈతͳૢ࡞͕ෆཁ w -BNCEBͷࣦഊ࣌ϦτϥΠ΋ࣗಈ w 4USFBNҎ֎ʹঢ়ଶΛ࣋ͨͳ͍ w %ZOBNP%#ΞΠςϜ4Φϒ δΣΫτ

Slide 54

Slide 54 text

4 EJBNPOE"11 NBDLFSFM"11 (FU0CKFDU )551 )551

Slide 55

Slide 55 text

γϯϓϧ

Slide 56

Slide 56 text

"84ͷΞοϓσʔτʹΑ࣮ͬͯݱ

Slide 57

Slide 57 text

೥ͷ%ZOBNP%#Ξοϓσʔτ w 55-d w "VUP4DBMF %"9d w 71$&OEQPJOUd w (MPCBM5BCMFT ΦϯσϚϯυόοΫΞοϓd

Slide 58

Slide 58 text

4΁ͷߴස౓ॻ͖ࠐΈ

Slide 59

Slide 59 text

w 4΁ͷߴස౓ॻ͖ࠐΈΛߦ͏৔߹͸ରԠ͕ඞཁ w ಺෦γϟʔσΟϯάରԠͷͨΊͷQSFpY෇༩ w ٸܹͳ૿Ճͷ৔߹͸ "84αϙʔτܦ༝Ͱͷ  ࣄલରԠґཔ

Slide 60

Slide 60 text

Amazon S3 όέοτͷϫʔΫϩʔυ͕ຖඵ 100 ճͷ PUT/LIST/DELETE ϦΫΤετ·ͨ͸ຖඵ 300 ճͷ GET ϦΫΤετΛසൟʹ௒͑Δ৔߹ ͸ɺ࠷ߴͷύϑΥʔϚϯεͱεέʔϥϏϦςΟ Λ֬อ͢ΔͨΊɺ͜ͷτϐοΫͷΨΠυϥΠϯ ʹै͍ͬͯͩ͘͞ɻ ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲ https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request- rate-perf-considerations.html

Slide 61

Slide 61 text

EJBNPOEͰ4΁ͷॻ͖ࠐΈ͸ίϯελϯτʹൃੜ SQT͸༏ʹ௒͑Δ

Slide 62

Slide 62 text

Slide 63

Slide 63 text

55-&YQJSF %ZOBNP%# %%#4USFBN -BNCEB 4 1VU0CKFDU w 55-෇༩Ҏ֎ͷ ೳಈతͳૢ࡞͕ෆཁ w -BNCEBͷࣦഊ࣌ϦτϥΠ΋ࣗಈ w %ZOBNP%#ͷΞΠςϜ4Φϒ δΣΫτ ࠶ܝ

Slide 64

Slide 64 text

ϦΫΤετ཰͓ΑͼϦΫΤετύϑΥʔϚϯεʹؔ͢Δཹҙࣄ߲ https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/request- rate-perf-considerations.html

Slide 65

Slide 65 text

w ΦϒδΣΫτ໊ઌ಄ʹ.%ϋογϡͷ෇༩ w ॻ͖ࠐΈ։࢝લʹαϙʔτʹରԠΛґཔ w ਺೔Φʔμʔͷ଴͕ͪ࣌ؒൃੜ͢Δ

Slide 66

Slide 66 text

ա৒ͳϖʔεͰͷߋ৽͸)551YYΤϥʔ͕ ൃੜ͢ΔʢΒ͍͠ʣ ϞχλϦϯάʂ

Slide 67

Slide 67 text

"844ͷϞχλϦϯά

Slide 68

Slide 68 text

w 4ͷඪ४$MPVE8BUDI.FUSJDT͸ݶఆత w όέοτͷ೔࣍ͷετϨʔδϝτϦΫε w ϦΫΤετͷ෼ੳ͸Ͱ͖ͳ͍ w ΦϓγϣϯͰϦΫΤετͷϞχλϦϯά΋Մೳ w 4όέοτͷઃఆͰ༗ޮʹͰ͖Δ https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/ cloudwatch-monitoring.html

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

$MPVE8BUDI.BDLFSFM

Slide 71

Slide 71 text

mackerel-plugin-aws-s3-requests [plugin.metrics.diamond-s3] command = "/path/to/mackerel-plugin-aws-s3-requests --bucket- name=XXX --region=ap-northeast-1 --filter-id YYY" ެࣜϓϥάΠϯͱͯ͠ϦϦʔε༧ఆ ͖ͬͱࠓिதʹ͸ʜ

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

w (FU 1VUͦΕͧΕͷϦΫΤετ਺ w ୹ظతͳόʔετɺ௕ظతͳτϨϯυ w YYΤϥʔͷ਺ ؂ࢹϧʔϧ

Slide 74

Slide 74 text

͓ΘΓʹ

Slide 75

Slide 75 text

w ࣌ܥྻσʔλ෼ཻ౓௕ظอଘͷ಺෦࣮૷ w ߴස౓4ॻ͖ࠐΈ w ཹҙࣄ߲ϞχλϦϯά

Slide 76

Slide 76 text

༗Γ೉͏͍͟͝·ͨ͠ʂ