Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8

by astj

Slide 1

Slide 1 text

Observability: Mackerel ʹΑΔ؍ଌ ͱ Mackerel ͷ؍ଌ 2019/09/17 NoOps Meetup Tokyo #8 Asato WAKISAKA (Hatena)

Slide 2

Slide 2 text

Asato WAKISAKA GitHub: @astj / Twitter @ast_j ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ Mackerel ։ൃνʔϜ (2016/08~) Mackerel ςοΫϦʔυ (2018/05~)

Slide 3

Slide 3 text

कඋൣғ: ϑϩϯτΤϯυ͔ΒSREing·Ͱʢࣗশʣ Scala, Golang, TypeScript, …

Slide 4

Slide 4 text

Mackerel

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

• Mackerel ஌ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • Mackerel ৮ͬͨ͜ͱ͕͋Δਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂ • Mackerel ࢖ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂ Ξϯέʔτ

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

؂ࢹ

Slide 9

Slide 9 text

Monitoring

Slide 10

Slide 10 text

Observability

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Observability ͱ Monitoring

Slide 13

Slide 13 text

• Observability: ʮՄ؍ଌੑʯ • γεςϜͷ಺෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ • 3ͭͷओཁͳߏ੒ཁૉ • Metrics, Logging, Tracing • Mackerel ͸͜ͷ͏ͪ Metrics த৺

Slide 14

Slide 14 text

• ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ • ʮՄ؍ଌੑʯ͸γεςϜ͕࣋ͭੑ࣭ • ʮ؍ଌʯ͸γεςϜ֎෦ͷߦಈ

Slide 15

Slide 15 text

Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation Trace Observation

Slide 16

Slide 16 text

Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation Trace Observation • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…

Slide 17

Slide 17 text

Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation Trace Observation ʮϝτϦοΫ؂ࢹʯ͸͜ͷลʁ • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…

Slide 18

Slide 18 text

• ͱ͸͍͑ɺͦͷઌͰ͸ਓ͕ؒ؍ଌ͍ͯ͠Δ͸ͣ • ݹయతʹ͸ΞϥʔτରԠͳͲ • ʮάϥϑΛݟΔʯ΋؍ଌ

Slide 19

Slide 19 text

Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation Trace Observation wਓؒ • ͋Ε΍ • ͜Ε΍ ˞͜Ε͸ۃ୺

Slide 20

Slide 20 text

• લஈʹ͸ʮՄ؍ଌੑʯͦͷ΋ͷ • ޙஈͷʮ؍ଌʯΛ;ͨͭʹ෼͚ͯଊ͑Δ • ʮ؍ଌͷ࣮૷ʯ • ʮʢਓؒͷʣ؍ଌʯ

Slide 21

Slide 21 text

Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl.

Slide 22

Slide 22 text

Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. ٕज़ͱͯ͠ͷ Observability ͕  ͜ͷลΓΛࢦ͢͜ͱ΋͋Ε͹

Slide 23

Slide 23 text

Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. ιϦϡʔγϣϯͷ࿩୊Ͱ Observability ͕ݴٴ͞ΕΔ࣌͸  ͜͜·Ͱؚ·ΕΔ͜ͱ΋ଟ͍

Slide 24

Slide 24 text

……Ͱɺ͍͍Ͱ͔͢Ͷʁ

Slide 25

Slide 25 text

• ٞ࿦ͷ༨஍͸͍Ζ͍Ζ͋Γͦ͏ • ͔͠͠ࠓ೔͸͜ͷ࿩ͷਂ௥͍͸͠·ͤΜ • ౎߹্ࠨ୺ΛՄ؍ଌੑͱͯ͠࿩͠·͢

Slide 26

Slide 26 text

Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. • ϝτϦοΫ؂ࢹ͸: • ϝτϦοΫΛऩू͢Δ • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ • άϥϑ΍μογϡϘʔυͰՄࢹԽ͢Δ • ΞϥʔτΛൃใ͢Δ

Slide 27

Slide 27 text

Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. ϝτϦοΫऩू ϝτϦοΫूੵ • άϥϑ • Ξϥʔτ • μογϡϘʔυ ˞Πϝʔδ • ਓؒ • ࣗಈԽରԠ

Slide 28

Slide 28 text

Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

Mackerel ͷ֓೦

Slide 31

Slide 31 text

ϗετ

Slide 32

Slide 32 text

• ؂ࢹɾ؅ཧͷى఺ʹͳΔ֓೦ • యܕతʹ͸αʔόʔ • ʮαʔόʔͷΑ͏ͳ΋ͷʯ΋ϗετ • ίϯςφͷू߹ (e.g. ECS Task / k8s pod) • Ϋϥ΢υϦιʔε (e.g. Lambda Function) ϗετ

Slide 33

Slide 33 text

Service:Blog Role:App Role:DB Host A Host B Host C Host D Service:Bookmark Role:App Host E Host F … αʔϏεɾϩʔϧ

Slide 34

Slide 34 text

• ϗετΛʮ໾ׂʯͰ෼ྨ • μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ • αʔϏε΍ϩʔϧ୯ҐͰ؂ࢹϧʔϧΛ؅ཧ αʔϏεɾϩʔϧ

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

಺෦ঢ়ଶͷऩू

Slide 39

Slide 39 text

• ࣌ܥྻͰࢀরՄೳ ϝτϦοΫ

Slide 40

Slide 40 text

• ୯७ͳ੒൱ͱϝοηʔδΛه࿥ • ฏৗ࣌ͷཤྺ͸ه࿥ʹ࢒Βͳ͍ νΣοΫ؂ࢹ

Slide 41

Slide 41 text

• ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)؂ࢹͰදݱ • ϩάΛ૸ࠪͯ͠ύλʔϯϚον͢Δ • LTSV ܗࣜͷΞΫηεϩάΛूܭ͢Δ ϩάΛ؂ࢹ͢Δ

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

ऩूͷλʔήοτ • ΤʔδΣϯτܥ • αʔόʔ • ίϯςφ  • αʔϏεϝτϦοΫ • URL ֎ܗ؂ࢹ • ΠϯςάϨʔγϣϯ

Slide 44

Slide 44 text

• αʔόʔ্Ͱಈ͘ΤʔδΣϯτ • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ • γεςϜϝτϦοΫ͸ࣗಈͰऩू • CPU, memory, interface… • ϓϥάΠϯͰऔಘ৘ใΛ֦ுͰ͖Δ αʔόʔ͔Βͷऩू

Slide 45

Slide 45 text

Server Mackerel mackerel-agent Fluentd Golang app OS stats monitor-agent Plugins HTTP gostats POST ྫ

Slide 46

Slide 46 text

• ECS 1 Task ΍ k8s 1 pod Λ1ϗετ • αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ • γεςϜϝτϦοΫΛࣗಈͰऩू • ϓϥάΠϯͰ֦ுՄೳ ίϯςφ͔Βͷऩू

Slide 47

Slide 47 text

Task Task Task Mackerel Agent Container Fluentd container Golang app container monitor-agent Plugins HTTP gostats POST Orchestration Metadata ྫ

Slide 48

Slide 48 text

• ಛఆͷϗετʹඥ෇͔ͳ͍ϝτϦοΫ • Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ • API Ͱࣗ༝ʹ౤ߘ • e.g. fluentd αʔϏεϝτϦοΫ Service:Bookmark Role:App Host

Slide 49

Slide 49 text

• Mackerel ଆ͔Β URL ʹ઀ଓ • ઀ଓՄ൱΍ϨεϙϯελΠϜͳͲΛऩू URL ֎ܗ؂ࢹ

Slide 50

Slide 50 text

• Ϋϥ΢υࣄۀऀͷϞχλϦϯά API ͔Βऔಘ • ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ • ऩूϝτϦοΫ͸Ϋϥ΢υࣄۀऀґଘ AWS / Azure ΠϯςάϨʔγϣϯ Mackerel User Cloud Services Crawl

Slide 51

Slide 51 text

ऩू͢Δ಺༰

Slide 52

Slide 52 text

• γεςϜϝτϦοΫ͸ࣗಈͰऩू • ϛυϧ΢ΣΞ͸ެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫ͸ʁʁ

Slide 53

Slide 53 text

• γεςϜϝτϦοΫ͸ࣗಈͰऩू • ϛυϧ΢ΣΞ͸ެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫ͸ʁʁ

Slide 54

Slide 54 text

• ݴޠ΍ϑϨʔϜϫʔΫ޲͚ͷϓϥάΠϯ͸͋Δ • JVM ΍ Go ͸ GC ౷ܭ౳΋؍ଌͰ͖Δ • ଍Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ

Slide 55

Slide 55 text

Slide 56

Slide 56 text

؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES

Slide 57

Slide 57 text

؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES

Slide 58

Slide 58 text

• ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮૷͕ܾ·Δ ؍ଌՄೳʹ͢Δ

Slide 59

Slide 59 text

؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES

Slide 60

Slide 60 text

• ϓϥάΠϯΛࣗ࡞ͯ͠೚ҙͷखஈͰ؍ଌ • HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ ؍ଌखஈ

Slide 61

Slide 61 text

• ϓϥάΠϯ = ࣮ߦϑΝΠϧ • ॴఆͷग़ྗͰϝτϦοΫ໊ͱ஋Λग़ྗ͢Δ͚ͩ • ࣮ߦ؀ڥ಺Ͱ“ݟ͑Δ”஋ͳΒԿͰ΋औΕΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू $ mackerel-plugin-uptime uptime.seconds 7749392 1568662368

Slide 62

Slide 62 text

• ࣮૷ࣗମ͸ Bash Ͱ΋ Go Ͱ΋ԿͰ΋ • ग़ྗϑΥʔϚοτ͕ૉ๿ͳͷͰ࣮૷Λબ͹ͳ͍ • ؍ଌର৅ʹ͋Θ࣮ͤͯ૷͢Δඞཁ͕͋Δ • ؍ଌର৅ଆͷࣗ༝౓͸ߴ͘ͳΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू

Slide 63

Slide 63 text

HTTP ܦ༝Ͱͷऩू • mackerel-plugin-json • ؍ଌ͍ͨ͠஋Λ JSON Ͱ HTTP ഑৴͢Δ • ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ

Slide 64

Slide 64 text

૝ఆ໰౴ • Q: Prometheus Exporter Έ͍ͨͳʁ • => ํ޲ੑ͸ yesʢ΋ͬͱγϯϓϧʣ • Q: Prometheus Exporter ࢖͑ͳ͍ͷʁ • => ……!

Slide 65

Slide 65 text

HTTP ܦ༝Ͱͷऩू $ mackerel-plugin-json \ > -url=http://localhost/stats active3 1568666460 waiting 2 1568666460 $ curl -fsSL http://localhost/stats {“active”:3,”waiting”:2}

Slide 66

Slide 66 text

HTTP ܦ༝Ͱͷऩू

Slide 67

Slide 67 text

• ͲͪΒ͕ྑ͍͔͸৔߹ʹΑΔ • طʹHTTPΛ஻ΔͳΒHTTP͕ศར • طଘͷ؍ଌखஈ͕͋Ε͹࠶ར༻͍ͨ͠

Slide 68

Slide 68 text

ऩूͨ͠ϝτϦοΫͷར༻

Slide 69

Slide 69 text

• Mackerel Ͱͷओͳ࢖͍ಓ • ؂ࢹʢΞϥʔτʣ • ՄࢹԽʢάϥϑɾμογϡϘʔυʣ

Slide 70

Slide 70 text

؂ࢹʢΞϥʔτʣ

Slide 71

Slide 71 text

;ͨͭͷΞϥʔτ • ଈ࣌ରԠΛଅ͢Ξϥʔτ • ӡ༻ऀͱͯ͠͸θϩʹ͍ͨ͠ʂ • Information ͱͯ͠ͷΞϥʔτ

Slide 72

Slide 72 text

ଈ࣌ରԠͷΞϥʔτ (on-call) • γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ • ରԠΛඞཁͱ͢Δ • “௚Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰ͸ͳ͍

Slide 73

Slide 73 text

Information ͱͯ͠ͷΞϥʔτ • ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ • ྫ: ৑௕Խϗετ͕1୆μ΢ϯ • γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ • ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Ε͹ଈ࣌ରԠ

Slide 74

Slide 74 text

• ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔  Ͳ͏͔Λ؂ࢹ͢Δ • e.g. ֎ܗ؂ࢹ, ΤϥʔϨʔτ • ˱ CPU 100%Ͱ΋ಈ͍ͯΕ͹ྑ͍ ଈ࣌ରԠͷΞϥʔτ

Slide 75

Slide 75 text

• ͱ͸͍͑৔߹ʹΑΓ͚Γ • ௚઀ͷՔಇͷ؂ࢹͰͳͯ͘΋ɺ  ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻

Slide 76

Slide 76 text

Information ͱͯ͠ͷΞϥʔτ • γεςϜՔಇҎ֎ͷࢦඪ΋औΓಘΔ • ଈ࣌ରԠͱಉ͡௨஌ܦ࿏ʹࡌͤͳ͍

Slide 77

Slide 77 text

Ξϥʔτશൠ • ͳΜͰ΋͔ΜͰ΋Ξϥʔτʹ͠ͳ͍ • ຊ౰ʹඞཁͳΞϥʔτ͕ຒ΋ΕΔ • Information ͸ͩΒͩΒ૿͕͑ͪ

Slide 78

Slide 78 text

ՄࢹԽ

Slide 79

Slide 79 text

• ಺෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه࿥͢Δ • खಈͰऩू͠ͳͯ͘΋؍ଌͰ͖Δ • ݱࡏͷঢ়ଶΛ೺Ѳ͢Δखஈ • ఆৗ࣌ɾো֐࣌໰Θͣ • աڈͷঢ়ଶΛৼΓฦΔखஈ άϥϑɾμογϡϘʔυ

Slide 80

Slide 80 text

؍ଌΛҭͯΔ

Slide 81

Slide 81 text

ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱ͸ݶΒͳ͍

Slide 82

Slide 82 text

• ։ൃɾݕূதʹؾͮ͘ • ؂ࢹ༻ҙ͠Α͏ͱͯ͠ෆ଍ʹؾͮ͘ • ো֐ى͖ͯΈͯෆ଍ʹؾͮ͘ ͨͱ͑͹…

Slide 83

Slide 83 text

ඞཁͳ؍ଌ΋࠷ॳ͔Βἧ͏ͱ͸ݶΒͳ͍

Slide 84

Slide 84 text

• ։ൃɾݕূதʹؾͮ͘ • ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ • ো֐ى͖ͯΈͯෆ଍ʹؾͮ͘ ͨͱ͑͹…

Slide 85

Slide 85 text

αΠΫϧΛճ͍ͯ͘͠

Slide 86

Slide 86 text

• ໪࿦࠷ॳ͔Β੔උͰ͖Ε͹ྑ͍ • ಛʹඞਢͷ߲໨͸཈͍͑ͯ͘ • γεςϜͷՔಇͷ؂ࢹʂʂ • ܧଓతʹมߋɾվળ͍ͯ͘͠

Slide 87

Slide 87 text

No content

Slide 88

Slide 88 text

Mackerel Ͱ Mackerel Λ؍ଌ͢Δ

Slide 89

Slide 89 text

Mackerel ͷٕज़ελοΫ • AWS ্ʹߏங • EC2 ӡ༻ͱ ECS Fargate ͕ࠞࡏ • ϝΠϯͷ JVM ͳͲ͕ EC2 • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத

Slide 90

Slide 90 text

୤ઢ

Slide 91

Slide 91 text

ίϯςφศར

Slide 92

Slide 92 text

• Πϯελϯεͷੈ࿩͕ཁΒͳ͍ • Πϯϑϥ·ͰؔΘΔߏ੒มߋ͕खܰ • “chef ߋ৽ͯ͠Πϯελϯε࡞Γ௚ͯ͠ೖΕ ׵͑ͯ…” • ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……

Slide 93

Slide 93 text

୤ઢ͜͜·Ͱ

Slide 94

Slide 94 text

Mackerel ͷٕज़ελοΫ ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε

Slide 95

Slide 95 text

୤ઢ

Slide 96

Slide 96 text

αʔόϨεΞʔΩςΫνϟศར

Slide 97

Slide 97 text

• ૄ݁߹ɾϦτϥΠՄೳͳઃܭ • ϝοηʔδΩϡʔΛڬΉ • Lambda ͷॲཧ಺༰Λႈ౳ʹઃܭ • ύϥϝʔλมߋͰεέʔϧ͢Δ • ʢ໪࿦ FaaS ಛ༗ͷӡ༻՝୊͸θϩͰ͸ͳ͍ʣ

Slide 98

Slide 98 text

୤ઢ͜͜·Ͱ

Slide 99

Slide 99 text

Mackerel ͷٕज़ελοΫ ʢڵຯ͕͋Ε͹աڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ

Slide 100

Slide 100 text

JVM ΞϓϦέʔγϣϯͷ؍ଌ • ϝτϦΫεɾ؂ࢹ: mackerel • jstat ͱ jolokia ʹΑΔJVM಺෦ߏ଄ͷ؍ଌ • αʔόʔ͝ͱͷϨΠςϯγ؂ࢹ • ίωΫγϣϯϓʔϧ؂ࢹ • etc…

Slide 101

Slide 101 text

JVM ΞϓϦέʔγϣϯͷ؍ଌ • APM: NewRelic APM • ΑΓਫ਼៛ͳ಺෦෼ੳ͕ඞཁͳͱ͖ʹ • ϩάղੳ: fluentd => Kibana • ΞΫηεϩάɺΞϓϦέʔγϣϯϩά

Slide 102

Slide 102 text

։ൃ؀ڥͷ؍ଌ • ։ൃ؀ڥʢ݉εςʔδϯά؀ڥʣ • ຊ൪ͱಉ༷ͷߏ੒ʢن໛͸খ͍͞ʣ • ಉ༷ͷ؍ଌπʔϧߏ੒ • => ϦϦʔεલ͔Β։ൃ؀ڥͰ؍ଌ

Slide 103

Slide 103 text

૝ఆ໰౴ • Q: ։ൃ؀ڥͷ؂ࢹͷίετ͸ʁ • => զʑ͸MackerelΛແྉͰ࢖͑Δʢ൓ଇʣ

Slide 104

Slide 104 text

• த֩ʹ Mackerel • ීஈͷʮ؂ࢹʯʮ؅ཧʯͷى఺͸͜͜ • ଞͷπʔϧͱ૊Έ߹Θͤͯ؍ଌΛิڧ