Tweet
Tweet

Slide 1

Slide 1 text

Building Dashboards as a Hobby 2024/04/24 Sohei Iwahori (GREE, Inc.)

Slide 2

Slide 2 text

who? » Sohei Iwahori (@egmc) » GREE, Inc. » Πϯϑϥ / Monitoring Unit Leader / Senior Lead Engineer » ओʹήʔϜͷΠϯϑϥͱ؂ࢹγεςϜ » σʔλιʔε͸ओʹPrometheus

Slide 3

Slide 3 text

࠷ۙͷ೰Έ » systemd_exporterʹsystemd_resolved ͷϝτϦΫεΛ௥Ճ͢ΔPR͕Ϛʔδ ͞Εͨ(1݄) » ͕৽όʔδϣϯ͕ͳ͔ͳ͔ϦϦʔε ͞Εͳ͍ » ޙ൒΁ͷϑϦ

Slide 4

Slide 4 text

࠷ۙͷ೰Έ

Slide 5

Slide 5 text

agenda » μογϡϘʔυͷຽओԽ » झຯͷμογϡϘʔυ࡞੒Ͱ࢖͏πʔϧ » ࡞ͬͨμογϡϘʔυͷ঺հ » ·ͱΊ

Slide 6

Slide 6 text

μογϡϘʔυͷຽओԽ

Slide 7

Slide 7 text

GrafanaͷμογϡϘʔυͷخ͠͞ » ࢥ͍͍ͭͨμογϡϘʔυΛUI্͔Β࡞੒Ͱ͖Δ » τϥΠΞϯυΤϥʔͷ͸΍͞ » RRDtoolͰٯϙʔϥϯυه๏ͰάϥϑΛॻ࣌͘୅͔Β͸ѹ౗తͳརศੑ

Slide 8

Slide 8 text

RRDTool -> Grafana + PromQL

Slide 9

Slide 9 text

झຯͷμογϡϘʔυ࡞੒Ͱ ࢖͏πʔϧ

Slide 10

Slide 10 text

μογϡϘʔυ։ൃͷΤίγεςϜ » Grafana as code ৭ʑ͋Δɾɾ1 » Grafana Terraform provider » Grafana Ansible collection » Grizzly » Grafana Crossplane provider » Kubernetes Grafana Operator » ͦͷ΄͔ScenesͳͲ΋ 1 A complete guide to managing Grafana as code: tools, tips, and tricks Ishan Jain.

Slide 11

Slide 11 text

ͲΕʹ͢Δ͔ » ਖ਼௚ͦΜͳʹ͜Ε͕͍͍ʂΈ͍ͨͳͷͳ͍ؾ͕͢ΔͷͰϒϩάͷ௨Γطଘͷ ίʔυࢿ࢈ʹ߹ΘͤͯͰ » τϥΠΞϯυΤϥʔ͸खಈͰɺ௕ظతʹ࡞ΓࠐΈ͍ͨ৔߹ʹίʔυԽ » ͱ͸͍͑JSONܗࣜͳΒ݁ہϕʔεΛखͰ࡞ΔɺͰ΋ » ݸਓͰ͓खܰʹ΍Δʹ͸खಈ࡞੒+Ұ෦GrizzlyΛར༻

Slide 12

Slide 12 text

Grizzly » https://github.com/grafana/grizzly » Grafana։ൃͷcliπʔϧʢgrrίϚϯυʣ » ݱࡏ͸Dashbaord؅ཧ͚ͩͰͳ͘σʔλιʔεɺGrafana Cloud prometheus ͷrules/alertsɺSynthetic Monitoringͷ؅ཧʹ΋ରԠ » v0.4.1࣌఺ » جຊͷه๏͸YAMLɺJSONɺJsonnetʢ+Grafonnetʣ΋͔ͭ͑Δ

Slide 13

Slide 13 text

Grizzyͱͷग़ձ͍ » 2020೥ࠒ0.0.1ͷalpha࣌఺ͰJsonnet(+Grafonnet)Ͱͷར༻ΨΠυ͕͋Γɺ࢖͍࢝Ίͨ » લड़ͷϦΞϧλΠϜ൓өػೳͳͲ࢖͍উख͕ྑ͔ͬͨͷͰ͜Ε͸ͳ͔ͳ͔͍͍ͷͰ͸ͱ ࢥ͍ͬͯͨ » جຊ͸GrafonnetͰදݱ͠ɺgrafanaDashboards:: ഑ԼʹμογϡϘʔυΦϒδΣΫτ Λ഑ஔͯ͠apply͢ΔͱμογϡϘʔυʹ൓ө͞ΕΔ » grr watchͰϑΝΠϧͷมߋݕ஌ͯ͠ଈ࠲ʹremoteͷGrafanaΠϯελϯεʹ൓өͳͲɺ ։ൃମݧ͕Α͔ͬͨ » ͦͯ͠2021-02-27 v0.1.0͕ϦϦʔε͞Εͨ

Slide 14

Slide 14 text

Grizzy 0.1.0 » ٸʹه๏͕k8sελΠϧͷYAMLʹͳͬͯhidden element͕ඇਪ঑ͱͳͬͨɾɾ

Slide 15

Slide 15 text

Jsonnet + GrafonnetͰ΋Ҿ͖ଓ͖࢖͑·͢ (0.4.1ݱࡏ) » dashboardఆٛΛspecʹೖΕΔܗͰJsonnetͷ··ͰOK local g = import 'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet'; local dashboard = g.dashboard.new("my dahsboard"); ... { apiVersion: 'grizzly.grafana.com/v1alpha1', kind: 'Dashboard', metadata: { name: "dashboard name", folder: "dashboard folder" }, spec: dashboard }

Slide 16

Slide 16 text

grr serve(from 0.4.0) » ϩʔΧϧͰGrafanaཱͯͯϦϞʔτͷGrafanaΛ มߋ͢Δ͜ͱͳ͘μογϡϘʔυฤू͕Ͱ͖Δ » ϦΫΤετ͸ϓϩΩγ͞ΕΔͷͰϦϞʔτͷ σʔλιʔε͕࢖͑Δ » UI্Ͱอଘ͢ΔͱϩʔΧϧͷμογϡϘʔυ yaml͕ߋ৽͞ΕΔʢ!?ʣ » ͱΓ͋͑ͣ਽ܗΛ࡞ͬͯɺμογϡϘʔυΛखಈ ฤूͭͭ͠όʔδϣϯ؅ཧ͢ΔɺͳͲ͕Ͱ͖· ͢ɺศར

Slide 17

Slide 17 text

GrizzlyͷΠϚΠνͳ఺ » Τϥʔϝοηʔδ͕ෆ਌੾ » GrafanaͷΫϨσϯγϟϧ͕ෆ଍͍ͯ͠Δঢ়ଶͰ invalid memory address ͱ͔ੜͬΆ͍Τϥʔ͕ग़ͨΓ » എܠͱͯ͠࠷ۙ։ൃׂ͕ͱεϐʔσΟʔʹਐΜͰͦ͏ɺ൓໘࢓༷͸·ͩ҆ఆ͠ ͯͳ͍ » υΩϡϝϯτ͕؆ૉͰॳखͷ࢖͍ํ͕Α͘Θ͔Βͳ͍ » ೔ຊޠ৘ใ΋૿΍͍͖͍ͯͨ͠Ͱ͢Ͷ

Slide 18

Slide 18 text

࡞ͬͨμογϡϘʔυͷ ঺հ

Slide 19

Slide 19 text

࡞ͬͨμογϡϘʔυͷ঺հ » systemd_resolved » sysload » process dashbaord with treemap

Slide 20

Slide 20 text

࡞ͬͨμογϡϘʔυͷ঺հ » systemd_resolvedʢ͜Εͱʣ » sysload » process dashbaord with treemapʢ͜Εͷ࿩Λগ͠ʣ

Slide 21

Slide 21 text

systemd_resolved https://grafana.com/grafana/dashboards/15816-systemd-resolved-exporter/

Slide 22

Slide 22 text

systemd_resolved » ๯಄Ͱ࿩ͨ͠೰Έͷճऩ » systemtd_resolvedͷϝτϦΫεΛऔΔͨΊʹಠཱͨ͠exporterΛ2022೥ʹॻ͍ͨ » https://github.com/egmc/systemd_resolved_exporter » ࣌Λܦͯcommunityϝϯςͱͳͬͨsystemd_exporterʹಉ༷ͷ࣮૷Λߦ͏PRΛ ೖΕͯϚʔδ͞Εͨ » σϞ༻ʹμογϡϘʔυ࡞͚ͬͨͲ·ͩʢsystemd_exporterͷํͰ͸ʣ࢖͑ͳ ͍ɺϦϦʔε͞ΕͨΒྑ͔ͬͨΒ࢖ͬͯΈ͍ͯͩ͘͞ɾɾ

Slide 23

Slide 23 text

process dashbaord with treemap https://grafana.com/grafana/dashboards/13882-process-exporter-dashboard-with-treemap/

Slide 24

Slide 24 text

process dashbaord with treemap » DatadogͷμογϡϘʔυʹ͋ΔϝϞϦϚοϓతͳλΠϧදࣔΛߦ͏ෳ߹άϥϑ͕Α͔ͬͨͷͰਅࣅͯ࡞ͬͨ » process-exporeter2ͷϝτϦΫεΛදࣔ͢Δ » /proc ҎԼͷ৘ใΛ΋ͱʹ » ೚ҙʹάϧʔϐϯάͨ͠ϓϩηε͝ͱͷϝϞϦ/cpu/io/uptimeͳͲ » ݸਓతʹಛʹ࢖͍͍ͨͷ͸ϝϞϦͷ࢖༻ঢ়گͱͦͷมԽ » ͳ͔ͳ͔ศརʹ࢖͑ͨͷͰࣗࣾͷ؀ڥʹ΋࣋ͪࠐΜͰ࢖͍ͬͯΔ 2 process-exporter(https://github.com/ncabatoff/process-exporter)

Slide 25

Slide 25 text

ϝϞϦϦʔΫͷՄࢹԽͱ͔

Slide 26

Slide 26 text

OOMൃੜલͷঢ়گ֬ೝͱ͔

Slide 27

Slide 27 text

μογϡϘʔυ ΋ͬͱڞ༗͞Εͯ΄͍͠ » ެࣜαΠτͷํ΋গ͠վળ͞Ε͖ͯͨ » ΧςΰϦ·ΘΓ͸΋͏ͪΐ͍΄͍͠ » ͱ͸͍͑ݟ͚ͭΔͷ͸େมͳͷͰڞ༗ɺ ൃ৴͕૿͑Δͱخ͍͠ » ΍͍͖ͬͯ·͠ΐ͏

Slide 28

Slide 28 text

·ͱΊ

Slide 29

Slide 29 text

·ͱΊ » μογϡϘʔυ։ൃ͸ָ͍͠ » ͕ɺ͋·Γ։ൃɺڞ༗ࣄྫ͕ྲྀΕͯ͜ͳ͍ʢؾ͕͢ΔʣɺͷͰڞ༗͍͖ͯ͠ ·͠ΐ͏ » πʔϧ͸৭ʑ͋Δ͕ɺݸਓతʹ͸·ͣ͸खಈ࡞੒Ͱ » Grizzly͸ෳ਺ͷ࡞๏Ͱ࢖͑ΔͷͰ͕͠ΒΈ͕ͳ͚Ε͹͓͢͢ΊͰ͢

Slide 30

Slide 30 text

Thank you for listening