Building Dashboards as a Hobby

趣味のダッシュボード開発とDashboards as codeまわりのお話

Transcript

1. Building Dashboards as a Hobby 2024/04/24 Sohei Iwahori (GREE, Inc.)
2. who? » Sohei Iwahori (@egmc) » GREE, Inc. » Πϯϑϥ / Monitoring Unit Leader / Senior Lead Engineer » ओʹήʔϜͷΠϯϑϥͱ؂ࢹγεςϜ » σʔλιʔε͸ओʹPrometheus
3. ࠷ۙͷ೰Έ » systemd_exporterʹsystemd_resolved ͷϝτϦΫεΛ௥Ճ͢ΔPR͕Ϛʔδ ͞Εͨ(1݄) » ͕৽όʔδϣϯ͕ͳ͔ͳ͔ϦϦʔε ͞Εͳ͍ » ޙ൒΁ͷϑϦ
4. ࠷ۙͷ೰Έ
5. agenda » μογϡϘʔυͷຽओԽ » झຯͷμογϡϘʔυ࡞੒Ͱ࢖͏πʔϧ » ࡞ͬͨμογϡϘʔυͷ঺հ » ·ͱΊ
6. μογϡϘʔυͷຽओԽ
7. GrafanaͷμογϡϘʔυͷخ͠͞ » ࢥ͍͍ͭͨμογϡϘʔυΛUI্͔Β࡞੒Ͱ͖Δ » τϥΠΞϯυΤϥʔͷ͸΍͞ » RRDtoolͰٯϙʔϥϯυه๏ͰάϥϑΛॻ࣌͘୅͔Β͸ѹ౗తͳརศੑ
8. RRDTool -> Grafana + PromQL
9. झຯͷμογϡϘʔυ࡞੒Ͱ ࢖͏πʔϧ
10. μογϡϘʔυ։ൃͷΤίγεςϜ » Grafana as code ৭ʑ͋Δɾɾ1 » Grafana Terraform provider » Grafana Ansible collection » Grizzly » Grafana Crossplane provider » Kubernetes Grafana Operator » ͦͷ΄͔ScenesͳͲ΋ 1 A complete guide to managing Grafana as code: tools, tips, and tricks Ishan Jain.
11. ͲΕʹ͢Δ͔ » ਖ਼௚ͦΜͳʹ͜Ε͕͍͍ʂΈ͍ͨͳͷͳ͍ؾ͕͢ΔͷͰϒϩάͷ௨Γطଘͷ ίʔυࢿ࢈ʹ߹ΘͤͯͰ » τϥΠΞϯυΤϥʔ͸खಈͰɺ௕ظతʹ࡞ΓࠐΈ͍ͨ৔߹ʹίʔυԽ » ͱ͸͍͑JSONܗࣜͳΒ݁ہϕʔεΛखͰ࡞ΔɺͰ΋ » ݸਓͰ͓खܰʹ΍Δʹ͸खಈ࡞੒+Ұ෦GrizzlyΛར༻
12. Grizzly » https://github.com/grafana/grizzly » Grafana։ൃͷcliπʔϧʢgrrίϚϯυʣ » ݱࡏ͸Dashbaord؅ཧ͚ͩͰͳ͘σʔλιʔεɺGrafana Cloud prometheus ͷrules/alertsɺSynthetic Monitoringͷ؅ཧʹ΋ରԠ » v0.4.1࣌఺ » جຊͷه๏͸YAMLɺJSONɺJsonnetʢ+Grafonnetʣ΋͔ͭ͑Δ
13. Grizzyͱͷग़ձ͍ » 2020೥ࠒ0.0.1ͷalpha࣌఺ͰJsonnet(+Grafonnet)Ͱͷར༻ΨΠυ͕͋Γɺ࢖͍࢝Ίͨ » લड़ͷϦΞϧλΠϜ൓өػೳͳͲ࢖͍উख͕ྑ͔ͬͨͷͰ͜Ε͸ͳ͔ͳ͔͍͍ͷͰ͸ͱ ࢥ͍ͬͯͨ » جຊ͸GrafonnetͰදݱ͠ɺgrafanaDashboards:: ഑ԼʹμογϡϘʔυΦϒδΣΫτ Λ഑ஔͯ͠apply͢ΔͱμογϡϘʔυʹ൓ө͞ΕΔ » grr watchͰϑΝΠϧͷมߋݕ஌ͯ͠ଈ࠲ʹremoteͷGrafanaΠϯελϯεʹ൓өͳͲɺ ։ൃମݧ͕Α͔ͬͨ » ͦͯ͠2021-02-27 v0.1.0͕ϦϦʔε͞Εͨ
14. Grizzy 0.1.0 » ٸʹه๏͕k8sελΠϧͷYAMLʹͳͬͯhidden element͕ඇਪ঑ͱͳͬͨɾɾ
15. Jsonnet + GrafonnetͰ΋Ҿ͖ଓ͖࢖͑·͢ (0.4.1ݱࡏ) » dashboardఆٛΛspecʹೖΕΔܗͰJsonnetͷ··ͰOK local g = import 'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet'; local dashboard = g.dashboard.new("my dahsboard"); ... { apiVersion: 'grizzly.grafana.com/v1alpha1', kind: 'Dashboard', metadata: { name: "dashboard name", folder: "dashboard folder" }, spec: dashboard }
16. grr serve(from 0.4.0) » ϩʔΧϧͰGrafanaཱͯͯϦϞʔτͷGrafanaΛ มߋ͢Δ͜ͱͳ͘μογϡϘʔυฤू͕Ͱ͖Δ » ϦΫΤετ͸ϓϩΩγ͞ΕΔͷͰϦϞʔτͷ σʔλιʔε͕࢖͑Δ » UI্Ͱอଘ͢ΔͱϩʔΧϧͷμογϡϘʔυ yaml͕ߋ৽͞ΕΔʢ!?ʣ » ͱΓ͋͑ͣ਽ܗΛ࡞ͬͯɺμογϡϘʔυΛखಈ ฤूͭͭ͠όʔδϣϯ؅ཧ͢ΔɺͳͲ͕Ͱ͖· ͢ɺศར
17. GrizzlyͷΠϚΠνͳ఺ » Τϥʔϝοηʔδ͕ෆ਌੾ » GrafanaͷΫϨσϯγϟϧ͕ෆ଍͍ͯ͠Δঢ়ଶͰ invalid memory address ͱ͔ੜͬΆ͍Τϥʔ͕ग़ͨΓ » എܠͱͯ͠࠷ۙ։ൃׂ͕ͱεϐʔσΟʔʹਐΜͰͦ͏ɺ൓໘࢓༷͸·ͩ҆ఆ͠ ͯͳ͍ » υΩϡϝϯτ͕؆ૉͰॳखͷ࢖͍ํ͕Α͘Θ͔Βͳ͍ » ೔ຊޠ৘ใ΋૿΍͍͖͍ͯͨ͠Ͱ͢Ͷ
18. ࡞ͬͨμογϡϘʔυͷ ঺հ
19. ࡞ͬͨμογϡϘʔυͷ঺հ » systemd_resolved » sysload » process dashbaord with treemap
20. ࡞ͬͨμογϡϘʔυͷ঺հ » systemd_resolvedʢ͜Εͱʣ » sysload » process dashbaord with treemapʢ͜Εͷ࿩Λগ͠ʣ
21. systemd_resolved https://grafana.com/grafana/dashboards/15816-systemd-resolved-exporter/
22. systemd_resolved » ๯಄Ͱ࿩ͨ͠೰Έͷճऩ » systemtd_resolvedͷϝτϦΫεΛऔΔͨΊʹಠཱͨ͠exporterΛ2022೥ʹॻ͍ͨ » https://github.com/egmc/systemd_resolved_exporter » ࣌Λܦͯcommunityϝϯςͱͳͬͨsystemd_exporterʹಉ༷ͷ࣮૷Λߦ͏PRΛ ೖΕͯϚʔδ͞Εͨ » σϞ༻ʹμογϡϘʔυ࡞͚ͬͨͲ·ͩʢsystemd_exporterͷํͰ͸ʣ࢖͑ͳ ͍ɺϦϦʔε͞ΕͨΒྑ͔ͬͨΒ࢖ͬͯΈ͍ͯͩ͘͞ɾɾ
23. process dashbaord with treemap https://grafana.com/grafana/dashboards/13882-process-exporter-dashboard-with-treemap/
24. process dashbaord with treemap » DatadogͷμογϡϘʔυʹ͋ΔϝϞϦϚοϓతͳλΠϧදࣔΛߦ͏ෳ߹άϥϑ͕Α͔ͬͨͷͰਅࣅͯ࡞ͬͨ » process-exporeter2ͷϝτϦΫεΛදࣔ͢Δ » /proc ҎԼͷ৘ใΛ΋ͱʹ » ೚ҙʹάϧʔϐϯάͨ͠ϓϩηε͝ͱͷϝϞϦ/cpu/io/uptimeͳͲ » ݸਓతʹಛʹ࢖͍͍ͨͷ͸ϝϞϦͷ࢖༻ঢ়گͱͦͷมԽ » ͳ͔ͳ͔ศརʹ࢖͑ͨͷͰࣗࣾͷ؀ڥʹ΋࣋ͪࠐΜͰ࢖͍ͬͯΔ 2 process-exporter(https://github.com/ncabatoff/process-exporter)
25. ϝϞϦϦʔΫͷՄࢹԽͱ͔
26. OOMൃੜલͷঢ়گ֬ೝͱ͔
27. μογϡϘʔυ ΋ͬͱڞ༗͞Εͯ΄͍͠ » ެࣜαΠτͷํ΋গ͠վળ͞Ε͖ͯͨ » ΧςΰϦ·ΘΓ͸΋͏ͪΐ͍΄͍͠ » ͱ͸͍͑ݟ͚ͭΔͷ͸େมͳͷͰڞ༗ɺ ൃ৴͕૿͑Δͱخ͍͠ » ΍͍͖ͬͯ·͠ΐ͏
28. ·ͱΊ
29. ·ͱΊ » μογϡϘʔυ։ൃ͸ָ͍͠ » ͕ɺ͋·Γ։ൃɺڞ༗ࣄྫ͕ྲྀΕͯ͜ͳ͍ʢؾ͕͢ΔʣɺͷͰڞ༗͍͖ͯ͠ ·͠ΐ͏ » πʔϧ͸৭ʑ͋Δ͕ɺݸਓతʹ͸·ͣ͸खಈ࡞੒Ͱ » Grizzly͸ෳ਺ͷ࡞๏Ͱ࢖͑ΔͷͰ͕͠ΒΈ͕ͳ͚Ε͹͓͢͢ΊͰ͢
30. Thank you for listening