Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building Dashboards as a Hobby
Search
Sohei Iwahori
April 24, 2024
Technology
0
590
Building Dashboards as a Hobby
趣味のダッシュボード開発とDashboards as codeまわりのお話
Sohei Iwahori
April 24, 2024
Tweet
Share
More Decks by Sohei Iwahori
See All by Sohei Iwahori
Runbookに何を書き、どのようにアラートを振り分けるか?
egmc
2
2.8k
プロダクション環境の信頼性を損ねず観測する技術
egmc
8
9.4k
practices-for-making-alerts-actionable.pdf
egmc
3
7.7k
Other Decks in Technology
See All in Technology
コンテナ・K8s研修 - 後半 Kubernetes 基礎&ハンズオン【MIXI 24新卒技術研修】
mixi_engineers
PRO
1
120
Amazon FSx for NetApp ONTAPのパフォーマンスチューニング要素をまとめてみた #cm_odyssey #devio2024
non97
0
220
Matterport を使ってクラスメソッド各拠点のバーチャルオフィスツアーを作成してみた
wakatsuki
0
160
セキュリティ研修 Day1【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
160
ABEMAにおけるLLMを用いたコンテンツベース推薦システム導入と効果検証
cyberagentdevelopers
PRO
1
720
Luupの開発組織におけるインシデントマネジメントの変遷 ver.RoadtoSRENEXT2024
grimoh
1
270
【基調講演】変える、今ここから ― IoTとAIで紡ぐ未来
soracom
PRO
0
320
dxd2024-生成AIに振り回された3か月間の成功と失敗/dxd2024-link-and-motivation
lmi
2
260
[I/O Extended Android 2024] What`s new in Android 2024
kyeongwan
0
220
RAGのサービスをリリースして1年3ヶ月が経ちました
segavvy
4
910
簡単に始めるSnowflakeの機械学習
nayuts
1
190
MySQLのロックの種類とその競合
yoku0825
6
1.6k
Featured
See All Featured
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
277
13k
We Have a Design System, Now What?
morganepeng
46
7k
RailsConf 2023
tenderlove
16
720
Build The Right Thing And Hit Your Dates
maggiecrowley
28
2.2k
Stop Working from a Prison Cell
hatefulcrawdad
266
20k
How GitHub (no longer) Works
holman
305
140k
Agile that works and the tools we love
rasmusluckow
325
20k
Leading Effective Engineering Teams 2024
addyosmani
3
300
Designing the Hi-DPI Web
ddemaree
276
34k
Bash Introduction
62gerente
607
210k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
149
45k
Building a Modern Day E-commerce SEO Strategy
aleyda
25
6.7k
Transcript
Building Dashboards as a Hobby 2024/04/24 Sohei Iwahori (GREE, Inc.)
who? » Sohei Iwahori (@egmc) » GREE, Inc. » Πϯϑϥ
/ Monitoring Unit Leader / Senior Lead Engineer » ओʹήʔϜͷΠϯϑϥͱࢹγεςϜ » σʔλιʔεओʹPrometheus
࠷ۙͷΈ » systemd_exporterʹsystemd_resolved ͷϝτϦΫεΛՃ͢ΔPR͕Ϛʔδ ͞Εͨ(1݄) » ͕৽όʔδϣϯ͕ͳ͔ͳ͔ϦϦʔε ͞Εͳ͍ » ޙͷϑϦ
࠷ۙͷΈ
agenda » μογϡϘʔυͷຽओԽ » झຯͷμογϡϘʔυ࡞Ͱ͏πʔϧ » ࡞ͬͨμογϡϘʔυͷհ » ·ͱΊ
μογϡϘʔυͷຽओԽ
GrafanaͷμογϡϘʔυͷخ͠͞ » ࢥ͍͍ͭͨμογϡϘʔυΛUI্͔Β࡞Ͱ͖Δ » τϥΠΞϯυΤϥʔͷ͞ » RRDtoolͰٯϙʔϥϯυه๏ͰάϥϑΛॻ͔࣌͘Βѹతͳརศੑ
RRDTool -> Grafana + PromQL
झຯͷμογϡϘʔυ࡞Ͱ ͏πʔϧ
μογϡϘʔυ։ൃͷΤίγεςϜ » Grafana as code ৭ʑ͋Δɾɾ1 » Grafana Terraform provider
» Grafana Ansible collection » Grizzly » Grafana Crossplane provider » Kubernetes Grafana Operator » ͦͷ΄͔ScenesͳͲ 1 A complete guide to managing Grafana as code: tools, tips, and tricks Ishan Jain.
ͲΕʹ͢Δ͔ » ਖ਼ͦΜͳʹ͜Ε͕͍͍ʂΈ͍ͨͳͷͳ͍ؾ͕͢ΔͷͰϒϩάͷ௨Γطଘͷ ίʔυࢿ࢈ʹ߹ΘͤͯͰ » τϥΠΞϯυΤϥʔखಈͰɺظతʹ࡞ΓࠐΈ͍ͨ߹ʹίʔυԽ » ͱ͍͑JSONܗࣜͳΒ݁ہϕʔεΛखͰ࡞ΔɺͰ » ݸਓͰ͓खܰʹΔʹखಈ࡞+Ұ෦GrizzlyΛར༻
Grizzly » https://github.com/grafana/grizzly » Grafana։ൃͷcliπʔϧʢgrrίϚϯυʣ » ݱࡏDashbaordཧ͚ͩͰͳ͘σʔλιʔεɺGrafana Cloud prometheus ͷrules/alertsɺSynthetic
MonitoringͷཧʹରԠ » v0.4.1࣌ » جຊͷه๏YAMLɺJSONɺJsonnetʢ+Grafonnetʣ͔ͭ͑Δ
Grizzyͱͷग़ձ͍ » 2020ࠒ0.0.1ͷalpha࣌ͰJsonnet(+Grafonnet)Ͱͷར༻ΨΠυ͕͋Γɺ͍࢝Ίͨ » લड़ͷϦΞϧλΠϜөػೳͳͲ͍উख͕ྑ͔ͬͨͷͰ͜Εͳ͔ͳ͔͍͍ͷͰͱ ࢥ͍ͬͯͨ » جຊGrafonnetͰදݱ͠ɺgrafanaDashboards:: ԼʹμογϡϘʔυΦϒδΣΫτ Λஔͯ͠apply͢ΔͱμογϡϘʔυʹө͞ΕΔ
» grr watchͰϑΝΠϧͷมߋݕͯ͠ଈ࠲ʹremoteͷGrafanaΠϯελϯεʹөͳͲɺ ։ൃମݧ͕Α͔ͬͨ » ͦͯ͠2021-02-27 v0.1.0͕ϦϦʔε͞Εͨ
Grizzy 0.1.0 » ٸʹه๏͕k8sελΠϧͷYAMLʹͳͬͯhidden element͕ඇਪͱͳͬͨɾɾ
Jsonnet + GrafonnetͰҾ͖ଓ͖͑·͢ (0.4.1ݱࡏ) » dashboardఆٛΛspecʹೖΕΔܗͰJsonnetͷ··ͰOK local g = import
'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet'; local dashboard = g.dashboard.new("my dahsboard"); ... { apiVersion: 'grizzly.grafana.com/v1alpha1', kind: 'Dashboard', metadata: { name: "dashboard name", folder: "dashboard folder" }, spec: dashboard }
grr serve(from 0.4.0) » ϩʔΧϧͰGrafanaཱͯͯϦϞʔτͷGrafanaΛ มߋ͢Δ͜ͱͳ͘μογϡϘʔυฤू͕Ͱ͖Δ » ϦΫΤετϓϩΩγ͞ΕΔͷͰϦϞʔτͷ σʔλιʔε͕͑Δ »
UI্Ͱอଘ͢ΔͱϩʔΧϧͷμογϡϘʔυ yaml͕ߋ৽͞ΕΔʢ!?ʣ » ͱΓ͋͑ͣܗΛ࡞ͬͯɺμογϡϘʔυΛखಈ ฤूͭͭ͠όʔδϣϯཧ͢ΔɺͳͲ͕Ͱ͖· ͢ɺศར
GrizzlyͷΠϚΠνͳ » Τϥʔϝοηʔδ͕ෆ » GrafanaͷΫϨσϯγϟϧ͕ෆ͍ͯ͠Δঢ়ଶͰ invalid memory address ͱ͔ੜͬΆ͍Τϥʔ͕ग़ͨΓ »
എܠͱͯ͠࠷ۙ։ൃׂ͕ͱεϐʔσΟʔʹਐΜͰͦ͏ɺ໘༷·ͩ҆ఆ͠ ͯͳ͍ » υΩϡϝϯτ͕؆ૉͰॳखͷ͍ํ͕Α͘Θ͔Βͳ͍ » ຊޠใ૿͍͖͍ͯͨ͠Ͱ͢Ͷ
࡞ͬͨμογϡϘʔυͷ հ
࡞ͬͨμογϡϘʔυͷհ » systemd_resolved » sysload » process dashbaord with treemap
࡞ͬͨμογϡϘʔυͷհ » systemd_resolvedʢ͜Εͱʣ » sysload » process dashbaord with treemapʢ͜ΕͷΛগ͠ʣ
systemd_resolved https://grafana.com/grafana/dashboards/15816-systemd-resolved-exporter/
systemd_resolved » ಄Ͱͨ͠Έͷճऩ » systemtd_resolvedͷϝτϦΫεΛऔΔͨΊʹಠཱͨ͠exporterΛ2022ʹॻ͍ͨ » https://github.com/egmc/systemd_resolved_exporter » ࣌Λܦͯcommunityϝϯςͱͳͬͨsystemd_exporterʹಉ༷ͷ࣮Λߦ͏PRΛ ೖΕͯϚʔδ͞Εͨ
» σϞ༻ʹμογϡϘʔυ࡞͚ͬͨͲ·ͩʢsystemd_exporterͷํͰʣ͑ͳ ͍ɺϦϦʔε͞ΕͨΒྑ͔ͬͨΒͬͯΈ͍ͯͩ͘͞ɾɾ
process dashbaord with treemap https://grafana.com/grafana/dashboards/13882-process-exporter-dashboard-with-treemap/
process dashbaord with treemap » DatadogͷμογϡϘʔυʹ͋ΔϝϞϦϚοϓతͳλΠϧදࣔΛߦ͏ෳ߹άϥϑ͕Α͔ͬͨͷͰਅࣅͯ࡞ͬͨ » process-exporeter2ͷϝτϦΫεΛදࣔ͢Δ » /proc
ҎԼͷใΛͱʹ » ҙʹάϧʔϐϯάͨ͠ϓϩηε͝ͱͷϝϞϦ/cpu/io/uptimeͳͲ » ݸਓతʹಛʹ͍͍ͨͷϝϞϦͷ༻ঢ়گͱͦͷมԽ » ͳ͔ͳ͔ศརʹ͑ͨͷͰࣗࣾͷڥʹ࣋ͪࠐΜͰ͍ͬͯΔ 2 process-exporter(https://github.com/ncabatoff/process-exporter)
ϝϞϦϦʔΫͷՄࢹԽͱ͔
OOMൃੜલͷঢ়گ֬ೝͱ͔
μογϡϘʔυ ͬͱڞ༗͞Εͯ΄͍͠ » ެࣜαΠτͷํগ͠վળ͞Ε͖ͯͨ » ΧςΰϦ·ΘΓ͏ͪΐ͍΄͍͠ » ͱ͍͑ݟ͚ͭΔͷେมͳͷͰڞ༗ɺ ൃ৴͕૿͑Δͱخ͍͠ »
͍͖ͬͯ·͠ΐ͏
·ͱΊ
·ͱΊ » μογϡϘʔυ։ൃָ͍͠ » ͕ɺ͋·Γ։ൃɺڞ༗ࣄྫ͕ྲྀΕͯ͜ͳ͍ʢؾ͕͢ΔʣɺͷͰڞ༗͍͖ͯ͠ ·͠ΐ͏ » πʔϧ৭ʑ͋Δ͕ɺݸਓతʹ·ͣखಈ࡞Ͱ » Grizzlyෳͷ࡞๏Ͱ͑ΔͷͰ͕͠ΒΈ͕ͳ͚Ε͓͢͢ΊͰ͢
Thank you for listening