Dashboards as Code

Versioning dashboards and alerts using “Infrastructure as code” makes the infrastructure more robust learn about them.

Transcript

1. Dashboards as Code Sanooj Mananghat RootConf Data Engineer @ Intuit Hyderabad
4. Demo Day
5. Scenario I: Teamwork in Dashboard
6. Problems in Manually created Dashboards - Human errors - No Meaningful History - No Rollbacks - Incosistent Dashbaords
7. Scenario II : Concurrent Modification - Difficult to Collaborate - Introduces Conflicts.
8. Scenarios III: External Consumers
9. Problems in Manually created Dashboards - Lacs Hygiene - No separate test environment - No/Minimal Peer Review System
10. DRY - Effects productivity - Zero value add - Difficult to scale
11. Problem Statement • Difficult to collaborate • Difficult to Scale • No/Minimal Peer Review • Effects Productivity • Introduces conflicts • Inconsistent • No Meaningful History • Untracked Temporary changes • No Rollback • Lacks Hygiene
12. Process • Process with Source control
13. Easiest Approach • Dashboards store data in JSON files.
14. Bloaty JSON
15. Dashboards as JSON - Bloaty huge JSON files - PR review became impossible - Duplicate blocks
16. How can we squeeze the elephant into the car trunk?
17. JSONNET Templates • Reduce Boilerplate and redundant code blocks
18. JSONET Loops • { "person1": { "name": "Alice", "welcome": "Hello Alice!" }, "person2": { "name": "Bob", "welcome": "Hello Bob!" }, "person3": { "name": "Jack", "welcome": "Hello Jack!" }, "person4": { "name": "Jill", "welcome": "Hello Jill!" }, "person5": { "name": "Pokemon", "welcome": "Hello Pokemon!" }, "person6": { "name": "Dora", "welcome": "Hello Dora!" }, "person7": { "name": "Jhony", "welcome": "Hello Jhony!" }, "person8": { "name": "Humpty", "welcome": "Hello Humpty!" }, "person99": { "name": "Dumpty", "welcome": "Hello Dumpty!" }, "person10": { "name": "Tom", "welcome": "Hello Tom!" } } { person1: { name: "Alice", welcome: "Hello " + self.name + "!", }, person2: self.person1 { name: “Bob”}, person3: self.person1 { name: “Jack" }, person4: self.person1 { name: “Jill" }, person5: self.person1 { name: “Pokemon" }, person6: self.person1 { name: “Dora" }, person7: self.person1 { name: “Jhony" }, person8: self.person1 { name: “Humty" }, person9: self.person1 { name: “Dumpty" }, person10: self.person1 { name: “Tom" }, }
19. Architecture
20. Dashboards as JSONNET - Simplest solution with less overhead. - Learning a new language. - Provisioning is still a challenge. - Jsonnet for grafana: grafonnet-lib
21. Manual Provisioning • Provisioning is still a challenge.
22. What?
23. Terraform • Terraform providers/plugins • terraform-provider-grafana • terraform-provider-kibana • terraform-provider-wavefront
24. Show me the code! resource "wavefront_dashboard" "test_dashboard" { name = "Terraform 12 Dashboard" url = "tf12-test" dynamic "section" { for_each = var.sections content { name = section.key dynamic "row" { iterator = rows for_each = section.value content { chart { name = rows.value.name description = rows.value.description units = "unit" source { name = rows.value.source query = rows.value.query } chart_setting { type = "line" } summarization = "MEAN” } } } } } } • terraform apply For loop
25. Architecture
26. Dashboards as Code - No dashboard specialist - Introduced strict peer review concept - Reviewing became meaningful - Consistent and collaborative. - Easily scalable. - Rollbacks become easier. - No repeated works, no clicks.
27. Dashboards as Code - Oncall more easier with consistent dashboard. - Less cognitive overhead. - Visual Diff with test and prod environment - Templatizing helped similar dashboard for similar services. - Monitoring as code
28. Wrapping Up • Initial Onboarding time. • Moved majority of our dashboards to Terraform • Moving further to Monitoring as Code. • Grafana-Lib (JSONNET based solution) • https://github.com/grafana/grafonnet-lib • Grafana python module - from Weave works • https://github.com/weaveworks/grafanalib • Grafana dashboard generator - from Uber • https://github.com/uber/grafana-dash-gen
29. And they lived happily ever after!!
30. Sanooj Mananghat Data Engineer @ Intuit