Dashboards as Code Sanooj Mananghat RootConf Data Engineer @ Intuit Hyderabad 

No content

No content

Demo Day 

Scenario I: Teamwork in Dashboard 

Problems in Manually created Dashboards - Human errors - No Meaningful History - No Rollbacks - Incosistent Dashbaords 

Scenario II : Concurrent Modification - Difficult to Collaborate - Introduces Conflicts. 

Scenarios III: External Consumers 

Problems in Manually created Dashboards - Lacs Hygiene - No separate test environment - No/Minimal Peer Review System 

DRY - Effects productivity - Zero value add - Difficult to scale 

Problem Statement • Difficult to collaborate • Difficult to Scale • No/Minimal Peer Review • Effects Productivity • Introduces conflicts • Inconsistent • No Meaningful History • Untracked Temporary changes • No Rollback • Lacks Hygiene 

Process • Process with Source control 

Easiest Approach • Dashboards store data in JSON files. 

Bloaty JSON 

Dashboards as JSON - Bloaty huge JSON files - PR review became impossible - Duplicate blocks 

How can we squeeze the elephant into the car trunk? 

JSONNET Templates • Reduce Boilerplate and redundant code blocks 

JSONET Loops • { "person1": { "name": "Alice", "welcome": "Hello Alice!" }, "person2": { "name": "Bob", "welcome": "Hello Bob!" }, "person3": { "name": "Jack", "welcome": "Hello Jack!" }, "person4": { "name": "Jill", "welcome": "Hello Jill!" }, "person5": { "name": "Pokemon", "welcome": "Hello Pokemon!" }, "person6": { "name": "Dora", "welcome": "Hello Dora!" }, "person7": { "name": "Jhony", "welcome": "Hello Jhony!" }, "person8": { "name": "Humpty", "welcome": "Hello Humpty!" }, "person99": { "name": "Dumpty", "welcome": "Hello Dumpty!" }, "person10": { "name": "Tom", "welcome": "Hello Tom!" } } { person1: { name: "Alice", welcome: "Hello " + self.name + "!", }, person2: self.person1 { name: “Bob”}, person3: self.person1 { name: “Jack" }, person4: self.person1 { name: “Jill" }, person5: self.person1 { name: “Pokemon" }, person6: self.person1 { name: “Dora" }, person7: self.person1 { name: “Jhony" }, person8: self.person1 { name: “Humty" }, person9: self.person1 { name: “Dumpty" }, person10: self.person1 { name: “Tom" }, } 

Architecture 

Dashboards as JSONNET - Simplest solution with less overhead. - Learning a new language. - Provisioning is still a challenge. - Jsonnet for grafana: grafonnet-lib 

Manual Provisioning • Provisioning is still a challenge. 

What? 

Terraform • Terraform providers/plugins • terraform-provider-grafana • terraform-provider-kibana • terraform-provider-wavefront 

Show me the code! resource "wavefront_dashboard" "test_dashboard" { name = "Terraform 12 Dashboard" url = "tf12-test" dynamic "section" { for_each = var.sections content { name = section.key dynamic "row" { iterator = rows for_each = section.value content { chart { name = rows.value.name description = rows.value.description units = "unit" source { name = rows.value.source query = rows.value.query } chart_setting { type = "line" } summarization = "MEAN” } } } } } } • terraform apply For loop 

Architecture 

Dashboards as Code - No dashboard specialist - Introduced strict peer review concept - Reviewing became meaningful - Consistent and collaborative. - Easily scalable. - Rollbacks become easier. - No repeated works, no clicks. 

Dashboards as Code - Oncall more easier with consistent dashboard. - Less cognitive overhead. - Visual Diff with test and prod environment - Templatizing helped similar dashboard for similar services. - Monitoring as code 

Wrapping Up • Initial Onboarding time. • Moved majority of our dashboards to Terraform • Moving further to Monitoring as Code. • Grafana-Lib (JSONNET based solution) • https://github.com/grafana/grafonnet-lib • Grafana python module - from Weave works • https://github.com/weaveworks/grafanalib • Grafana dashboard generator - from Uber • https://github.com/uber/grafana-dash-gen 

And they lived happily ever after!! 

Sanooj Mananghat Data Engineer @ Intuit