Tran Tuan Linh - 2020/10 CentralDogma Flash’s configuration backbone

About me • I am Linh, from Observability Team • github.com/linxGnu • My team deals with: • Traces, Logs, and Metrics

Impact Measurement Alerting Trending Root cause Analysis Anomaly Detection cpu [host=10.58.15.27] {
 timestamp: 1600649643,
 value: 0.95 /* 95% */ }

Challenges • “Millions” metrics, “Billions” data points per day • Resolution variety • Thus, application/server may export a metric every 60s/30s/15s • 60s -> 1 point per minute • 30s -> 2 points per minute • 15s -> 4 points per minute • 10s -> 6 points per minute • So, higher resolution (shorter interval) -> more points • But give us more detail what’s going on with servers/services/apps • Metrics Storage must be fast for realtime usage (graphing, alerting)

Journey • For various reasons, in 2018, we decide to take a ride • Build in-house distributed metrics storage from scratch • Named “Flash” - Avengers hero

Flash’s Concern • Configuration management is another challenge • Successful distributed software system needs effective configuration management • Planning, identifying, tracking and verifying changes in configurations • Maintaining configuration integrity

Flash’s Concern • What if configuration “looks” like “git” • Changes by "Pull Request” • Workflow through “Review” • Applying changes by “Merge” • Tracking with “Commits” • Great team collaboration

• To the rescue • An open-source HA version-controlled service configuration repository based on Git, ZooKeeper and HTTP/2 • Features: • Store configuration files into “Git” (such as .json, .xml, .yaml) • PR for changes, get it reviewed, and merged for applying • Realtime updates-notification • Once PR got merged, services/apps are notified, no reboot

No content

No content

No content

Cluster Topology Flash - CentralDogma Raft Consensus Shard

Shard 1 Shard 2 Shard N … Cluster Topology Raft Configs

No content

Metric Input Ingestor Querier Downsampler Save/query/dump

Provisioning Flash - CentralDogma • Git PR makes it all • Scalability: • Add/remove a shard • Add/remove a node in shard • User provisioning: authorization, query scope, etc • On-the-fly, no servers reboot • Realtime feature adjustment: • Enable/Disable input source(s) • Enable/Disable module • Rate limit adjustment • …

No content

Other configurations Flash - CentralDogma • Also stored in git repo, managed through CentralDogma • Flash deployment package: • Binaries only • No file config!

Finally, with the help of CentralDogma

Keynotes • CentralDogma provides native pure Java/Go client library • And more in the future • Awesome, most of Flash modules are written in Go :) • Zero runtime overhead • Clean and deadly simple API • Reliability, Stability • Cost reduction

Keynotes • CentralDogma authors • They are friendly, supportive and professional • Stay in user shoes • Problems are often solved quickly • Up to date with patches and detail guidelines

I bet you will love it “Why don’t give CentralDogma a try”

Thank you for listening