CentralDogma as powerful realtime configuration

Transcript

1. Tran Tuan Linh - 2020/10 CentralDogma Flash’s configuration backbone

2. About me • I am Linh, from Observability Team •

   github.com/linxGnu • My team deals with: • Traces, Logs, and Metrics

3. Impact Measurement Alerting Trending Root cause Analysis Anomaly Detection cpu

   [host=10.58.15.27] {
 timestamp: 1600649643,
 value: 0.95 /* 95% */ }

4. Challenges • “Millions” metrics, “Billions” data points per day •

   Resolution variety • Thus, application/server may export a metric every 60s/30s/15s • 60s -> 1 point per minute • 30s -> 2 points per minute • 15s -> 4 points per minute • 10s -> 6 points per minute • So, higher resolution (shorter interval) -> more points • But give us more detail what’s going on with servers/services/apps • Metrics Storage must be fast for realtime usage (graphing, alerting)

5. Journey • For various reasons, in 2018, we decide to

   take a ride • Build in-house distributed metrics storage from scratch • Named “Flash” - Avengers hero

6. Flash’s Concern • Configuration management is another challenge • Successful

   distributed software system needs effective configuration management • Planning, identifying, tracking and verifying changes in configurations • Maintaining configuration integrity

7. Flash’s Concern • What if configuration “looks” like “git” •

   Changes by "Pull Request” • Workflow through “Review” • Applying changes by “Merge” • Tracking with “Commits” • Great team collaboration

8. • To the rescue • An open-source HA version-controlled service

   configuration repository based on Git, ZooKeeper and HTTP/2 • Features: • Store configuration files into “Git” (such as .json, .xml, .yaml) • PR for changes, get it reviewed, and merged for applying • Realtime updates-notification • Once PR got merged, services/apps are notified, no reboot
9. None
10. None
11. None

12. Cluster Topology Flash - CentralDogma Raft Consensus Shard

13. Shard 1 Shard 2 Shard N … Cluster Topology Raft

    Configs
14. None

15. Metric Input Ingestor Querier Downsampler Save/query/dump

16. Provisioning Flash - CentralDogma • Git PR makes it all

    • Scalability: • Add/remove a shard • Add/remove a node in shard • User provisioning: authorization, query scope, etc • On-the-fly, no servers reboot • Realtime feature adjustment: • Enable/Disable input source(s) • Enable/Disable module • Rate limit adjustment • …
17. None

18. Other configurations Flash - CentralDogma • Also stored in git

    repo, managed through CentralDogma • Flash deployment package: • Binaries only • No file config!

19. Finally, with the help of CentralDogma

20. Keynotes • CentralDogma provides native pure Java/Go client library •

    And more in the future • Awesome, most of Flash modules are written in Go :) • Zero runtime overhead • Clean and deadly simple API • Reliability, Stability • Cost reduction

21. Keynotes • CentralDogma authors • They are friendly, supportive and

    professional • Stay in user shoes • Problems are often solved quickly • Up to date with patches and detail guidelines

22. I bet you will love it “Why don’t give CentralDogma

    a try”

23. Thank you for listening