CentralDogma as powerful realtime configuration

Tran Tuan Linh (LINE Corporation, Software Engineer)
Central Dogmaは、Git、ZooKeeperおよびHTTP/2をベースにした高可用性のバージョン管理サービスをサポートする設定リポジトリです。 FlashチームがCentralDogmaを使用してシステムを作成し、大量の時系列データワークロードを処理する方法についてご紹介します。

how Flash team use CentralDogma to create a system handle huge amount time series data workload

Transcript

1. Tran Tuan Linh - 2020/10 CentralDogma Flash’s configuration backbone
2. About me • I am Linh, from Observability Team • github.com/linxGnu • My team deals with: • Traces, Logs, and Metrics
3. Impact Measurement Alerting Trending Root cause Analysis Anomaly Detection cpu [host=10.58.15.27] {
 timestamp: 1600649643,
 value: 0.95 /* 95% */ }
4. Challenges • “Millions” metrics, “Billions” data points per day • Resolution variety • Thus, application/server may export a metric every 60s/30s/15s • 60s -> 1 point per minute • 30s -> 2 points per minute • 15s -> 4 points per minute • 10s -> 6 points per minute • So, higher resolution (shorter interval) -> more points • But give us more detail what’s going on with servers/services/apps • Metrics Storage must be fast for realtime usage (graphing, alerting)
5. Journey • For various reasons, in 2018, we decide to take a ride • Build in-house distributed metrics storage from scratch • Named “Flash” - Avengers hero
6. Flash’s Concern • Configuration management is another challenge • Successful distributed software system needs effective configuration management • Planning, identifying, tracking and verifying changes in configurations • Maintaining configuration integrity
7. Flash’s Concern • What if configuration “looks” like “git” • Changes by "Pull Request” • Workflow through “Review” • Applying changes by “Merge” • Tracking with “Commits” • Great team collaboration
8. • To the rescue • An open-source HA version-controlled service configuration repository based on Git, ZooKeeper and HTTP/2 • Features: • Store configuration files into “Git” (such as .json, .xml, .yaml) • PR for changes, get it reviewed, and merged for applying • Realtime updates-notification • Once PR got merged, services/apps are notified, no reboot
12. Cluster Topology Flash - CentralDogma Raft Consensus Shard
13. Shard 1 Shard 2 Shard N … Cluster Topology Raft Configs
15. Metric Input Ingestor Querier Downsampler Save/query/dump
16. Provisioning Flash - CentralDogma • Git PR makes it all • Scalability: • Add/remove a shard • Add/remove a node in shard • User provisioning: authorization, query scope, etc • On-the-fly, no servers reboot • Realtime feature adjustment: • Enable/Disable input source(s) • Enable/Disable module • Rate limit adjustment • …
18. Other configurations Flash - CentralDogma • Also stored in git repo, managed through CentralDogma • Flash deployment package: • Binaries only • No file config!
19. Finally, with the help of CentralDogma
20. Keynotes • CentralDogma provides native pure Java/Go client library • And more in the future • Awesome, most of Flash modules are written in Go :) • Zero runtime overhead • Clean and deadly simple API • Reliability, Stability • Cost reduction
21. Keynotes • CentralDogma authors • They are friendly, supportive and professional • Stay in user shoes • Problems are often solved quickly • Up to date with patches and detail guidelines
22. I bet you will love it “Why don’t give CentralDogma a try”
23. Thank you for listening