CentralDogma as powerful realtime configuration

CentralDogma as powerful realtime configuration

Tran Tuan Linh (LINE Corporation, Software Engineer)
Central Dogmaは、Git、ZooKeeperおよびHTTP/2をベースにした高可用性のバージョン管理サービスをサポートする設定リポジトリです。 FlashチームがCentralDogmaを使用してシステムを作成し、大量の時系列データワークロードを処理する方法についてご紹介します。

how Flash team use CentralDogma to create a system handle huge amount time series data workload

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers

October 07, 2020
Tweet

Transcript

  1. Tran Tuan Linh - 2020/10 CentralDogma Flash’s configuration backbone

  2. About me • I am Linh, from Observability Team •

    github.com/linxGnu • My team deals with: • Traces, Logs, and Metrics
  3. Impact Measurement Alerting Trending Root cause Analysis Anomaly Detection cpu

    [host=10.58.15.27] {
 timestamp: 1600649643,
 value: 0.95 /* 95% */ }
  4. Challenges • “Millions” metrics, “Billions” data points per day •

    Resolution variety • Thus, application/server may export a metric every 60s/30s/15s • 60s -> 1 point per minute • 30s -> 2 points per minute • 15s -> 4 points per minute • 10s -> 6 points per minute • So, higher resolution (shorter interval) -> more points • But give us more detail what’s going on with servers/services/apps • Metrics Storage must be fast for realtime usage (graphing, alerting)
  5. Journey • For various reasons, in 2018, we decide to

    take a ride • Build in-house distributed metrics storage from scratch • Named “Flash” - Avengers hero
  6. Flash’s Concern • Configuration management is another challenge • Successful

    distributed software system needs effective configuration management • Planning, identifying, tracking and verifying changes in configurations • Maintaining configuration integrity
  7. Flash’s Concern • What if configuration “looks” like “git” •

    Changes by "Pull Request” • Workflow through “Review” • Applying changes by “Merge” • Tracking with “Commits” • Great team collaboration
  8. • To the rescue • An open-source HA version-controlled service

    configuration repository based on Git, ZooKeeper and HTTP/2 • Features: • Store configuration files into “Git” (such as .json, .xml, .yaml) • PR for changes, get it reviewed, and merged for applying • Realtime updates-notification • Once PR got merged, services/apps are notified, no reboot
  9. None
  10. None
  11. None
  12. Cluster Topology Flash - CentralDogma Raft Consensus Shard

  13. Shard 1 Shard 2 Shard N … Cluster Topology Raft

    Configs
  14. None
  15. Metric Input Ingestor Querier Downsampler Save/query/dump

  16. Provisioning Flash - CentralDogma • Git PR makes it all

    • Scalability: • Add/remove a shard • Add/remove a node in shard • User provisioning: authorization, query scope, etc • On-the-fly, no servers reboot • Realtime feature adjustment: • Enable/Disable input source(s) • Enable/Disable module • Rate limit adjustment • …
  17. None
  18. Other configurations Flash - CentralDogma • Also stored in git

    repo, managed through CentralDogma • Flash deployment package: • Binaries only • No file config!
  19. Finally, with the help of CentralDogma

  20. Keynotes • CentralDogma provides native pure Java/Go client library •

    And more in the future • Awesome, most of Flash modules are written in Go :) • Zero runtime overhead • Clean and deadly simple API • Reliability, Stability • Cost reduction
  21. Keynotes • CentralDogma authors • They are friendly, supportive and

    professional • Stay in user shoes • Problems are often solved quickly • Up to date with patches and detail guidelines
  22. I bet you will love it “Why don’t give CentralDogma

    a try”
  23. Thank you for listening