Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Logging in the era of microservices

Logging in the era of microservices

Most of us build software, which is distributed on multiple services, systems or datacenters. Distributed systems produce distributed log files, but actually we need a centralized API to monitor and search this log files. The stack with elasticsearch, logstash and kibana (ELK) is well accepted to solve these issues and offers good starting point for our architecture.

We demo in this session how to integrate the ELK stack to your existing service architecture to achieve the best monitoring experience of your application. We will sync different kinds of logs to elasticsearch and show some tricks how to enable service comprehensive semantics and traceability. Additionally we show how to build dashboards for different stakeholders (Sales, Marketing, Ops, Devs, …). But the best part of it: Our services will not recognize the new logging infrastructure :-)

Johannes Unterstein

May 04, 2016
Tweet

More Decks by Johannes Unterstein

Other Decks in Technology

Transcript

  1. Agenda • Motivation • The elastic stack • Cloud solutions

    • DEMO! • Dashboards • Conclusion
  2. Motivation • „WTF microservices?“ a.k.a. „vi is not enough“ •

    Logging is important and even got harder • Satisfy information needs of different stakeholders • Not only logging, also monitoring and alerting
  3. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer UI BasketService ArticleService

    CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db
  4. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  5. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  6. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  7. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  8. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  9. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  10. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  11. The elastic stack • Elasticsearch • Distributed database • Search

    engine based on Lucene • Scaleable • Replicated • REST & JSON
  12. The elastic stack • Logstash • Sync logs from host

    to elasticsearch • Own operation system process • Unify logs with grok patterns • https://github.com/logstash-plugins/logstash- patterns-core/tree/master/patterns • http://grokdebug.herokuapp.com/
  13. The elastic stack • Beats • Analyzes protocols HTTP, SQL,

    … • Analyzes own and third party services • Syncs to Elasticsearch
  14. The elastic stack • Watcher • Premium plugin • Alerting

    based on Elasticsearch queries • „More than 10 errors in 1 minute -> SMS all Ops“ • „More than $X in sales -> Mail sales manager“
  15. The elastic stack • ELK introduction webinar by Jordan Sissel

    • https://www.elastic.co/webinars/introduction-elk-stack • „Awesome logging infrastructure“ talk by Alex Reelsen • https://speakerdeck.com/elastic/awesome-logging- infrastructure-using-the-elastic-stack • X-Pack overview • https://www.elastic.co/de/blog/heya-elastic-stack-and-x- pack
  16. DEMO! Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  17. Dashboards • Different stakeholders, different questions • Different views on

    same data • „… just another Kibana configuration!“ • Import and export with elasticdump
  18. Dashboards • Sales • How many sales this week? •

    How many conversions by form A, B, C ? • How many traffic from channel X, Y, Z?
  19. Dashboards • Marketing • How successful was campaign A, B,

    C? • Was mailing X, Y, Z accepted? • Was button relocate/restyling successful?
  20. Dashboards • Dev • F*@k! Something crashed - what happened?

    • Was my performance enhancement successful? • Did someone hit my new button? :-)
  21. Dashboards • Ops • Why is service A, B, C

    slow? • How healthy is my system? • When do i have traffic peaks?
  22. Conclusion • Just a quick demo - imagine the possibilities!

    • Services do not know about ELK • Logging is awesome (again) and you will love it! • Enjoy your free drinks ;-)