Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Logging in the era of microservices

Logging in the era of microservices

Most of us build software, which is distributed on multiple services, systems or datacenters. Distributed systems produce distributed log files, but actually we need a centralized API to monitor and search this log files. The stack with elasticsearch, logstash and kibana (ELK) is well accepted to solve these issues and offers good starting point for our architecture.

We demo in this session how to integrate the ELK stack to your existing service architecture to achieve the best monitoring experience of your application. We will sync different kinds of logs to elasticsearch and show some tricks how to enable service comprehensive semantics and traceability. Additionally we show how to build dashboards for different stakeholders (Sales, Marketing, Ops, Devs, …). But the best part of it: Our services will not recognize the new logging infrastructure :-)

7c6d957f56a234a73c0a321bbdd5ebee?s=128

Johannes Unterstein

May 04, 2016
Tweet

More Decks by Johannes Unterstein

Other Decks in Technology

Transcript

  1. Logging im Zeitalter von Microservices Johannes Unterstein @unterstein https://git.io/vwV6S

  2. JavaForumNord.de

  3. Agenda • Motivation • The elastic stack • Cloud solutions

    • DEMO! • Dashboards • Conclusion
  4. Motivation • „WTF microservices?“ a.k.a. „vi is not enough“ •

    Logging is important and even got harder • Satisfy information needs of different stakeholders • Not only logging, also monitoring and alerting
  5. Motivation Shop UI db

  6. Motivation BasketService ArticleService CheckoutService UI db db db

  7. Motivation Proxy LoadBalancer UI BasketService ArticleService CheckoutService db db db

  8. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer UI BasketService ArticleService

    CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db
  9. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  10. Motivation BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService

    BasketService ArticleService CheckoutService db db db UI
  11. Motivation BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService

    BasketService ArticleService CheckoutService db db db UI
  12. Motivation Hystrix / Eureka

  13. Motivation WTF?

  14. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  15. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  16. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  17. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  18. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  19. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  20. The elastic stack • • • • • (Watcher)

  21. The elastic stack • Elasticsearch • Distributed database • Search

    engine based on Lucene • Scaleable • Replicated • REST & JSON
  22. The elastic stack

  23. The elastic stack • Logstash • Sync logs from host

    to elasticsearch • Own operation system process • Unify logs with grok patterns • https://github.com/logstash-plugins/logstash- patterns-core/tree/master/patterns • http://grokdebug.herokuapp.com/
  24. The elastic stack • Kibana

  25. The elastic stack • Beats • Analyzes protocols HTTP, SQL,

    … • Analyzes own and third party services • Syncs to Elasticsearch
  26. The elastic stack • Watcher • Premium plugin • Alerting

    based on Elasticsearch queries • „More than 10 errors in 1 minute -> SMS all Ops“ • „More than $X in sales -> Mail sales manager“
  27. The elastic stack • ELK introduction webinar by Jordan Sissel

    • https://www.elastic.co/webinars/introduction-elk-stack • „Awesome logging infrastructure“ talk by Alex Reelsen • https://speakerdeck.com/elastic/awesome-logging- infrastructure-using-the-elastic-stack • X-Pack overview • https://www.elastic.co/de/blog/heya-elastic-stack-and-x- pack
  28. Cloud solutions

  29. DEMO! Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  30. DEMO!! Proxy LoadBalancer BasketService ArticleService CheckoutService UI Machine 2 Machine

    3 Machine 4 Machine 1
  31. DEMO!!! UI + Proxy BasketService ArticleService CheckoutService https://git.io/vwV6S

  32. Dashboards • Different stakeholders, different questions • Different views on

    same data • „… just another Kibana configuration!“ • Import and export with elasticdump
  33. Dashboards • Sales • How many sales this week? •

    How many conversions by form A, B, C ? • How many traffic from channel X, Y, Z?
  34. Dashboards

  35. Dashboards • Marketing • How successful was campaign A, B,

    C? • Was mailing X, Y, Z accepted? • Was button relocate/restyling successful?
  36. Dashboards • Dev • F*@k! Something crashed - what happened?

    • Was my performance enhancement successful? • Did someone hit my new button? :-)
  37. Dashboards • Ops • Why is service A, B, C

    slow? • How healthy is my system? • When do i have traffic peaks?
  38. Conclusion • Just a quick demo - imagine the possibilities!

    • Services do not know about ELK • Logging is awesome (again) and you will love it! • Enjoy your free drinks ;-)