Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Logging in the era of microservices - wjax 2016

Logging in the era of microservices - wjax 2016

Most of us build software, which is distributed on multiple services, systems or datacenters. Distributed systems produce distributed log files, but actually we need a centralized API to monitor and search this log files. The stack with elasticsearch, logstash and kibana (ELK) is well accepted to solve these issues and offers good starting point for our architecture.

We demo in this session how to integrate the ELK stack to your existing service architecture to achieve the best monitoring experience of your application. We will sync different kinds of logs to elasticsearch and show some tricks how to enable service comprehensive semantics and traceability. Additionally we show how to build dashboards for different stakeholders (Sales, Marketing, Ops, Devs, …). But the best part of it: Our services will not recognize the new logging infrastructure :-)

7c6d957f56a234a73c0a321bbdd5ebee?s=128

Johannes Unterstein

November 08, 2016
Tweet

More Decks by Johannes Unterstein

Other Decks in Technology

Transcript

  1. Logging in the era of Microservices Johannes Unterstein Software Engineer

    @mesosphere @unterstein
  2. Agenda • Motivation • The elastic stack • Cloud solutions

    • DEMO! • Dashboards • Conclusion
  3. Motivation • „WTF microservices?“ a.k.a. „vi is not enough“ •

    Logging is important and even got harder • Satisfy information needs of different stakeholders • Not only logging, also monitoring and alerting
  4. Motivation Shop UI db

  5. Motivation BasketService ArticleService CheckoutService UI db db db

  6. Motivation Proxy LoadBalancer UI BasketService ArticleService CheckoutService db db db

  7. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer UI BasketService ArticleService

    CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db
  8. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  9. Motivation BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService

    BasketService ArticleService CheckoutService db db db UI
  10. Motivation BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService

    BasketService ArticleService CheckoutService db db db UI
  11. Motivation Hystrix / Eureka

  12. Motivation WTF?

  13. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  14. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  15. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  16. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  17. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  18. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  19. The elastic stack • • • • • (Watcher)

  20. The elastic stack • Elasticsearch • Distributed database • Search

    engine based on Lucene • Scaleable • Replicated • REST & JSON
  21. The elastic stack

  22. The elastic stack • Logstash • Sync logs from host

    to elasticsearch • Own operation system process • Unify logs with grok patterns • https://github.com/logstash-plugins/logstash- patterns-core/tree/master/patterns • http://grokdebug.herokuapp.com/
  23. The elastic stack • Kibana

  24. The elastic stack • Beats • Analyzes protocols HTTP, SQL,

    Files, … • Analyzes own and third party services • Syncs to Elasticsearch • Filebeat, Metricbeat, Dockbeat, SpringBeat,…
  25. The elastic stack • Watcher • Premium plugin • Alerting

    based on Elasticsearch queries • „More than 10 errors in 1 minute -> SMS all Ops“ • „More than $X in sales -> Mail sales manager“
  26. Cloud solutions

  27. DEMO! Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService

    ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
  28. DEMO!! Proxy LoadBalancer BasketService ArticleService CheckoutService UI Machine 2 Machine

    3 Machine 4 Machine 1
  29. DEMO!!! UI + Proxy BasketService ArticleService CheckoutService https://git.io/vwV6S

  30. Logging 101 • Create RequestId/SessionId in first load balancer and

    pass to following systems • Enrich logging statements with markers • Add currently logging app, node, instanceId, … • Long static part of log message • Use different log messages
  31. Dashboards • Different stakeholders, different questions • Different views on

    same data • „… just another Kibana configuration!“ • Import and export with elasticdump
  32. Dashboards • Sales • How many sales this week? •

    How many conversions by form A, B, C ? • How many traffic from channel X, Y, Z?
  33. Dashboards

  34. Dashboards • Marketing • How successful was campaign A, B,

    C? • Was mailing X, Y, Z accepted? • Was button relocate/restyling successful?
  35. Dashboards • Dev • F*@k! Something crashed - what happened?

    • Was my performance enhancement successful? • Did someone hit my new button? :-)
  36. Dashboards • Ops • Why is service A, B, C

    slow? • How healthy is my system? • Do I run out of resources? • When do I have traffic peaks?
  37. Conclusion • Just a quick demo - imagine the possibilities!

    • Services do not know about ELK • Logging is awesome (again) and you will love it! • Enjoy the conference :-) https://git.io/vwV6S
  38. More about x-pack • ELK introduction webinar by Jordan Sissel

    • https://www.elastic.co/webinars/introduction-elk-stack • „Awesome logging infrastructure“ talk by Alex Reelsen • https://speakerdeck.com/elastic/awesome-logging- infrastructure-using-the-elastic-stack • X-Pack overview • https://www.elastic.co/de/blog/heya-elastic-stack-and-x- pack