Logging in the era of microservices

Most of us build software, which is distributed on multiple services, systems or datacenters. Distributed systems produce distributed log files, but actually we need a centralized API to monitor and search this log files. The stack with elasticsearch, logstash and kibana (ELK) is well accepted to solve these issues and offers good starting point for our architecture.

We demo in this session how to integrate the ELK stack to your existing service architecture to achieve the best monitoring experience of your application. We will sync different kinds of logs to elasticsearch and show some tricks how to enable service comprehensive semantics and traceability. Additionally we show how to build dashboards for different stakeholders (Sales, Marketing, Ops, Devs, …). But the best part of it: Our services will not recognize the new logging infrastructure :-)

Transcript

1. Logging im Zeitalter von Microservices Johannes Unterstein @unterstein https://git.io/vwV6S
2. JavaForumNord.de
3. Agenda • Motivation • The elastic stack • Cloud solutions • DEMO! • Dashboards • Conclusion
4. Motivation • „WTF microservices?“ a.k.a. „vi is not enough“ • Logging is important and even got harder • Satisfy information needs of different stakeholders • Not only logging, also monitoring and alerting
5. Motivation Shop UI db
6. Motivation BasketService ArticleService CheckoutService UI db db db
7. Motivation Proxy LoadBalancer UI BasketService ArticleService CheckoutService db db db
8. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer UI BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db
9. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
10. Motivation BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
11. Motivation BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
12. Motivation Hystrix / Eureka
13. Motivation WTF?
14. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
15. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
16. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
17. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
18. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
19. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
20. The elastic stack • • • • • (Watcher)
21. The elastic stack • Elasticsearch • Distributed database • Search engine based on Lucene • Scaleable • Replicated • REST & JSON
22. The elastic stack
23. The elastic stack • Logstash • Sync logs from host to elasticsearch • Own operation system process • Unify logs with grok patterns • https://github.com/logstash-plugins/logstash- patterns-core/tree/master/patterns • http://grokdebug.herokuapp.com/
24. The elastic stack • Kibana
25. The elastic stack • Beats • Analyzes protocols HTTP, SQL, … • Analyzes own and third party services • Syncs to Elasticsearch
26. The elastic stack • Watcher • Premium plugin • Alerting based on Elasticsearch queries • „More than 10 errors in 1 minute -> SMS all Ops“ • „More than $X in sales -> Mail sales manager“
27. The elastic stack • ELK introduction webinar by Jordan Sissel • https://www.elastic.co/webinars/introduction-elk-stack • „Awesome logging infrastructure“ talk by Alex Reelsen • https://speakerdeck.com/elastic/awesome-logging- infrastructure-using-the-elastic-stack • X-Pack overview • https://www.elastic.co/de/blog/heya-elastic-stack-and-x- pack
28. Cloud solutions
29. DEMO! Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
30. DEMO!! Proxy LoadBalancer BasketService ArticleService CheckoutService UI Machine 2 Machine 3 Machine 4 Machine 1
31. DEMO!!! UI + Proxy BasketService ArticleService CheckoutService https://git.io/vwV6S
32. Dashboards • Different stakeholders, different questions • Different views on same data • „… just another Kibana configuration!“ • Import and export with elasticdump
33. Dashboards • Sales • How many sales this week? • How many conversions by form A, B, C ? • How many traffic from channel X, Y, Z?
34. Dashboards
35. Dashboards • Marketing • How successful was campaign A, B, C? • Was mailing X, Y, Z accepted? • Was button relocate/restyling successful?
36. Dashboards • Dev • F*@k! Something crashed - what happened? • Was my performance enhancement successful? • Did someone hit my new button? :-)
37. Dashboards • Ops • Why is service A, B, C slow? • How healthy is my system? • When do i have traffic peaks?
38. Conclusion • Just a quick demo - imagine the possibilities! • Services do not know about ELK • Logging is awesome (again) and you will love it! • Enjoy your free drinks ;-)