Logging in the era of microservices - wjax 2016

Most of us build software, which is distributed on multiple services, systems or datacenters. Distributed systems produce distributed log files, but actually we need a centralized API to monitor and search this log files. The stack with elasticsearch, logstash and kibana (ELK) is well accepted to solve these issues and offers good starting point for our architecture.

We demo in this session how to integrate the ELK stack to your existing service architecture to achieve the best monitoring experience of your application. We will sync different kinds of logs to elasticsearch and show some tricks how to enable service comprehensive semantics and traceability. Additionally we show how to build dashboards for different stakeholders (Sales, Marketing, Ops, Devs, …). But the best part of it: Our services will not recognize the new logging infrastructure :-)

Transcript

1. Logging in the era of Microservices Johannes Unterstein Software Engineer @mesosphere @unterstein
2. Agenda • Motivation • The elastic stack • Cloud solutions • DEMO! • Dashboards • Conclusion
3. Motivation • „WTF microservices?“ a.k.a. „vi is not enough“ • Logging is important and even got harder • Satisfy information needs of different stakeholders • Not only logging, also monitoring and alerting
4. Motivation Shop UI db
5. Motivation BasketService ArticleService CheckoutService UI db db db
6. Motivation Proxy LoadBalancer UI BasketService ArticleService CheckoutService db db db
7. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer UI BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db
8. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
9. Motivation BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
10. Motivation BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
11. Motivation Hystrix / Eureka
12. Motivation WTF?
13. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
14. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
15. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
16. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
17. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
18. Motivation Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
19. The elastic stack • • • • • (Watcher)
20. The elastic stack • Elasticsearch • Distributed database • Search engine based on Lucene • Scaleable • Replicated • REST & JSON
21. The elastic stack
22. The elastic stack • Logstash • Sync logs from host to elasticsearch • Own operation system process • Unify logs with grok patterns • https://github.com/logstash-plugins/logstash- patterns-core/tree/master/patterns • http://grokdebug.herokuapp.com/
23. The elastic stack • Kibana
24. The elastic stack • Beats • Analyzes protocols HTTP, SQL, Files, … • Analyzes own and third party services • Syncs to Elasticsearch • Filebeat, Metricbeat, Dockbeat, SpringBeat,…
25. The elastic stack • Watcher • Premium plugin • Alerting based on Elasticsearch queries • „More than 10 errors in 1 minute -> SMS all Ops“ • „More than $X in sales -> Mail sales manager“
26. Cloud solutions
27. DEMO! Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer Proxy LoadBalancer BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService BasketService ArticleService CheckoutService db db db UI
28. DEMO!! Proxy LoadBalancer BasketService ArticleService CheckoutService UI Machine 2 Machine 3 Machine 4 Machine 1
29. DEMO!!! UI + Proxy BasketService ArticleService CheckoutService https://git.io/vwV6S
30. Logging 101 • Create RequestId/SessionId in first load balancer and pass to following systems • Enrich logging statements with markers • Add currently logging app, node, instanceId, … • Long static part of log message • Use different log messages
31. Dashboards • Different stakeholders, different questions • Different views on same data • „… just another Kibana configuration!“ • Import and export with elasticdump
32. Dashboards • Sales • How many sales this week? • How many conversions by form A, B, C ? • How many traffic from channel X, Y, Z?
33. Dashboards
34. Dashboards • Marketing • How successful was campaign A, B, C? • Was mailing X, Y, Z accepted? • Was button relocate/restyling successful?
35. Dashboards • Dev • F*@k! Something crashed - what happened? • Was my performance enhancement successful? • Did someone hit my new button? :-)
36. Dashboards • Ops • Why is service A, B, C slow? • How healthy is my system? • Do I run out of resources? • When do I have traffic peaks?
37. Conclusion • Just a quick demo - imagine the possibilities! • Services do not know about ELK • Logging is awesome (again) and you will love it! • Enjoy the conference :-) https://git.io/vwV6S
38. More about x-pack • ELK introduction webinar by Jordan Sissel • https://www.elastic.co/webinars/introduction-elk-stack • „Awesome logging infrastructure“ talk by Alex Reelsen • https://speakerdeck.com/elastic/awesome-logging- infrastructure-using-the-elastic-stack • X-Pack overview • https://www.elastic.co/de/blog/heya-elastic-stack-and-x- pack