The API Side of Monitoring

"With microservices every outage is like a murder mystery" is a common complaint. But it doesn’t have to be! This talk gives an overview on how to monitor distributed applications and how common monitoring APIs can help with this effort. We dive into:

* Logs: Why structured logs should be the new norm of logging and how to centralize them.
* Metrics: This includes both application metrics (using REST APIs or JMX for example) as well as system metrics, which are more similar to the classical ‘top’ output.
* Traces: The Open Tracing Foundation is basically the golden standard of tracing at the moment. We use Sleuth to show how to trace requests through a distributed system and Zipkin to display how long each call takes.

This combination of logs, metrics, and traces is also called the holy trinity of monitoring.

Transcript

1. Logs, Metrics, Traces The API Side of Monitoring Philipp Krenn̴̴̴̴̴@xeraa
7. Infrastructure | Developer Advocate
8. Disclaimer I build highly monitored Hello World apps
10. http://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html
11. Logs
12. 2017-10-10 09:34:16.895 INFO [frontend,cf4dc621b927b96e,cf4dc621b927b96e,true] 25308 --- [nio-8080-exec-8] net.xeraa.frontend.FrontendController: X-Span-Export=true, X-B3-SpanId=cf4dc621b927b96e, X-B3-TraceId=cf4dc621b927b96e, name=Peter Calling something good
13. Nobody wants to parse
14. { "X-Span-Export":"true", "X-B3-SpanId":"cf4dc621b927b96e", "name":"Peter", "X-B3-TraceId":"cf4dc621b927b96e", "@timestamp":"2017-10-10T09:34:16.895+00:00", "severity":"INFO", "service":"frontend", "trace":"cf4dc621b927b96e", "span":"cf4dc621b927b96e", "parent":"", "exportable":"true", "pid":"25308", "thread":"http-nio-8080-exec-8", "class":"net.xeraa.frontend.FrontendController", "mdc":"X-Span-Export=true, X-B3-SpanId=cf4dc621b927b96e, name=Peter, X-B3-TraceId=cf4dc621b927b96e", "rest":"Calling something good" }
15. Metrics
17. HTTP
18. JMX
19. Traces
20. 2017-03-29 08:08:35.090 INFO [frontend,69a851ed700bf1fa,69a851ed700bf1fa,true] 23575 --- [p-nio-80-exec-2] n.x.MicroserviceMonitoringApplication : Calling home 2017-03-29 08:08:36.149 INFO [frontend,69a851ed700bf1fa,10b1f0a0e2de1253,true] 23575 --- [p-nio-80-exec-3] n.x.MicroserviceMonitoringApplication : You called home
21. Application: frontend Trace ID: 69a851ed700bf1fa Span ID: 10b1f0a0e2de1253 Send to Zipkin: true
22. Zipkin Chrome Extension https://github.com/openzipkin/zipkin-browser-extension
23. Conclusion
25. Logs Metrics Traces
26. More Uptime, network, system & application metrics, security, auditing,...
27. Your Turn https://kibana.xeraa.wtf https://github.com/xeraa/microservice-monitoring
28. Questions? Philipp Krenn̴̴̴̴̴@xeraa PS: Stickers