JCConf 2020 Observing in Microservices

Lessons learned observing in Microservices

Transcript

1. Lessons learned observing in Microservices Matt Ho
2. SoftLeader RD Matt Ho https://github.com/shihyuho
3. Microservices!
4. Move from Spring Cloud to Kubernetes native!
5. 好多 App, 平常沒事, But...
6. Infrastructure 0
7. Tracing App App App container platform
8. Monitoring App App App container platform cpu/memory/VM heap … cpu/memory/VM heap …
9. Loggin g App App App container platform
10. Kubernetes on-premises
11. 原⽣的, ⼀裝好什麼都沒有!
12. 不能客製呦!
13. 不是在做比較
14. Tracing 1
15. Using OpenTracing #
16. github.com/opentracing-contrib/ java-spring-cloud
17. Put tracing in response #
18. App filter response request api add trace info
19. Add tracing in logger MDC #
20. App A log MDC
26. Create your Spans #
27. time Span A Span B Span C Span D 外部呼叫 trace
29. Propagating messages #
31. App A App B App C App D Queue Baggage
32. App A App B App C App D REST gRPC Queue Enqueue Dequeue Baggage
34. Monitoring 2
35. Prometheus & exporters #
36. - node-exporter - Hardware/OS metric s - cAdvisor - Container metric s - kube-state-metrics - Kubernetes Object metrics 環境類 exporters
37. github.com/prometheus-operator/ kube-prometheus
41. - blackbox - Probing over HTTP, HTTPS, DNS … - micrometer.io - Java app metrics 服務類 exporters
44. https://grafana.com/grafana/ dashboards
45. Logging 3
46. Define logging format #
47. JSON logging format
48. github.com/logstash/ logstash-logback-encoder
49. Append app information #
50. app_name, app_version…
53. ES index strategy #
54. By namespace and date
56. Forwarding logs from { … } #
57. Fluentd Forward Log format OCP DockerSwarm Forward Other Cluster
58. App App App App App App
59. 好多 App, 平常沒事, But...
60. Thanks