Lessons learned observing in Microservices
Lessons learned observingin MicroservicesMatt Ho
View Slide
SoftLeader RDMatt Hohttps://github.com/shihyuho
Microservices!
Move from Spring Cloudto Kubernetes native!
好多 App, 平常沒事, But...
Infrastructure0
TracingAppAppAppcontainer platform
MonitoringAppAppAppcontainer platformcpu/memory/VM heap …cpu/memory/VM heap …
LoggingAppAppAppcontainer platform
Kubernetes on-premises
原⽣的, ⼀裝好什麼都沒有!
不能客製呦!
不是在做比較
Tracing1
Using OpenTracing#
github.com/opentracing-contrib/java-spring-cloud
Put tracing in response#
Appfilterresponserequestapiadd trace info
Add tracing in logger MDC#
App AlogMDC
Create your Spans#
timeSpan ASpan B Span CSpan D外部呼叫trace
Propagating messages#
App A App BApp CApp DQueueBaggage
App A App BApp CApp DRESTgRPCQueueEnqueueDequeueBaggage
Monitoring2
Prometheus & exporters#
- node-exporter - Hardware/OS metrics- cAdvisor - Container metrics- kube-state-metrics - Kubernetes Object metrics環境類 exporters
github.com/prometheus-operator/kube-prometheus
- blackbox - Probing over HTTP, HTTPS, DNS…- micrometer.io - Java app metrics服務類 exporters
https://grafana.com/grafana/dashboards
Logging3
Define logging format#
JSON logging format
github.com/logstash/logstash-logback-encoder
Append app information#
app_name, app_version…
ES index strategy#
By namespace and date
Forwarding logs from { … }#
FluentdForwardLog formatOCPDockerSwarmForwardOther Cluster
AppAppAppAppAppApp
Thanks