Slide 1
Slide 1 text
Lessons learned observing
in Microservices
Matt Ho
Slide 2
Slide 2 text
SoftLeader RD
Matt Ho
https://github.com/shihyuho
Slide 3
Slide 3 text
Microservices!
Slide 4
Slide 4 text
Move from Spring Cloud
to Kubernetes native!
Slide 5
Slide 5 text
好多 App, 平常沒事, But...
Slide 6
Slide 6 text
Infrastructure
0
Slide 7
Slide 7 text
Tracing
App
App
App
container platform
Slide 8
Slide 8 text
Monitoring
App
App
App
container platform
cpu/memory/VM heap …
cpu/memory/VM heap …
Slide 9
Slide 9 text
Loggin
g
App
App
App
container platform
Slide 10
Slide 10 text
Kubernetes on-premises
Slide 11
Slide 11 text
原⽣的, ⼀裝好什麼都沒有!
Slide 12
Slide 12 text
不能客製呦!
Slide 13
Slide 13 text
不是在做比較
Slide 14
Slide 14 text
Tracing
1
Slide 15
Slide 15 text
Using OpenTracing
#
Slide 16
Slide 16 text
github.com/opentracing-contrib/
java-spring-cloud
Slide 17
Slide 17 text
Put tracing in response
#
Slide 18
Slide 18 text
App
filter
response
request
api
add trace info
Slide 19
Slide 19 text
Add tracing in logger MDC
#
Slide 20
Slide 20 text
App A
log
MDC
Slide 21
Slide 21 text
No content
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
No content
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
Create your Spans
#
Slide 27
Slide 27 text
time
Span A
Span B Span C
Span D
外部呼叫
trace
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
Propagating messages
#
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
App A App B
App C
App D
Queue
Baggage
Slide 32
Slide 32 text
App A App B
App C
App D
REST
gRPC
Queue
Enqueue
Dequeue
Baggage
Slide 33
Slide 33 text
No content
Slide 34
Slide 34 text
Monitoring
2
Slide 35
Slide 35 text
Prometheus & exporters
#
Slide 36
Slide 36 text
- node-exporter - Hardware/OS metric
s
- cAdvisor - Container metric
s
- kube-state-metrics - Kubernetes Object metrics
環境類 exporters
Slide 37
Slide 37 text
github.com/prometheus-operator/
kube-prometheus
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
- blackbox - Probing over HTTP, HTTPS, DNS
…
- micrometer.io - Java app metrics
服務類 exporters
Slide 42
Slide 42 text
No content
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
https://grafana.com/grafana/
dashboards
Slide 45
Slide 45 text
Logging
3
Slide 46
Slide 46 text
Define logging format
#
Slide 47
Slide 47 text
JSON logging format
Slide 48
Slide 48 text
github.com/logstash/
logstash-logback-encoder
Slide 49
Slide 49 text
Append app information
#
Slide 50
Slide 50 text
app_name, app_version…
Slide 51
Slide 51 text
No content
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
ES index strategy
#
Slide 54
Slide 54 text
By namespace and date
Slide 55
Slide 55 text
No content
Slide 56
Slide 56 text
Forwarding logs from { … }
#
Slide 57
Slide 57 text
Fluentd
Forward
Log format
OCP
DockerSwarm
Forward
Other Cluster
Slide 58
Slide 58 text
App
App
App
App
App
App
Slide 59
Slide 59 text
好多 App, 平常沒事, But...
Slide 60
Slide 60 text
Thanks