Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
JCConf 2020 Observing in Microservices
Search
Shihyu Ho
November 18, 2020
Programming
0
770
JCConf 2020 Observing in Microservices
Lessons learned observing in Microservices
Shihyu Ho
November 18, 2020
Tweet
Share
More Decks by Shihyu Ho
See All by Shihyu Ho
JCConf 2024 Jib
shihyuho
0
69
JCConf 2022 - Using ArchUnit to test your architecture
shihyuho
0
140
hybrid-cloud-seminar.pdf
shihyuho
0
100
JCConf 2021 Access Kubernetes API in Java
shihyuho
0
120
iThome Kubernetes Summit 2018
shihyuho
0
68
JCConf 2016 zookeeper
shihyuho
0
62
Other Decks in Programming
See All in Programming
More Approvers for Greater OSS and Japan Community
tkikuc
1
110
機能追加とリーダー業務の類似性
rinchoku
0
180
AIレビュアーをスケールさせるには / Scaling AI Reviewers
technuma
2
240
TROCCO×dbtで実現する人にもAIにもやさしいデータ基盤
nealle
0
390
CSC305 Summer Lecture 12
javiergs
PRO
0
130
テストカバレッジ100%を10年続けて得られた学びと品質
mottyzzz
2
420
Improving my own Ruby thereafter
sisshiki1969
1
140
Rancher と Terraform
fufuhu
2
170
個人軟體時代
ethanhuang13
0
280
Vue・React マルチプロダクト開発を支える Vite
andpad
0
110
複雑なドメインに挑む.pdf
yukisakai1225
4
880
パッケージ設計の黒魔術/Kyoto.go#63
lufia
3
390
Featured
See All Featured
A designer walks into a library…
pauljervisheath
207
24k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
910
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
8
510
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.4k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
790
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.9k
Why Our Code Smells
bkeepers
PRO
339
57k
The Art of Programming - Codeland 2020
erikaheidi
55
13k
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.5k
Transcript
Lessons learned observing in Microservices Matt Ho
SoftLeader RD Matt Ho https://github.com/shihyuho
Microservices!
Move from Spring Cloud to Kubernetes native!
好多 App, 平常沒事, But...
Infrastructure 0
Tracing App App App container platform
Monitoring App App App container platform cpu/memory/VM heap … cpu/memory/VM
heap …
Loggin g App App App container platform
Kubernetes on-premises
原⽣的, ⼀裝好什麼都沒有!
不能客製呦!
不是在做比較
Tracing 1
Using OpenTracing #
github.com/opentracing-contrib/ java-spring-cloud
Put tracing in response #
App filter response request api add trace info
Add tracing in logger MDC #
App A log MDC
None
None
None
None
None
Create your Spans #
time Span A Span B Span C Span D 外部呼叫
trace
None
Propagating messages #
None
App A App B App C App D Queue Baggage
App A App B App C App D REST gRPC
Queue Enqueue Dequeue Baggage
None
Monitoring 2
Prometheus & exporters #
- node-exporter - Hardware/OS metric s - cAdvisor - Container
metric s - kube-state-metrics - Kubernetes Object metrics 環境類 exporters
github.com/prometheus-operator/ kube-prometheus
None
None
None
- blackbox - Probing over HTTP, HTTPS, DNS … -
micrometer.io - Java app metrics 服務類 exporters
None
None
https://grafana.com/grafana/ dashboards
Logging 3
Define logging format #
JSON logging format
github.com/logstash/ logstash-logback-encoder
Append app information #
app_name, app_version…
None
None
ES index strategy #
By namespace and date
None
Forwarding logs from { … } #
Fluentd Forward Log format OCP DockerSwarm Forward Other Cluster
App App App App App App
好多 App, 平常沒事, But...
Thanks