Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレー...
Search
Shin Tanimoto
February 28, 2023
Programming
5
2.4k
クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレースモニタリング
Java Developers Summit (2023-02-28)
Shin Tanimoto
February 28, 2023
Tweet
Share
More Decks by Shin Tanimoto
See All by Shin Tanimoto
Spring Bootで実装とインフラをこれでもかと分離するための試み
shintanimoto
7
1.2k
26 Java Years
shintanimoto
0
90
Let’s Have Fun with Reactive Programming, Using Reactor and WebFlux
shintanimoto
0
290
Monitoring and Visualizing Your (Micro)services
shintanimoto
1
380
現代に求められるJavaコミュニティとは / What should be the Java Community of Today?
shintanimoto
0
950
人生がときめく「学び」の魔法 / The Life-Changing Magic of Studying
shintanimoto
6
1.7k
業務で使いたいWebFluxによるReactiveプログラミング / Introduction to Reactive Programming using Spring WebFlux
shintanimoto
9
9.5k
from old Java to modern Java (2017) #jjug
shintanimoto
4
1.7k
Spring Cloud Sleuth + Zipkin with Elasticsearch #zipkin_jp
shintanimoto
1
1.3k
Other Decks in Programming
See All in Programming
Goで作る、開発・CI環境
sin392
0
190
WebViewの現在地 - SwiftUI時代のWebKit - / The Current State Of WebView
marcy731
0
110
AIコーディング道場勉強会#2 君(エンジニア)たちはどう生きるか
misakiotb
1
270
既存デザインを変更せずにタップ領域を広げる方法
tahia910
1
260
イベントストーミング図からコードへの変換手順 / Procedure for Converting Event Storming Diagrams to Code
nrslib
1
570
Select API from Kotlin Coroutine
jmatsu
1
210
データの民主化を支える、透明性のあるデータ利活用への挑戦 2025-06-25 Database Engineering Meetup#7
y_ken
0
340
GraphRAGの仕組みまるわかり
tosuri13
8
510
関数型まつりレポート for JuliaTokai #22
antimon2
0
160
PipeCDのプラグイン化で目指すところ
warashi
1
230
ruby.wasmで多人数リアルタイム通信ゲームを作ろう
lnit
2
330
生成AIコーディングとの向き合い方、AIと共創するという考え方 / How to deal with generative AI coding and the concept of co-creating with AI
seike460
PRO
1
350
Featured
See All Featured
Speed Design
sergeychernyshev
32
1k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Raft: Consensus for Rubyists
vanstee
140
7k
Building an army of robots
kneath
306
45k
Agile that works and the tools we love
rasmusluckow
329
21k
KATA
mclloyd
30
14k
4 Signs Your Business is Dying
shpigford
184
22k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.8k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.5k
Code Review Best Practice
trishagee
69
18k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Transcript
ΫϥυωΠςΟϒ࣌ͷ ίϯςφڥʹ͓͚ΔJavaΞϓϦέʔγϣϯͷ ϝτϦΫεɾϩάɾτϨʔεϞχλϦϯά Everforth / Acroquest Technology / JJUG Shin
Tanimoto (@cero_t)
ࣗݾհ • ୩ຊ ৺ (Shin Tanimoto) • 2ࣾͰಇ͘ΤϯδχΞ • גࣜձࣾEverforthʢCTOʣ
• Acroquest Technologyגࣜձࣾ • ຊJavaϢʔβʔάϧʔϓʢදʣ • Twitter : @cero_t
ࠓͷϝΠϯςʔϚ
ྩʹ͓͚ΔΦϒβόϦςΟ ͷΞʔΩςΫνϟʹ͍ͭͯ
ϞχλϦϯάͷ3େཁૉ • ϝτϦΫε • ϩά • τϨʔε
ϞχλϦϯάͷ3େཁૉ • ϝτϦΫε : ࣌ܥྻͷใ • ϩά : λΠϜελϯϓ͖ͭͷจࣈྻ •
τϨʔε : ݺͼग़͠ͷه
ϝτϦΫε • ࣌ܥྻͷใ • ྫ • OS : CPU༻ɺϝϞϦ༻ྔʢʣɺσΟεΫ༻ྔʢʣ •
JavaVM : ώʔϓ༻ྔʢʣɺεϨουɺίωΫγϣϯϓʔϧ • App : ϦΫΤετɺΤϥʔʢʣ
ϝτϦΫε • αϯϓϦϯάϕʔε • ͦͷॠؒͷ͔ɺूܭͨ͠Λऔಘ͢Δ • CPU༻ : ʮඵؒʯͷ༻ͷฏۉԿ%͔ •
ϝϞϦ༻ྔ : ʮ͍·ʯԿMB͍ͬͯΔ͔ • GCճ : ʮ͍··ͰʯʹԿճGCΛߦ͔ͬͨ • αϯϓϦϯάִ͕ؒখ͍͞΄Ͳਫ਼্͕͕Δ͕ɺσʔλྔ͕૿͑Δ
ϝτϦΫε • ฏͷϞχλϦϯά • OSϨϕϧͷϝτϦΫε͚ͩऔΓ͕ͪ • JavaϨϕϧͷϝτϦΫεΛऔ͍ͬͯΕେͨ͠ͷ • ΞϓϦέʔγϣϯϨϕϧͷϝτϦΫεΛऔΕΔΑ͏ʹ͢Δͷ
૬ͳΓ͜Έ • ྩͰͯ͢औΔ͠
ϩά • ࣌ࠁΛ͏จࣈྻ • ྫ • ΞϓϦέʔγϣϯϩάɺOSͷϩάɺGCϩάɺ ϑϨʔϜϫʔΫϛυϧΣΞͷϩά •
ॲཧΛߦͬͨɺΤϥʔ͕ى͖ͨɺԿΒ͔ͷෆ߹ͳ͜ͱ͕ى͖ͨ • ͦͷ࣌ͷίϯςΩετใʢมఆͳͲʣ
ϩά • Πϕϯτϕʔε • ࣄ͕ى͖ͨ࣌ʹه͢Δ • αϯϓϦϯάϕʔεͱҟͳΓɺͯ͢ه͞ΕΔ • จࣈྻͳͷͰϝτϦΫεʹൺͯαΠζ͕େ͖͍ •
1ͭͷϦΫΤετʹΑͬͯۀଆͷσʔλετΞʹॻ͖ࠐ·ΕΔใྔΑΓ ϩάͷํ͕େ͖͘ͳΔ͜ͱ͕͋Δ • ΞϓϦέʔγϣϯͷσʔλετΞΑΓϩάͷσʔλετΞͷํ͕ ߴੑೳɾେ༰ྔͳͷΛٻΊΒΕΔ͜ͱ͕͋Δ
ϩά • ฏͷϞχλϦϯά • ͝ͱʹzipͯ͠ωοτϫʔΫετϨʔδʹஔ͖͕ͪ • ඞཁͳ࣌ʹμϯϩʔυͯ͠։͔͘ɺviͰݟʹߦ͖͕ͪ • ʮERRORʯͱ͍͏ϝοηʔδ͕͋Εύτϥϯϓ͕ճΓ͕ͪ •
ྩͷϞχλϦϯάͰৗʹݕࡧͰ͖Δঢ়ଶʹ͢Δ͠ • ͕ͦͯ͠ݟ͔ͭͬͨΒSlackͳͲʹ௨͢Δ͠
τϨʔε • ݺͼग़͠ͷه • ෳͷΞϓϦέʔγϣϯϓϩηεΛލ͍ͩελοΫτϨʔεͷΑ͏ͳͷ • ͍ͭɺ୭͕୭ΛݺΜͰ͔ɺͲΕ͘Βֻ͍͔͔ͬͨɺޭ͔ࣦͨ͠ഊ͔ͨ͠ • ΠϕϯτϕʔεͰऔಘ •
ͯ͢औಘ͢Δ͔ɺҰఆͷׂ߹Ͱऔಘ͢Δ͔ઃఆՄೳ
τϨʔε • ελοΫτϨʔε • Javaϓϩηε͚ͩͰ؆ܿ͢Δݺͼग़͠֊ • ओʹྫ֎ൃੜ࣌ʹऔಘ͢Δ • ͦΕͳΒෳΞϓϦέʔγϣϯΛލ͍ͩτϨʔεऔΓ͍ͨΑͶ •
τϨʔεɺൃੜ࣌ͷ֬ೝɺϘτϧωοΫݕग़ʹར༻Ͱ͖Δ
τϨʔε • ฏͷϞχλϦϯά • τϨʔεऔΒͳ͍ or औΕͳ͍ • ྩͰऔΓ·͠ΐ͏
Spring Boot 3.0࣌ͷ ϞχλϦϯά
σϞ
σϞͰͬͨιʔείʔυ • https://github.com/cero-t/spring-store-2022 • ࣮ߦखॱઃఆͷৄࡉͳ༰ϒϩάʹهࡌ • ຊεϥΠυͷ࠷ऴϖʔδʹ·ͱΊ·͢
ΞϓϦέʔγϣϯͷߏ BFF Payment Service Order Service Stock
Service Delivery Service Item Service Cart Service Queue ECαΠτͷόοΫΤϯυΛΠϝʔδ
ར༻ٕͨ͠ज़ελοΫ • ϚΠΫϩαʔϏεΞϓϦέʔγϣϯ • Java 17 • Spring Boot 3.0
(without Spring Cloud) • Spring Boot Web • Micrometer • OpenTelemetry
ར༻ٕͨ͠ज़ελοΫ • GrafanaελοΫ • Grafana : ϞχλϦϯάͷUI • Prometheus :
ϝτϦΫεͷऩूͱσʔλετΞ • Loki : ϩάͷσʔλετΞ • Tempo : τϨʔεͷσʔλετΞ
ϞχλϦϯάͷߏ Java Process Grafana Tempo Prometheus Loki औಘͯ͠ՄࢹԽ ϝτϦΫε
ϩά τϨʔε Java Process Java Process Spring Boot Application
Spring Boot + PrometheusͰ ϝτϦΫεऩू
Spring BootͰϝτϦΫεऩू • Spring BootΞϓϦέʔγϣϯͷϝτϦΫεใΛWeb APIͱͯ͠ ެ։ͯ͠ɺPrometheus͔Βఆظతʹऔಘͯ͠ऩूɾอଘ͢Δ • Grafana͕Prometheus͔ΒϝτϦΫεΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application ϝτϦΫε औಘAPI Prometheus ఆظతʹऔಘ Grafana औಘͯ͠ՄࢹԽ
Spring BootͰϝτϦΫεऩू • ϝτϦΫεऔಘͷWeb APIΛެ։͢Δํ๏ • Spring BootΞϓϦέʔγϣϯͷdependencyʹ ActuatorͱMicrometerΛՃ͢Δ
• application.propertiesʹͯActuatorͷΤϯυϙΠϯτΛެ։͢Δ
ActuatorͱMicrometerΛՃ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛՃ • Actuator : spring-boot-starter-actuator • Spring
BootͷઃఆϝτϦΫεΛWeb APIͱͯ͠ެ։͢ΔͨΊͷϥΠϒϥϦ • Micrometer : micrometer-registry-prometheus • OSJavaVMɺSpring BootͷϝτϦΫεใΛऔಘ͢ΔͨΊͷϥΠϒϥϦ • “-prometheus” Prometheus͔ΒΞΫηε͢ΔͨΊͷΤϯυϙΠϯτΛ ActuatorΛ༻͍ͯެ։͢Δɻ • “-datadog” ͱ͍͏ϥΠϒϥϦఆظతʹDataDogʹϝτϦΫεΛૹ৴͢ΔͳͲ ϥΠϒϥϦʹΑͬͯڍಈ͕ҟͳΔ
ActuatorͱMicrometerΛՃ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛՃ <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> <scope>runtime</scope> </dependency> <dependency>
<groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> <scope>runtime</scope> </dependency>
ActuatorͷΤϯυϙΠϯτΛެ։ • application.properties ʹͯɺMicrometerʹؔ͢Δ ActuatorͷΤϯυϙΠϯτΛެ։͢Δ management.endpoints.web.exposure.include=* management.metrics.distribution.percentiles-histogram.http.server.requests=true management.metrics.tags.application=${spring.application.name} #
Micrometerʹؔ͢ΔϝτϦΫε͚ͩެ։͢Δ߹ʹ࣍ͷઃఆͰྑ͍ # management.endpoints.web.exposure.include=prometheus
ActuatorͱMicrometerΛՃ • http://(ΞϓϦέʔγϣϯͷΞυϨε)/actuator/prometheus ʹ ΞΫηε͢ΔͱɺϝτϦΫεҰཡΛऔಘͰ͖Δ process_ fi les_max_ fi
les{application="bff",} 10240.0 process_uptime_seconds{application="bff",} 15057.411 jvm_threads_peak_threads{application="bff",} 43.0 jvm_threads_states_threads{application="bff",state="new",} 0.0 jvm_threads_states_threads{application="bff",state="runnable",} 11.0 jvm_threads_states_threads{application="bff",state="terminated",} 0.0 jvm_threads_states_threads{application="bff",state="waiting",} 12.0 jvm_threads_states_threads{application="bff",state="timed-waiting",} 9.0 jvm_threads_states_threads{application="bff",state="blocked",} 0.0
σϞ
PrometheusͰϝτϦΫεऩू • ʢGrafanaPrometheusࣗମͷߏஙʹׂ͍ͭͯѪʣ • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ
PrometheusͰϝτϦΫεऩू • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ scrape_con fi gs: -
job_name: "apps" metrics_path: "/actuator/prometheus" static_con fi gs: - targets: [ "host.docker.internal:9000", "host.docker.internal:9001", "host.docker.internal:9002", "host.docker.internal:9010" ]
ΫϥυωΠςΟϒ࣌ʹ ΞϓϦέʔγϣϯͷΞυϨεΛ ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ
ΫϥυωΠςΟϒ࣌ʹ ΞϓϦέʔγϣϯͷΞυϨεΛ ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ ͖ͪΜͱޙͰղઆ͠·͢
σϞ
GrafanaʹμογϡϘʔυΛՃ • GrafanaίϛϡχςΟ͕࡞ͨ͠μογϡϘʔυΛެ։͍ͯ͠Δ • https://grafana.com/grafana/dashboards/ • JVM (Micrometer) - 4701
͕Φεεϝ
σϞ
Prometheuspullܕ • ΞϓϦέʔγϣϯଆ͔ΒσʔλΛૹΔͷΛʮPushܕʯͱݺͿ • ΞϓϦέʔγϣϯʹऔΓʹߦ͘ͷΛʮPullܕʯͱݺͿ • PrometheusʮPullܕʯͰ͋ΓϝτϦΫεऩूͷϛυϧΣΞͱ ͯ͠ΘΓͱ͍͠෦ྨʢ*ݸਓͷײͰ͢ʣ •
PullܕͰࢄΞϓϦέʔγϣϯͷϝτϦΫεΛͳ͘ ऩूͰ͖ΔʢޙͰղઆ͠·͢ʣ
Prometheuspullܕ • Pullܕͷར • ࢹରͱͳΔΞϓϦέʔγϣϯ͕໌֬Ͱ͋ΔͨΊ ʮϝτϦΫεΛऔಘͰ͖ͳ͔ͬͨʯͱ͍͏Λݕग़͍͢͠ • PushܕͰʮૹΒΕͯ͜ͳ͔ͬͨʯ =
ʮه͠ͳ͍ʯ͚ͩͱ ͳΓɺͦΕΛͱͯ͠ݕग़͢Δ͜ͱ͕͍͠ • ࢹର͕ଟ͘ͳͬͨࡍʹɺPushܕͰϝτϦΫεऩूαʔό͕ύϯΫͯ͠ ͠·͏Մೳੑ͕͋Δ͕ɺPullܕͳΒऩूසΛ੍ޚ͢Δ͜ͱ͕Ͱ͖Δʮ͔ʯ
Spring Boot + LokiͰ ϩάऩू
Spring BootͰϩάऩू • Spring BootΞϓϦέʔγϣϯͷϩάΛPromtailͰఆظతʹऩूͯ͠ Lokiʹૹ৴͢ΔʢPullܕʣ • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ Spring
Boot Application ϩά ϑΝΠϧ Loki ఆظతʹऩू Grafana औಘͯ͠ՄࢹԽ Promtail ૹ৴
Ͱ͏ɺϩάϑΝΠϧͱ͔ ͍ͨ͘ͳ͍Ͱ͢ΑͶ
Spring BootͰϩάऩू • Spring BootΞϓϦέʔγϣϯͷϩάΛLoki4JͷLogback AppenderΛ ༻͍ͯఆظతʹLokiʹૹ৴͢ΔʢPushܕʣ • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application Loki4J Appender Loki ఆظతʹૹ৴ Grafana औಘͯ͠ՄࢹԽ
Loki4JͷLogback AppenderΛՃ • pom.xml ͷdependencyʹloki-logback-appenderΛՃ • ϑΝΠϧίϯιʔϧͰͳ͘LokiʹϩάΛૹΔΑ͏ͳ LogbackͷAppender <dependency>
<groupId>com.github.loki4j</groupId> <artifactId>loki-logback-appender</artifactId> <version>1.3.2</version> <scope>runtime</scope> </dependency>
LogbackͷઃఆϑΝΠϧΛ࡞ • LogbackͷઃఆϑΝΠϧΛ࡞͠ɺAppenderʹLoki4jAppenderΛ ࢦఆ͢Δʢlogback-loki.xmlʣ <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender"> <http> <url>http://localhost:3100/loki/api/v1/push</url>
</http> <format> <label> <pattern>app=${appName},host=${HOSTNAME},traceID=%X{traceId:-NONE},level=%level</pattern> </label> <message> <pattern>${FILE_LOG_PATTERN}</pattern> </message> <sortByTime>true</sortByTime> </format> </appender>
LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ • application.properties ʹͯɺ࡞ͨ͠LogbackͷઃఆϑΝΠϧΛ ༗ޮʹ͢Δ • ͍ͭͰʹɺޙͰτϨʔεऩूͷ࣌ʹ͏τϨʔεID ϩάʹग़ΔΑ͏ʹ͓ͯ͘͠
logging.pattern.level=%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}] logging.con fi g=classpath:logback-loki.xml
LokiͰϩάऩू • ʢLokiࣗମͷߏஙʹׂ͍ͭͯѪʣ • LokiΛىಈ͢ΕΞϓϦέʔγϣϯଆ͔ΒϩάΛPush͢ΔͷͰ LokiଆʹಛʹඞཁͳઃఆͳͲͳ͠
σϞ
ϩάpush͔pull͔ • ྺ࢙తʹϩάPullܕͰऩू • ϑΝΠϧʹग़ྗ͠ɺόονͰωοτϫʔΫετϨʔδʹҠಈͤ͞Δ • FluentdLogstashɺFilebeatͳͲͰऩूͯ͠ElasticsearchʹૹΔ • ϩάΛPushܕʹ͢Δͷෆ҆ •
ϩάʮγεςϜͷෆਖ਼ऴྃલͷ࠷ظͷʯ • ΞϓϦέʔγϣϯϓϩηε͕ࢮΜͰϩά͚͍ͩͨ͠ • PushܕͰड͚औΔଆͷϛυϧΣΞ͕ࢭ·Δͱϩά͕ࣦΘΕΔ
ͦΜͳʹॏཁͳใΛ ͳͥϩάͳͲͱ͍͏ᐆດͳํ๏Ͱ ͦ͏ͱ͢Δͷ͔ʁ
ϩάͷ֓೦Λݟ͖͢ • ఏݴ : ϩάͱ͍͏֓೦ͦͷͷΛݟ͖͢ • ϩάॏཁͱ͞ΕΔΘΓʹݫີʹઃܭ͞Εͳ͍ࣄ͕ଟ͍ • ํ͚ܾͩΊͯޙ։ൃऀͤʢͦͷΘΓʹɺ༗༻Ͱͳ͚ΕౖΒΕΔʣ •
ॏཁͳใͱɺͦ͏Ͱͳ͍ใ͕ࡶଟʹࠞࡏ͢Δ • ຊʹॏཁͳͷɺϩάͰͳ͘ۀσʔλͱͯ͠ଊ͑Δ͖ • ຊʹॏཁͳใԿ͔Λਫ਼ࠪ͠ɺݫີʹઃܭͨ͠͏͑Ͱ ৴པੑͷߴ͍σʔλετΞʹه͢Δ • ϩάͱ͍͏ΈʹཔΔͷʮ͋ͬͨΒخ͍͠ʯʹཹΊΔ͖
ϩάpush͔pull͔ • ͦͷ্ͰɺϩάPushܕ͕ྑ͍ͷ͔ɺPullܕ͕ྑ͍ͷ͔ • Pullܕͷํ͕ॊೈੑ͕ߴ͍ • ऩूͯ͠Ճ͢Δͱ͍͏ϓϩηεΛܦΔ͜ͱ͕Ͱ͖Δ • ͦͷաఔʹKafkaRabbitMQΛڬΜͰෛՙΛࢄͰ͖Δ •
Pushܕͷํ͕༰қ • ϩʔΧϧPCͰ࣮ߦ͢ΔࡍʹɺϩάϑΝΠϧΛग़ྗ͠ͳͯ͘ࡁΉ • Ճ͕ඞཁͳ͍ʢ͘͠JSONܗࣜͰग़ྗ͢Δʣ߹ ϩάऩूαʔόʹߴෛՙֻ͕͔ΔΑ͏ͳ݅Ͱͳ͍߹ͳͲͰ༗ޮ
Spring Boot + TempoͰ τϨʔεऩू
Spring BootͰτϨʔεऩू • τϨʔεऩूͷͨΊʹ࣍ͷ2͕ͭඞཁ • ᶃτϨʔεIDͷ • ʮͲͷϦΫΤετʹඥ͘ॲཧ͔ʯʮ୭͔ΒݺΕ͔ͨʯͱ͍͏ ใΛΞϓϦέʔγϣϯؒͰͤ͞Δ •
ᶄτϨʔεใͷੵ • τϨʔεใΛσʔλετΞʹૹͬͯੵ͢Δ
Spring BootͰτϨʔεIDͷ • Spring BootΞϓϦέʔγϣϯͷؒͷ௨৴ʢHTTPAMQPʣͷ ϔομʹτϨʔεIDΛೖΕͯૹ৴ɺड৴͢Δ Spring Boot
Application1 Spring Boot Application2 Spring Boot Application4 Spring Boot Application3 traceparent = req001-01 traceparent = req001-02 traceparent = req001-01
Spring BootͰτϨʔεIDͷ • τϨʔεIDͷૹड৴ʹSpring Boot 2.x·ͰSpring Cloud Sleuth Λ༻͍͍͕ͯͨɺSpring Boot
3.0͔ΒMicrometer Tracing͕ ར༻Ͱ͖ΔΑ͏ʹͳͬͨ Spring Boot Application Micrometer Tracing RestTemplate Spring Boot Application Micrometer Tracing Rest Controller HTTPϔομʹ τϨʔεIDͷ༩ ʢࣗಈʣ HTTPϔομ͔Β τϨʔεIDͷऔಘ ʢࣗಈʣ
Micrometer TracingͷՃ • pom.xml ͷdependencyʹ micrometer-tracing-bridge-otel ͔ -brave ΛՃ͢ΔʢͲͪΒͰৼΔ͍ʹҧ͍ͳ͍ʣ
• Spring BootͷRestTemplateSpring Cloud StreamΛ༻͍ͨ௨৴Ͱ τϨʔεIDͷ͕ߦΘΕΔʢࣗಈʣ <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing-bridge-otel</artifactId> <scope>runtime</scope> </dependency>
Spring BootͰτϨʔεऩू • Spring BootΞϓϦέʔγϣϯͷτϨʔεใͷهΛ Open Telemetry͔BraveͷϥΠϒϥϦΛ༻͍ͯTempoʹૹ৴͢Δ • Grafana͕Tempo͔ΒτϨʔεใΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application Open Telemetry Tempo ఆظతʹૹ৴ Grafana औಘͯ͠ՄࢹԽ
Open TelemetryͷՃ • Spring Boot͔ΒτϨʔεใΛૹ৴͢Δํ๏ • pom.xml ͷdependencyʹ opentelemetry-exporter-zipkin ͔
zipkin-reporter-brave ΛՃ͢ΔʢৼΔ͍ʹҧ͍ͳ͍ʣ • ZipkinʹτϨʔεใΛૹΔϥΠϒϥϦ ʢTempo͕ZipkinޓAPIΛඋ͍͑ͯΔͨΊར༻Ͱ͖Δʣ <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-exporter-zipkin</artifactId> <scope>runtime</scope> </dependency>
LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ • application.properties ʹͯɺτϨʔεใΛͲΕ͘Β͍ͷׂ߹Ͱ TempoʹૹΔ͔Λઃఆ͢ΔʢσϑΥϧτ10%ʣ • ։ൃதσϞͰ100%ʹͨ͠ํ͕ྑ͍ • ӡ༻தͷΤϥʔൃੜ࣌ʹ༨ͣ͞τϨʔεΛऔಘ͢ΔͨΊʹ
100%ʹઃఆ͓ͯ͘͠ͱྑ͍ management.tracing.sampling.probability=1.0
TempoͰτϨʔεऩू • ʢTempoࣗମͷߏஙʹׂ͍ͭͯѪʣ • TempoͷઃఆϑΝΠϧʹͯɺZipkinޓAPIΛ༗ޮʹ͢Δ server: http_listen_port: 3200 distributor: receivers:
zipkin:
σϞ
τϨʔεpushͷΈ • ZipkinʢޓAPIؚΉʣτϨʔεใΛड͚औΔͨΊͷAPIΛ ެ։͓ͯ͠Γ֤ΞϓϦέʔγϣϯ͔Βͦ͜ʹτϨʔεใΛૹΔ ʮPushܕʯͷΞʔΩςΫνϟ͕࠾༻͞Ε͍ͯΔ • ͍ͣΕPullܕͷΤʔδΣϯτ͕ग़ͯ͘Δ͔Εͳ͍
Spring Boot 3.0 + k8sͷ ϞχλϦϯά
k8s্ͰPullܕΛӡ༻͢Δ՝ • PrometheusͰϞχλϦϯάରͷΞυϨεΛྻڍͨ͠ • k8sͷΑ͏ʹಈతʹϞχλϦϯάର͕૿ݮɾมԽ͢Δ߹ʹ ͲͷΑ͏ʹऩू͢Δͷ͔ʁ
αʔϏεσΟεΧόϦ • ΞϓϦέʔγϣϯͷϧʔςΟϯάϩʔυόϥϯγϯάͷͨΊʹ ༻͍ΒΕΔαʔϏεσΟεΧόϦͷΈΛϞχλϦϯάͰར༻ ͢Δ • k8sͰCoreDNSΛ༻͍ͨαʔϏεσΟεΧόϦΛߦ͏
αʔϏεσΟεΧόϦ • 20ඵͰ͔ΔαʔϏεσΟεΧόϦ Spring Boot Application1 Service Registry
Spring Boot Application1 Spring Boot Application1 Spring Boot Application1 ᶃΞυϨεͷొ ᶄݺͼग़͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠ Spring Boot Application2
αʔϏεσΟεΧόϦ • αʔϏεσΟεΧόϦΛར༻ͨ͠ϞχλϦϯά Spring Boot Application1 Prometheus Promtail Service
Registry Spring Boot Application1 Spring Boot Application1 Spring Boot Application ᶃΞυϨεͷొ ᶄϞχλϦϯά͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠
PrometheusͷαʔϏεσΟεΧόϦઃఆ • PrometheusͷઃఆϑΝΠϧʹͯɺkubernetes_sd_con fi g ͰαʔϏεσΟεΧόϦΛઃఆ͢Δ scrape_con fi gs: -
job_name: "apps" metrics_path: "/actuator/prometheus" kubernetes_sd_con fi g: - role: pod relabel_con fi gs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (.+?)(?::\d+)?;(\d+) replacement: $1:$2 ଞʹ৭ʑͱઃఆ͕ඞཁ͔ͩΒ PrometheusPromtail helmͰΠϯετʔϧ͢Δͷ͕Φεεϝ
PrometheusͷαʔϏεσΟεΧόϦઃఆ • ΞϓϦέʔγϣϯΛσϓϩΠ͢ΔϚχϑΣετϑΝΠϧʹͯ annotationsͰϙʔτ൪߸ActuatorͷύεΛࢦఆ͢Δ spec: template: metadata: labels: app:
bff annotations: prometheus.io/scrape: "true" prometheus.io/port: "9000" prometheus.io/path: /actuator/prometheus
σϞ
ίϯςφ࣌ͰPullͰେৎ • ϚΠΫϩαʔϏεͷΑ͏ʹಈతʹ૿ݮ͢ΔΞϓϦέʔγϣϯ܈Λ ϞχλϦϯά͢ΔࡍɺαʔϏεσΟεΧόϦͷΈͱซ༻͢Ε PullܕͰಛʹͳ͘ϞχλϦϯάڥΛߏஙͰ͖Δ
·ͱΊ
·ͱΊ • ϝτϦΫεɺϩάɺτϨʔεऔΕΔ͚ͩऔΓ·͠ΐ͏ • GrafanaελοΫͰ͋ΔɺGrafanaɺPrometheusɺLokiɺTempoͰ ·ͱΊͯϞχλϦϯάͰ͖Δ • Spring Boot
3.0ͰϞχλϦϯάपΓͷϥΠϒϥϦͳͲཧ͞Ε GrafanaελοΫͳͲͰϞχλϦϯά͘͢͠ͳ͍ͬͯΔ • ϩά͔ΒτϨʔεͷભҠͳͲɺ͜ΕΒͷ3ͭΛͦΕͧΕ ભҠͰ͖ΔΑ͏ʹ͢ΔτϨϯυ͕͋Δ
·ͱΊ • PushܕɺPullܕͷ͍ͣΕͰͳ͘σʔλΛऩूͰ͖Δ • ૯ͯ͡ɺPushܕͷํ͕༰қɺPullܕͷํ͕ॊೈͱͳ͍ͬͯΔ • DataDogAWS CloudWatchͳͲجຊతʹPushܕ • ΞϓϦέʔγϣϯଆͷιʔείʔυઃఆϑΝΠϧ͕
Πϯϑϥʹґଘ͠ͳ͍Α͏ཹҙ͢Δͱ։ൃ͍͢͠Α
Resources • ιʔείʔυ • https://github.com/cero-t/spring-store-2022 • υΩϡϝϯτ • Spring Boot
3.0ΞϓϦέʔγϣϯΛGrafanaελοΫͰՄࢹԽͯ͠Έͨɻ • https://cero-t.hatenadiary.jp/entry/2023/01/10/163547 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ1 Grafana + Prometheusฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/11/181715 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ2 Grafana + Lokiฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/12/130022 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ3 Grafana + Tempoฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/15/203442