クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレースモニタリング

Java Developers Summit (2023-02-28)

Transcript

1. Ϋϥ΢υωΠςΟϒ࣌୅ͷ
 ίϯςφ؀ڥʹ͓͚ΔJavaΞϓϦέʔγϣϯͷ
 ϝτϦΫεɾϩάɾτϨʔεϞχλϦϯά Everforth / Acroquest Technology / JJUG Shin Tanimoto (@cero_t)
2. ࣗݾ঺հ • ୩ຊ ৺ (Shin Tanimoto) • 2ࣾͰಇ͘ΤϯδχΞ • גࣜձࣾEverforthʢCTOʣ • Acroquest Technologyגࣜձࣾ • ೔ຊJavaϢʔβʔάϧʔϓʢ୅දʣ • Twitter : @cero_t
3. ࠓ೔ͷϝΠϯςʔϚ
4. ྩ࿨ʹ͓͚ΔΦϒβόϦςΟ ͷΞʔΩςΫνϟʹ͍ͭͯ
5. ϞχλϦϯάͷ3େཁૉ • ϝτϦΫε • ϩά • τϨʔε
6. ϞχλϦϯάͷ3େཁૉ • ϝτϦΫε : ࣌ܥྻͷ਺஋৘ใ • ϩά : λΠϜελϯϓ͖ͭͷจࣈྻ • τϨʔε : ݺͼग़͠ͷه࿥
7. ϝτϦΫε • ࣌ܥྻͷ਺஋৘ใ • ྫ • OS : CPU࢖༻཰ɺϝϞϦ࢖༻ྔʢ཰ʣɺσΟεΫ࢖༻ྔʢ཰ʣ • JavaVM : ώʔϓ࢖༻ྔʢ཰ʣɺεϨου਺ɺίωΫγϣϯϓʔϧ਺ • App : ϦΫΤετ਺ɺΤϥʔ਺ʢ཰ʣ
8. ϝτϦΫε • αϯϓϦϯάϕʔε • ͦͷॠؒͷ஋͔ɺूܭͨ͠஋Λऔಘ͢Δ • CPU࢖༻཰ : ʮඵؒʯͷ࢖༻཰ͷฏۉ͸Կ%͔ • ϝϞϦ࢖༻ྔ : ʮ͍·ʯ͸ԿMB࢖͍ͬͯΔ͔ • GCճ਺ : ʮ͍··ͰʯʹԿճGCΛߦ͔ͬͨ • αϯϓϦϯάִ͕ؒখ͍͞΄Ͳਫ਼౓্͕͕Δ͕ɺσʔλྔ͕૿͑Δ
9. ϝτϦΫε • ฏ੒ͷϞχλϦϯά • OSϨϕϧͷϝτϦΫε͚ͩऔΓ͕ͪ • JavaϨϕϧͷϝτϦΫεΛऔ͍ͬͯΕ͹େͨ͠΋ͷ • ΞϓϦέʔγϣϯϨϕϧͷϝτϦΫεΛऔΕΔΑ͏ʹ͢Δͷ͸ 
 ૬౰ͳ΍Γ͜Έ • ྩ࿨Ͱ͸͢΂ͯऔΔ΂͠
10. ϩά • ࣌ࠁΛ൐͏จࣈྻ • ྫ • ΞϓϦέʔγϣϯϩάɺOSͷϩάɺGCϩάɺ 
 ϑϨʔϜϫʔΫ΍ϛυϧ΢ΣΞͷϩά • ॲཧΛߦͬͨɺΤϥʔ͕ى͖ͨɺԿΒ͔ͷෆ౎߹ͳ͜ͱ͕ى͖ͨ • ͦͷ࣌ͷίϯςΩετ৘ใʢม਺஋΍ఆ਺஋ͳͲʣ
11. ϩά • Πϕϯτϕʔε • ࣄ͕ى͖ͨ࣌ʹه࿥͢Δ • αϯϓϦϯάϕʔεͱ͸ҟͳΓɺ͢΂ͯه࿥͞ΕΔ • จࣈྻͳͷͰϝτϦΫεʹൺ΂ͯαΠζ͕େ͖͍ • 1ͭͷϦΫΤετʹΑͬͯۀ຿ଆͷσʔλετΞʹॻ͖ࠐ·ΕΔ৘ใྔΑΓ 
 ϩάͷํ͕େ͖͘ͳΔ͜ͱ͕͋Δ • ΞϓϦέʔγϣϯͷσʔλετΞΑΓ΋ϩάͷσʔλετΞͷํ͕ 
 ߴੑೳɾେ༰ྔͳ΋ͷΛٻΊΒΕΔ͜ͱ͕͋Δ
12. ϩά • ฏ੒ͷϞχλϦϯά • ೔͝ͱʹzipͯ͠ωοτϫʔΫετϨʔδʹஔ͖͕ͪ • ඞཁͳ࣌ʹμ΢ϯϩʔυͯ͠։͔͘ɺviͰݟʹߦ͖͕ͪ • ʮERRORʯͱ͍͏ϝοηʔδ͕͋Ε͹ύτϥϯϓ͕ճΓ͕ͪ • ྩ࿨ͷϞχλϦϯάͰ͸ৗʹݕࡧͰ͖Δঢ়ଶʹ͢Δ΂͠ • ͦͯ͠໰୊͕ݟ͔ͭͬͨΒSlackͳͲʹ௨஌͢Δ΂͠
13. τϨʔε • ݺͼग़͠ͷه࿥ • ෳ਺ͷΞϓϦέʔγϣϯϓϩηεΛލ͍ͩελοΫτϨʔεͷΑ͏ͳ΋ͷ • ͍ͭɺ୭͕୭ΛݺΜͰ͔ɺͲΕ͘Βֻ͍͔͔ͬͨɺ੒ޭ͔ࣦͨ͠ഊ͔ͨ͠ • ΠϕϯτϕʔεͰऔಘ • ͢΂ͯऔಘ͢Δ͔ɺҰఆͷׂ߹Ͱऔಘ͢Δ͔͸ઃఆՄೳ
14. τϨʔε • ελοΫτϨʔε • Javaϓϩηε಺͚ͩͰ؆ܿ͢Δݺͼग़͠֊૚ • ओʹྫ֎ൃੜ࣌ʹऔಘ͢Δ • ͦΕͳΒෳ਺ΞϓϦέʔγϣϯΛލ͍ͩτϨʔε΋औΓ͍ͨΑͶ • τϨʔε͸ɺ໰୊ൃੜ࣌ͷ֬ೝ΍ɺϘτϧωοΫݕग़ʹར༻Ͱ͖Δ
15. τϨʔε • ฏ੒ͷϞχλϦϯά • τϨʔε͸औΒͳ͍ or औΕͳ͍ • ྩ࿨Ͱ͸औΓ·͠ΐ͏
16. Spring Boot 3.0࣌୅ͷ
 ϞχλϦϯά
17. σϞ
18. σϞͰ࢖ͬͨιʔείʔυ • https://github.com/cero-t/spring-store-2022 • ࣮ߦखॱ΍ઃఆͷৄࡉͳ಺༰͸ϒϩάʹهࡌ • ຊεϥΠυͷ࠷ऴϖʔδʹ·ͱΊ·͢
19. ΞϓϦέʔγϣϯͷߏ੒ BFF Payment 
 Service Order 
 Service Stock 
 Service Delivery 
 Service Item 
 Service Cart 
 Service Queue ECαΠτͷόοΫΤϯυΛΠϝʔδ
20. ར༻ٕͨ͠ज़ελοΫ • ϚΠΫϩαʔϏεΞϓϦέʔγϣϯ • Java 17 • Spring Boot 3.0 (without Spring Cloud) • Spring Boot Web • Micrometer • OpenTelemetry
21. ར༻ٕͨ͠ज़ελοΫ • GrafanaελοΫ • Grafana : ϞχλϦϯάͷUI • Prometheus : ϝτϦΫεͷऩूͱσʔλετΞ • Loki : ϩάͷσʔλετΞ • Tempo : τϨʔεͷσʔλετΞ
22. ϞχλϦϯάͷߏ੒ Java 
 Process Grafana Tempo Prometheus Loki औಘͯ͠ՄࢹԽ ϝτϦΫε ϩά τϨʔε Java 
 Process Java 
 Process Spring Boot 
 Application
23. Spring Boot + PrometheusͰ
 ϝτϦΫεऩू
24. Spring BootͰϝτϦΫεऩू • Spring BootΞϓϦέʔγϣϯͷϝτϦΫε৘ใΛWeb APIͱͯ͠ 
 ެ։ͯ͠ɺPrometheus͔Βఆظతʹऔಘͯ͠ऩूɾอଘ͢Δ • Grafana͕Prometheus͔ΒϝτϦΫεΛऔಘͯ͠ՄࢹԽ͢Δ Spring Boot 
 Application ϝτϦΫε औಘAPI Prometheus ఆظతʹऔಘ Grafana औಘͯ͠ՄࢹԽ
25. Spring BootͰϝτϦΫεऩू • ϝτϦΫεऔಘͷWeb APIΛެ։͢Δํ๏ • Spring BootΞϓϦέʔγϣϯͷdependencyʹ 
 ActuatorͱMicrometerΛ௥Ճ͢Δ • application.propertiesʹͯActuatorͷΤϯυϙΠϯτΛެ։͢Δ
26. ActuatorͱMicrometerΛ௥Ճ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛ௥Ճ • Actuator : spring-boot-starter-actuator • Spring Bootͷઃఆ΍ϝτϦΫεΛWeb APIͱͯ͠ެ։͢ΔͨΊͷϥΠϒϥϦ • Micrometer : micrometer-registry-prometheus • OS΍JavaVMɺSpring BootͷϝτϦΫε৘ใΛऔಘ͢ΔͨΊͷϥΠϒϥϦ • “-prometheus” ͸Prometheus͔ΒΞΫηε͢ΔͨΊͷΤϯυϙΠϯτΛ 
 ActuatorΛ༻͍ͯެ։͢Δɻ • “-datadog” ͱ͍͏ϥΠϒϥϦ͸ఆظతʹDataDogʹϝτϦΫεΛૹ৴͢ΔͳͲ 
 ϥΠϒϥϦʹΑͬͯڍಈ͕ҟͳΔ
27. ActuatorͱMicrometerΛ௥Ճ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛ௥Ճ <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> <scope>runtime</scope> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> <scope>runtime</scope> </dependency>
28. ActuatorͷΤϯυϙΠϯτΛެ։ • application.properties ʹͯɺMicrometerʹؔ͢Δ 
 ActuatorͷΤϯυϙΠϯτΛެ։͢Δ management.endpoints.web.exposure.include=* management.metrics.distribution.percentiles-histogram.http.server.requests=true management.metrics.tags.application=${spring.application.name} # Micrometerʹؔ͢ΔϝτϦΫε͚ͩެ։͢Δ৔߹ʹ͸࣍ͷઃఆͰ΋ྑ͍ 
 # management.endpoints.web.exposure.include=prometheus
29. ActuatorͱMicrometerΛ௥Ճ • http://(ΞϓϦέʔγϣϯͷΞυϨε)/actuator/prometheus ʹ 
 ΞΫηε͢ΔͱɺϝτϦΫεҰཡΛऔಘͰ͖Δ process_ fi les_max_ fi les{application="bff",} 10240.0 process_uptime_seconds{application="bff",} 15057.411 jvm_threads_peak_threads{application="bff",} 43.0 jvm_threads_states_threads{application="bff",state="new",} 0.0 jvm_threads_states_threads{application="bff",state="runnable",} 11.0 jvm_threads_states_threads{application="bff",state="terminated",} 0.0 jvm_threads_states_threads{application="bff",state="waiting",} 12.0 jvm_threads_states_threads{application="bff",state="timed-waiting",} 9.0 jvm_threads_states_threads{application="bff",state="blocked",} 0.0
30. σϞ
31. PrometheusͰϝτϦΫεऩू • ʢGrafana΍Prometheusࣗମͷߏஙʹ͍ͭͯ͸ׂѪʣ • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ 
 ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ
32. PrometheusͰϝτϦΫεऩू • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ 
 ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ scrape_con fi gs: - job_name: "apps" metrics_path: "/actuator/prometheus" static_con fi gs: - targets: [ "host.docker.internal:9000", "host.docker.internal:9001", "host.docker.internal:9002", "host.docker.internal:9010" ]
33. Ϋϥ΢υωΠςΟϒ࣌୅ʹ 
 ΞϓϦέʔγϣϯͷΞυϨεΛ 
 ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ
34. Ϋϥ΢υωΠςΟϒ࣌୅ʹ 
 ΞϓϦέʔγϣϯͷΞυϨεΛ 
 ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ ͖ͪΜͱޙͰղઆ͠·͢
35. σϞ
36. GrafanaʹμογϡϘʔυΛ௥Ճ • Grafana͸ίϛϡχςΟ͕࡞੒ͨ͠μογϡϘʔυΛެ։͍ͯ͠Δ • https://grafana.com/grafana/dashboards/ • JVM (Micrometer) - 4701 ͕Φεεϝ
37. σϞ
38. Prometheus͸pullܕ • ΞϓϦέʔγϣϯଆ͔ΒσʔλΛૹΔ΋ͷΛʮPushܕʯͱݺͿ • ΞϓϦέʔγϣϯʹऔΓʹߦ͘΋ͷΛʮPullܕʯͱݺͿ • Prometheus͸ʮPullܕʯͰ͋ΓϝτϦΫεऩूͷϛυϧ΢ΣΞͱ 
 ͯ͠͸ΘΓͱ௝͍͠෦ྨʢ*ݸਓͷײ૝Ͱ͢ʣ • PullܕͰ΋෼ࢄΞϓϦέʔγϣϯͷϝτϦΫεΛ໰୊ͳ͘ 
 ऩूͰ͖Δʢޙ൒Ͱղઆ͠·͢ʣ
39. Prometheus͸pullܕ • Pullܕͷར఺ • ؂ࢹର৅ͱͳΔΞϓϦέʔγϣϯ͕໌֬Ͱ͋ΔͨΊ 
 ʮϝτϦΫεΛऔಘͰ͖ͳ͔ͬͨʯͱ͍͏໰୊Λݕग़͠΍͍͢ • PushܕͰ͸ʮૹΒΕͯ͜ͳ͔ͬͨʯ = ʮه࿥͠ͳ͍ʯ͚ͩͱ 
 ͳΓɺͦΕΛ໰୊ͱͯ͠ݕग़͢Δ͜ͱ͕೉͍͠ • ؂ࢹର৅͕ଟ͘ͳͬͨࡍʹɺPushܕͰ͸ϝτϦΫεऩूαʔό͕ύϯΫͯ͠ 
 ͠·͏Մೳੑ͕͋Δ͕ɺPullܕͳΒऩूස౓Λ੍ޚ͢Δ͜ͱ͕Ͱ͖Δʮ͔΋ʯ
40. Spring Boot + LokiͰ
 ϩάऩू
41. Spring BootͰϩάऩू • Spring BootΞϓϦέʔγϣϯͷϩάΛPromtailͰఆظతʹऩूͯ͠ 
 Lokiʹૹ৴͢ΔʢPullܕʣ • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ Spring Boot 
 Application ϩά 
 ϑΝΠϧ Loki ఆظతʹऩू Grafana औಘͯ͠ՄࢹԽ Promtail ૹ৴
42. Ͱ΋΋͏ɺϩάϑΝΠϧͱ͔ 
 ࢖͍ͨ͘ͳ͍Ͱ͢ΑͶ
43. Spring BootͰϩάऩू • Spring BootΞϓϦέʔγϣϯͷϩάΛLoki4JͷLogback AppenderΛ 
 ༻͍ͯఆظతʹLokiʹૹ৴͢ΔʢPushܕʣ • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ Spring Boot 
 Application Loki4J 
 Appender Loki ఆظతʹૹ৴ Grafana औಘͯ͠ՄࢹԽ
44. Loki4JͷLogback AppenderΛ௥Ճ • pom.xml ͷdependencyʹloki-logback-appenderΛ௥Ճ • ϑΝΠϧ΍ίϯιʔϧͰ͸ͳ͘LokiʹϩάΛૹΔΑ͏ͳ 
 LogbackͷAppender <dependency> <groupId>com.github.loki4j</groupId> <artifactId>loki-logback-appender</artifactId> 
 <version>1.3.2</version> <scope>runtime</scope> </dependency>
45. LogbackͷઃఆϑΝΠϧΛ࡞੒ • LogbackͷઃఆϑΝΠϧΛ࡞੒͠ɺAppenderʹLoki4jAppenderΛ 
 ࢦఆ͢Δʢlogback-loki.xmlʣ <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender"> <http> <url>http://localhost:3100/loki/api/v1/push</url> </http> <format> <label> <pattern>app=${appName},host=${HOSTNAME},traceID=%X{traceId:-NONE},level=%level</pattern> </label> <message> <pattern>${FILE_LOG_PATTERN}</pattern> </message> <sortByTime>true</sortByTime> </format> </appender>
46. LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ • application.properties ʹͯɺ࡞੒ͨ͠LogbackͷઃఆϑΝΠϧΛ 
 ༗ޮʹ͢Δ • ͍ͭͰʹɺޙͰτϨʔεऩूͷ࣌ʹ࢖͏τϨʔεID΋ 
 ϩάʹग़ΔΑ͏ʹ͓ͯ͘͠ logging.pattern.level=%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}] logging.con fi g=classpath:logback-loki.xml
47. LokiͰϩάऩू • ʢLokiࣗମͷߏஙʹ͍ͭͯ͸ׂѪʣ • LokiΛىಈ͢Ε͹ΞϓϦέʔγϣϯଆ͔ΒϩάΛPush͢ΔͷͰ 
 Lokiଆʹ͸ಛʹඞཁͳઃఆͳͲͳ͠
48. σϞ
49. ϩά͸push͔pull͔ • ྺ࢙తʹϩά͸PullܕͰऩू • ϑΝΠϧʹग़ྗ͠ɺόονͰωοτϫʔΫετϨʔδʹҠಈͤ͞Δ • Fluentd΍LogstashɺFilebeatͳͲͰऩूͯ͠ElasticsearchʹૹΔ • ϩάΛPushܕʹ͢Δͷ͸ෆ҆ • ϩά͸ʮγεςϜͷෆਖ਼ऴྃ௚લͷ࠷ظͷ੠ʯ • ΞϓϦέʔγϣϯϓϩηε͕ࢮΜͰ΋ϩά͚ͩ͸࢒͍ͨ͠ • PushܕͰ͸ड͚औΔଆͷϛυϧ΢ΣΞ͕ࢭ·Δͱϩά͕ࣦΘΕΔ
50. ͦΜͳʹॏཁͳ৘ใΛ
 ͳͥϩάͳͲͱ͍͏ᐆດͳํ๏Ͱ
 ࢒ͦ͏ͱ͢Δͷ͔ʁ
51. ϩάͷ֓೦Λݟ௚͢΂͖ • ఏݴ : ϩάͱ͍͏֓೦ͦͷ΋ͷΛݟ௚͢΂͖ • ϩά͸ॏཁͱ͞ΕΔΘΓʹݫີʹઃܭ͞Εͳ͍ࣄ͕ଟ͍ • ํ਑͚ܾͩΊͯޙ͸։ൃऀ೚ͤʢͦͷΘΓʹɺ༗༻Ͱͳ͚Ε͹ౖΒΕΔʣ • ॏཁͳ৘ใͱɺͦ͏Ͱͳ͍৘ใ͕ࡶଟʹࠞࡏ͢Δ • ຊ౰ʹॏཁͳ΋ͷ͸ɺϩάͰ͸ͳ͘ۀ຿σʔλͱͯ͠ଊ͑Δ΂͖ • ຊ౰ʹॏཁͳ৘ใ͸Կ͔Λਫ਼ࠪ͠ɺݫີʹઃܭͨ͠͏͑Ͱ 
 ৴པੑͷߴ͍σʔλετΞʹه࿥͢Δ • ϩάͱ͍͏࢓૊ΈʹཔΔ΋ͷ͸ʮ͋ͬͨΒخ͍͠ʯʹཹΊΔ΂͖
52. ϩά͸push͔pull͔ • ͦͷ্Ͱɺϩά͸Pushܕ͕ྑ͍ͷ͔ɺPullܕ͕ྑ͍ͷ͔ • Pullܕͷํ͕ॊೈੑ͕ߴ͍ • ऩूͯ͠Ճ޻͢Δͱ͍͏ϓϩηεΛܦΔ͜ͱ͕Ͱ͖Δ • ͦͷաఔʹKafka΍RabbitMQΛڬΜͰෛՙΛ෼ࢄͰ͖Δ • Pushܕͷํ͕༰қ • ϩʔΧϧPCͰ࣮ߦ͢Δࡍʹ΋ɺϩάϑΝΠϧΛग़ྗ͠ͳͯ͘ࡁΉ • Ճ޻͕ඞཁͳ͍ʢ΋͘͠͸JSONܗࣜͰग़ྗ͢Δʣ৔߹΍ 
 ϩάऩूαʔόʹߴෛՙֻ͕͔ΔΑ͏ͳ݅਺Ͱ͸ͳ͍৔߹ͳͲͰ͸༗ޮ
53. Spring Boot + TempoͰ
 τϨʔεऩू
54. Spring BootͰτϨʔεऩू • τϨʔεऩूͷͨΊʹ͸࣍ͷ2͕ͭඞཁ • ᶃτϨʔεIDͷ఻೻ • ʮͲͷϦΫΤετʹඥ෇͘ॲཧ͔ʯʮ୭͔Βݺ͹Ε͔ͨʯͱ͍͏ ৘ใΛΞϓϦέʔγϣϯؒͰ఻೻ͤ͞Δ • ᶄτϨʔε৘ใͷ஝ੵ • τϨʔε৘ใΛσʔλετΞʹૹͬͯ஝ੵ͢Δ
55. Spring BootͰτϨʔεIDͷ఻೻ • Spring BootΞϓϦέʔγϣϯͷؒͷ௨৴ʢHTTP΍AMQPʣͷ 
 ϔομʹτϨʔεIDΛೖΕͯૹ৴ɺड৴͢Δ Spring Boot 
 Application1 Spring Boot 
 Application2 Spring Boot 
 Application4 Spring Boot 
 Application3 traceparent 
 = req001-01 traceparent 
 = req001-02 traceparent 
 = req001-01
56. Spring BootͰτϨʔεIDͷ఻೻ • τϨʔεIDͷૹड৴ʹ͸Spring Boot 2.x·Ͱ͸Spring Cloud Sleuth Λ༻͍͍͕ͯͨɺSpring Boot 3.0͔ΒMicrometer Tracing͕ 
 ར༻Ͱ͖ΔΑ͏ʹͳͬͨ Spring Boot 
 Application Micrometer 
 Tracing RestTemplate Spring Boot 
 Application Micrometer 
 Tracing Rest Controller HTTPϔομʹ 
 τϨʔεIDͷ෇༩ 
 ʢࣗಈʣ HTTPϔομ͔Β 
 τϨʔεIDͷऔಘ 
 ʢࣗಈʣ
57. Micrometer Tracingͷ௥Ճ • pom.xml ͷdependencyʹ micrometer-tracing-bridge-otel ͔ 
 -brave Λ௥Ճ͢ΔʢͲͪΒͰ΋ৼΔ෣͍ʹҧ͍͸ͳ͍ʣ • Spring BootͷRestTemplate΍Spring Cloud StreamΛ༻͍ͨ௨৴Ͱ 
 τϨʔεIDͷ఻೻͕ߦΘΕΔʢࣗಈʣ <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing-bridge-otel</artifactId> <scope>runtime</scope> </dependency>
58. Spring BootͰτϨʔεऩू • Spring BootΞϓϦέʔγϣϯͷτϨʔε৘ใͷه࿥Λ 
 Open Telemetry͔BraveͷϥΠϒϥϦΛ༻͍ͯTempoʹૹ৴͢Δ • Grafana͕Tempo͔ΒτϨʔε৘ใΛऔಘͯ͠ՄࢹԽ͢Δ Spring Boot 
 Application Open 
 Telemetry Tempo ఆظతʹૹ৴ Grafana औಘͯ͠ՄࢹԽ
59. Open Telemetryͷ௥Ճ • Spring Boot͔ΒτϨʔε৘ใΛૹ৴͢Δํ๏ • pom.xml ͷdependencyʹ opentelemetry-exporter-zipkin ͔ 
 zipkin-reporter-brave Λ௥Ճ͢ΔʢৼΔ෣͍ʹҧ͍͸ͳ͍ʣ • ZipkinʹτϨʔε৘ใΛૹΔϥΠϒϥϦ 
 ʢTempo͕Zipkinޓ׵APIΛඋ͍͑ͯΔͨΊར༻Ͱ͖Δʣ <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-exporter-zipkin</artifactId> <scope>runtime</scope> </dependency>
60. LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ • application.properties ʹͯɺτϨʔε৘ใΛͲΕ͘Β͍ͷׂ߹Ͱ TempoʹૹΔ͔Λઃఆ͢ΔʢσϑΥϧτ͸10%ʣ • ։ൃத΍σϞͰ͸100%ʹͨ͠ํ͕ྑ͍ • ӡ༻தͷΤϥʔൃੜ࣌ʹ༨ͣ͞τϨʔεΛऔಘ͢ΔͨΊʹ͸ 
 100%ʹઃఆ͓ͯ͘͠ͱྑ͍ management.tracing.sampling.probability=1.0
61. TempoͰτϨʔεऩू • ʢTempoࣗମͷߏஙʹ͍ͭͯ͸ׂѪʣ • TempoͷઃఆϑΝΠϧʹͯɺZipkinޓ׵APIΛ༗ޮʹ͢Δ server: http_listen_port: 3200 distributor: receivers: zipkin:
62. σϞ
63. τϨʔε͸pushͷΈ • Zipkinʢޓ׵APIؚΉʣ͸τϨʔε৘ใΛड͚औΔͨΊͷAPIΛ 
 ެ։͓ͯ͠Γ֤ΞϓϦέʔγϣϯ͔Βͦ͜ʹτϨʔε৘ใΛૹΔ 
 ʮPushܕʯͷΞʔΩςΫνϟ͕࠾༻͞Ε͍ͯΔ • ͍ͣΕPullܕͷΤʔδΣϯτ͕ग़ͯ͘Δ͔΋஌Εͳ͍
64. Spring Boot 3.0 + k8sͷ
 ϞχλϦϯά
65. k8s্ͰPullܕΛӡ༻͢Δ՝୊ • PrometheusͰ͸ϞχλϦϯάର৅ͷΞυϨεΛྻڍͨ͠ • k8sͷΑ͏ʹಈతʹϞχλϦϯάର৅͕૿ݮɾมԽ͢Δ৔߹ʹ͸ 
 ͲͷΑ͏ʹऩू͢Δͷ͔ʁ
66. αʔϏεσΟεΧόϦ • ΞϓϦέʔγϣϯͷϧʔςΟϯά΍ϩʔυόϥϯγϯάͷͨΊʹ 
 ༻͍ΒΕΔαʔϏεσΟεΧόϦͷ࢓૊ΈΛϞχλϦϯάͰ΋ར༻ ͢Δ • k8sͰ͸CoreDNSΛ༻͍ͨαʔϏεσΟεΧόϦΛߦ͏
67. αʔϏεσΟεΧόϦ • 20ඵͰ෼͔ΔαʔϏεσΟεΧόϦ Spring Boot 
 Application1 Service 
 Registry Spring Boot 
 Application1 Spring Boot 
 Application1 Spring Boot 
 Application1 ᶃΞυϨεͷొ࿥ ᶄݺͼग़͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠ Spring Boot 
 Application2
68. αʔϏεσΟεΧόϦ • αʔϏεσΟεΧόϦΛར༻ͨ͠ϞχλϦϯά Spring Boot 
 Application1 Prometheus Promtail Service 
 Registry Spring Boot 
 Application1 Spring Boot 
 Application1 Spring Boot 
 Application ᶃΞυϨεͷొ࿥ ᶄϞχλϦϯά͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠
69. PrometheusͷαʔϏεσΟεΧόϦઃఆ • PrometheusͷઃఆϑΝΠϧʹͯɺkubernetes_sd_con fi g ͰαʔϏεσΟεΧόϦΛઃఆ͢Δ scrape_con fi gs: - job_name: "apps" metrics_path: "/actuator/prometheus" kubernetes_sd_con fi g: - role: pod 
 relabel_con fi gs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (.+?)(?::\d+)?;(\d+) replacement: $1:$2 ଞʹ΋৭ʑͱઃఆ͕ඞཁ͔ͩΒ 
 Prometheus΍Promtail͸ 
 helmͰΠϯετʔϧ͢Δͷ͕Φεεϝ
70. PrometheusͷαʔϏεσΟεΧόϦઃఆ • ΞϓϦέʔγϣϯΛσϓϩΠ͢ΔϚχϑΣετϑΝΠϧʹͯ 
 annotationsͰϙʔτ൪߸΍ActuatorͷύεΛࢦఆ͢Δ spec: template: metadata: labels: app: bff annotations: prometheus.io/scrape: "true" prometheus.io/port: "9000" prometheus.io/path: /actuator/prometheus
71. σϞ
72. ίϯςφ࣌୅Ͱ΋PullͰେৎ෉ • ϚΠΫϩαʔϏεͷΑ͏ʹಈతʹ૿ݮ͢ΔΞϓϦέʔγϣϯ܈Λ 
 ϞχλϦϯά͢ΔࡍɺαʔϏεσΟεΧόϦͷ࢓૊Έͱซ༻͢Ε͹ 
 PullܕͰ΋ಛʹ໰୊ͳ͘ϞχλϦϯά؀ڥΛߏஙͰ͖Δ
73. ·ͱΊ
74. ·ͱΊ • ϝτϦΫεɺϩάɺτϨʔε͸औΕΔ͚ͩऔΓ·͠ΐ͏ • GrafanaελοΫͰ͋ΔɺGrafanaɺPrometheusɺLokiɺTempoͰ 
 ·ͱΊͯϞχλϦϯάͰ͖Δ • Spring Boot 3.0Ͱ͸ϞχλϦϯάपΓͷϥΠϒϥϦͳͲ΋੔ཧ͞Ε 
 GrafanaελοΫͳͲͰϞχλϦϯά͠΍͘͢ͳ͍ͬͯΔ • ϩά͔ΒτϨʔε΁ͷભҠͳͲɺ͜ΕΒͷ3ͭΛͦΕͧΕ 
 ભҠͰ͖ΔΑ͏ʹ͢ΔτϨϯυ͕͋Δ
75. ·ͱΊ • PushܕɺPullܕͷ͍ͣΕͰ΋໰୊ͳ͘σʔλΛऩूͰ͖Δ • ૯ͯ͡ɺPushܕͷํ͕༰қɺPullܕͷํ͕ॊೈͱͳ͍ͬͯΔ • DataDog΍AWS CloudWatchͳͲ͸جຊతʹPushܕ • ΞϓϦέʔγϣϯଆͷιʔείʔυ΍ઃఆϑΝΠϧ͕ 
 Πϯϑϥʹґଘ͠ͳ͍Α͏ཹҙ͢Δͱ։ൃ͠΍͍͢Α
76. Resources • ιʔείʔυ • https://github.com/cero-t/spring-store-2022 • υΩϡϝϯτ • Spring Boot 3.0ΞϓϦέʔγϣϯΛGrafanaελοΫͰՄࢹԽͯ͠Έͨɻ • https://cero-t.hatenadiary.jp/entry/2023/01/10/163547 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯ؂ࢹͷৄࡉʢͦͷ1 Grafana + Prometheusฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/11/181715 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯ؂ࢹͷৄࡉʢͦͷ2 Grafana + Lokiฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/12/130022 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯ؂ࢹͷৄࡉʢͦͷ3 Grafana + Tempoฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/15/203442