Upgrade to Pro — share decks privately, control downloads, hide ads and more …

クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレー...

Shin Tanimoto
February 28, 2023

 クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレースモニタリング

Java Developers Summit (2023-02-28)

Shin Tanimoto

February 28, 2023
Tweet

More Decks by Shin Tanimoto

Other Decks in Programming

Transcript

  1. ࣗݾ঺հ • ୩ຊ ৺ (Shin Tanimoto) • 2ࣾͰಇ͘ΤϯδχΞ • גࣜձࣾEverforthʢCTOʣ

    • Acroquest Technologyגࣜձࣾ • ೔ຊJavaϢʔβʔάϧʔϓʢ୅දʣ • Twitter : @cero_t
  2. ϝτϦΫε • ࣌ܥྻͷ਺஋৘ใ • ྫ • OS : CPU࢖༻཰ɺϝϞϦ࢖༻ྔʢ཰ʣɺσΟεΫ࢖༻ྔʢ཰ʣ •

    JavaVM : ώʔϓ࢖༻ྔʢ཰ʣɺεϨου਺ɺίωΫγϣϯϓʔϧ਺ • App : ϦΫΤετ਺ɺΤϥʔ਺ʢ཰ʣ
  3. ϝτϦΫε • αϯϓϦϯάϕʔε • ͦͷॠؒͷ஋͔ɺूܭͨ͠஋Λऔಘ͢Δ • CPU࢖༻཰ : ʮඵؒʯͷ࢖༻཰ͷฏۉ͸Կ%͔ •

    ϝϞϦ࢖༻ྔ : ʮ͍·ʯ͸ԿMB࢖͍ͬͯΔ͔ • GCճ਺ : ʮ͍··ͰʯʹԿճGCΛߦ͔ͬͨ • αϯϓϦϯάִ͕ؒখ͍͞΄Ͳਫ਼౓্͕͕Δ͕ɺσʔλྔ͕૿͑Δ
  4. ϩά • ࣌ࠁΛ൐͏จࣈྻ • ྫ • ΞϓϦέʔγϣϯϩάɺOSͷϩάɺGCϩάɺ 
 ϑϨʔϜϫʔΫ΍ϛυϧ΢ΣΞͷϩά •

    ॲཧΛߦͬͨɺΤϥʔ͕ى͖ͨɺԿΒ͔ͷෆ౎߹ͳ͜ͱ͕ى͖ͨ • ͦͷ࣌ͷίϯςΩετ৘ใʢม਺஋΍ఆ਺஋ͳͲʣ
  5. ϩά • Πϕϯτϕʔε • ࣄ͕ى͖ͨ࣌ʹه࿥͢Δ • αϯϓϦϯάϕʔεͱ͸ҟͳΓɺ͢΂ͯه࿥͞ΕΔ • จࣈྻͳͷͰϝτϦΫεʹൺ΂ͯαΠζ͕େ͖͍ •

    1ͭͷϦΫΤετʹΑͬͯۀ຿ଆͷσʔλετΞʹॻ͖ࠐ·ΕΔ৘ใྔΑΓ 
 ϩάͷํ͕େ͖͘ͳΔ͜ͱ͕͋Δ • ΞϓϦέʔγϣϯͷσʔλετΞΑΓ΋ϩάͷσʔλετΞͷํ͕ 
 ߴੑೳɾେ༰ྔͳ΋ͷΛٻΊΒΕΔ͜ͱ͕͋Δ
  6. ΞϓϦέʔγϣϯͷߏ੒ BFF Payment 
 Service Order 
 Service Stock 


    Service Delivery 
 Service Item 
 Service Cart 
 Service Queue ECαΠτͷόοΫΤϯυΛΠϝʔδ
  7. ར༻ٕͨ͠ज़ελοΫ • ϚΠΫϩαʔϏεΞϓϦέʔγϣϯ • Java 17 • Spring Boot 3.0

    (without Spring Cloud) • Spring Boot Web • Micrometer • OpenTelemetry
  8. ར༻ٕͨ͠ज़ελοΫ • GrafanaελοΫ • Grafana : ϞχλϦϯάͷUI • Prometheus :

    ϝτϦΫεͷऩूͱσʔλετΞ • Loki : ϩάͷσʔλετΞ • Tempo : τϨʔεͷσʔλετΞ
  9. ϞχλϦϯάͷߏ੒ Java 
 Process Grafana Tempo Prometheus Loki औಘͯ͠ՄࢹԽ ϝτϦΫε

    ϩά τϨʔε Java 
 Process Java 
 Process Spring Boot 
 Application
  10. ActuatorͱMicrometerΛ௥Ճ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛ௥Ճ • Actuator : spring-boot-starter-actuator • Spring

    Bootͷઃఆ΍ϝτϦΫεΛWeb APIͱͯ͠ެ։͢ΔͨΊͷϥΠϒϥϦ • Micrometer : micrometer-registry-prometheus • OS΍JavaVMɺSpring BootͷϝτϦΫε৘ใΛऔಘ͢ΔͨΊͷϥΠϒϥϦ • “-prometheus” ͸Prometheus͔ΒΞΫηε͢ΔͨΊͷΤϯυϙΠϯτΛ 
 ActuatorΛ༻͍ͯެ։͢Δɻ • “-datadog” ͱ͍͏ϥΠϒϥϦ͸ఆظతʹDataDogʹϝτϦΫεΛૹ৴͢ΔͳͲ 
 ϥΠϒϥϦʹΑͬͯڍಈ͕ҟͳΔ
  11. ActuatorͱMicrometerΛ௥Ճ • http://(ΞϓϦέʔγϣϯͷΞυϨε)/actuator/prometheus ʹ 
 ΞΫηε͢ΔͱɺϝτϦΫεҰཡΛऔಘͰ͖Δ process_ fi les_max_ fi

    les{application="bff",} 10240.0 process_uptime_seconds{application="bff",} 15057.411 jvm_threads_peak_threads{application="bff",} 43.0 jvm_threads_states_threads{application="bff",state="new",} 0.0 jvm_threads_states_threads{application="bff",state="runnable",} 11.0 jvm_threads_states_threads{application="bff",state="terminated",} 0.0 jvm_threads_states_threads{application="bff",state="waiting",} 12.0 jvm_threads_states_threads{application="bff",state="timed-waiting",} 9.0 jvm_threads_states_threads{application="bff",state="blocked",} 0.0
  12. PrometheusͰϝτϦΫεऩू • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ 
 ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ scrape_con fi gs: -

    job_name: "apps" metrics_path: "/actuator/prometheus" static_con fi gs: - targets: [ "host.docker.internal:9000", "host.docker.internal:9001", "host.docker.internal:9002", "host.docker.internal:9010" ]
  13. Prometheus͸pullܕ • Pullܕͷར఺ • ؂ࢹର৅ͱͳΔΞϓϦέʔγϣϯ͕໌֬Ͱ͋ΔͨΊ 
 ʮϝτϦΫεΛऔಘͰ͖ͳ͔ͬͨʯͱ͍͏໰୊Λݕग़͠΍͍͢ • PushܕͰ͸ʮૹΒΕͯ͜ͳ͔ͬͨʯ =

    ʮه࿥͠ͳ͍ʯ͚ͩͱ 
 ͳΓɺͦΕΛ໰୊ͱͯ͠ݕग़͢Δ͜ͱ͕೉͍͠ • ؂ࢹର৅͕ଟ͘ͳͬͨࡍʹɺPushܕͰ͸ϝτϦΫεऩूαʔό͕ύϯΫͯ͠ 
 ͠·͏Մೳੑ͕͋Δ͕ɺPullܕͳΒऩूස౓Λ੍ޚ͢Δ͜ͱ͕Ͱ͖Δʮ͔΋ʯ
  14. Loki4JͷLogback AppenderΛ௥Ճ • pom.xml ͷdependencyʹloki-logback-appenderΛ௥Ճ • ϑΝΠϧ΍ίϯιʔϧͰ͸ͳ͘LokiʹϩάΛૹΔΑ͏ͳ 
 LogbackͷAppender <dependency>

    <groupId>com.github.loki4j</groupId> <artifactId>loki-logback-appender</artifactId> 
 <version>1.3.2</version> <scope>runtime</scope> </dependency>
  15. LogbackͷઃఆϑΝΠϧΛ࡞੒ • LogbackͷઃఆϑΝΠϧΛ࡞੒͠ɺAppenderʹLoki4jAppenderΛ 
 ࢦఆ͢Δʢlogback-loki.xmlʣ <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender"> <http> <url>http://localhost:3100/loki/api/v1/push</url>

    </http> <format> <label> <pattern>app=${appName},host=${HOSTNAME},traceID=%X{traceId:-NONE},level=%level</pattern> </label> <message> <pattern>${FILE_LOG_PATTERN}</pattern> </message> <sortByTime>true</sortByTime> </format> </appender>
  16. ϩά͸push͔pull͔ • ྺ࢙తʹϩά͸PullܕͰऩू • ϑΝΠϧʹग़ྗ͠ɺόονͰωοτϫʔΫετϨʔδʹҠಈͤ͞Δ • Fluentd΍LogstashɺFilebeatͳͲͰऩूͯ͠ElasticsearchʹૹΔ • ϩάΛPushܕʹ͢Δͷ͸ෆ҆ •

    ϩά͸ʮγεςϜͷෆਖ਼ऴྃ௚લͷ࠷ظͷ੠ʯ • ΞϓϦέʔγϣϯϓϩηε͕ࢮΜͰ΋ϩά͚ͩ͸࢒͍ͨ͠ • PushܕͰ͸ड͚औΔଆͷϛυϧ΢ΣΞ͕ࢭ·Δͱϩά͕ࣦΘΕΔ
  17. ϩάͷ֓೦Λݟ௚͢΂͖ • ఏݴ : ϩάͱ͍͏֓೦ͦͷ΋ͷΛݟ௚͢΂͖ • ϩά͸ॏཁͱ͞ΕΔΘΓʹݫີʹઃܭ͞Εͳ͍ࣄ͕ଟ͍ • ํ਑͚ܾͩΊͯޙ͸։ൃऀ೚ͤʢͦͷΘΓʹɺ༗༻Ͱͳ͚Ε͹ౖΒΕΔʣ •

    ॏཁͳ৘ใͱɺͦ͏Ͱͳ͍৘ใ͕ࡶଟʹࠞࡏ͢Δ • ຊ౰ʹॏཁͳ΋ͷ͸ɺϩάͰ͸ͳ͘ۀ຿σʔλͱͯ͠ଊ͑Δ΂͖ • ຊ౰ʹॏཁͳ৘ใ͸Կ͔Λਫ਼ࠪ͠ɺݫີʹઃܭͨ͠͏͑Ͱ 
 ৴པੑͷߴ͍σʔλετΞʹه࿥͢Δ • ϩάͱ͍͏࢓૊ΈʹཔΔ΋ͷ͸ʮ͋ͬͨΒخ͍͠ʯʹཹΊΔ΂͖
  18. ϩά͸push͔pull͔ • ͦͷ্Ͱɺϩά͸Pushܕ͕ྑ͍ͷ͔ɺPullܕ͕ྑ͍ͷ͔ • Pullܕͷํ͕ॊೈੑ͕ߴ͍ • ऩूͯ͠Ճ޻͢Δͱ͍͏ϓϩηεΛܦΔ͜ͱ͕Ͱ͖Δ • ͦͷաఔʹKafka΍RabbitMQΛڬΜͰෛՙΛ෼ࢄͰ͖Δ •

    Pushܕͷํ͕༰қ • ϩʔΧϧPCͰ࣮ߦ͢Δࡍʹ΋ɺϩάϑΝΠϧΛग़ྗ͠ͳͯ͘ࡁΉ • Ճ޻͕ඞཁͳ͍ʢ΋͘͠͸JSONܗࣜͰग़ྗ͢Δʣ৔߹΍ 
 ϩάऩूαʔόʹߴෛՙֻ͕͔ΔΑ͏ͳ݅਺Ͱ͸ͳ͍৔߹ͳͲͰ͸༗ޮ
  19. Spring BootͰτϨʔεIDͷ఻೻ • Spring BootΞϓϦέʔγϣϯͷؒͷ௨৴ʢHTTP΍AMQPʣͷ 
 ϔομʹτϨʔεIDΛೖΕͯૹ৴ɺड৴͢Δ Spring Boot 


    Application1 Spring Boot 
 Application2 Spring Boot 
 Application4 Spring Boot 
 Application3 traceparent 
 = req001-01 traceparent 
 = req001-02 traceparent 
 = req001-01
  20. Spring BootͰτϨʔεIDͷ఻೻ • τϨʔεIDͷૹड৴ʹ͸Spring Boot 2.x·Ͱ͸Spring Cloud Sleuth Λ༻͍͍͕ͯͨɺSpring Boot

    3.0͔ΒMicrometer Tracing͕ 
 ར༻Ͱ͖ΔΑ͏ʹͳͬͨ Spring Boot 
 Application Micrometer 
 Tracing RestTemplate Spring Boot 
 Application Micrometer 
 Tracing Rest Controller HTTPϔομʹ 
 τϨʔεIDͷ෇༩ 
 ʢࣗಈʣ HTTPϔομ͔Β 
 τϨʔεIDͷऔಘ 
 ʢࣗಈʣ
  21. Micrometer Tracingͷ௥Ճ • pom.xml ͷdependencyʹ micrometer-tracing-bridge-otel ͔ 
 -brave Λ௥Ճ͢ΔʢͲͪΒͰ΋ৼΔ෣͍ʹҧ͍͸ͳ͍ʣ

    • Spring BootͷRestTemplate΍Spring Cloud StreamΛ༻͍ͨ௨৴Ͱ 
 τϨʔεIDͷ఻೻͕ߦΘΕΔʢࣗಈʣ <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing-bridge-otel</artifactId> <scope>runtime</scope> </dependency>
  22. Open Telemetryͷ௥Ճ • Spring Boot͔ΒτϨʔε৘ใΛૹ৴͢Δํ๏ • pom.xml ͷdependencyʹ opentelemetry-exporter-zipkin ͔

    
 zipkin-reporter-brave Λ௥Ճ͢ΔʢৼΔ෣͍ʹҧ͍͸ͳ͍ʣ • ZipkinʹτϨʔε৘ใΛૹΔϥΠϒϥϦ 
 ʢTempo͕Zipkinޓ׵APIΛඋ͍͑ͯΔͨΊར༻Ͱ͖Δʣ <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-exporter-zipkin</artifactId> <scope>runtime</scope> </dependency>
  23. αʔϏεσΟεΧόϦ • 20ඵͰ෼͔ΔαʔϏεσΟεΧόϦ Spring Boot 
 Application1 Service 
 Registry

    Spring Boot 
 Application1 Spring Boot 
 Application1 Spring Boot 
 Application1 ᶃΞυϨεͷొ࿥ ᶄݺͼग़͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠ Spring Boot 
 Application2
  24. αʔϏεσΟεΧόϦ • αʔϏεσΟεΧόϦΛར༻ͨ͠ϞχλϦϯά Spring Boot 
 Application1 Prometheus Promtail Service

    
 Registry Spring Boot 
 Application1 Spring Boot 
 Application1 Spring Boot 
 Application ᶃΞυϨεͷొ࿥ ᶄϞχλϦϯά͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠
  25. PrometheusͷαʔϏεσΟεΧόϦઃఆ • PrometheusͷઃఆϑΝΠϧʹͯɺkubernetes_sd_con fi g ͰαʔϏεσΟεΧόϦΛઃఆ͢Δ scrape_con fi gs: -

    job_name: "apps" metrics_path: "/actuator/prometheus" kubernetes_sd_con fi g: - role: pod 
 relabel_con fi gs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (.+?)(?::\d+)?;(\d+) replacement: $1:$2 ଞʹ΋৭ʑͱઃఆ͕ඞཁ͔ͩΒ 
 Prometheus΍Promtail͸ 
 helmͰΠϯετʔϧ͢Δͷ͕Φεεϝ
  26. ·ͱΊ • ϝτϦΫεɺϩάɺτϨʔε͸औΕΔ͚ͩऔΓ·͠ΐ͏ • GrafanaελοΫͰ͋ΔɺGrafanaɺPrometheusɺLokiɺTempoͰ 
 ·ͱΊͯϞχλϦϯάͰ͖Δ • Spring Boot

    3.0Ͱ͸ϞχλϦϯάपΓͷϥΠϒϥϦͳͲ΋੔ཧ͞Ε 
 GrafanaελοΫͳͲͰϞχλϦϯά͠΍͘͢ͳ͍ͬͯΔ • ϩά͔ΒτϨʔε΁ͷભҠͳͲɺ͜ΕΒͷ3ͭΛͦΕͧΕ 
 ભҠͰ͖ΔΑ͏ʹ͢ΔτϨϯυ͕͋Δ
  27. Resources • ιʔείʔυ • https://github.com/cero-t/spring-store-2022 • υΩϡϝϯτ • Spring Boot

    3.0ΞϓϦέʔγϣϯΛGrafanaελοΫͰՄࢹԽͯ͠Έͨɻ • https://cero-t.hatenadiary.jp/entry/2023/01/10/163547 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯ؂ࢹͷৄࡉʢͦͷ1 Grafana + Prometheusฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/11/181715 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯ؂ࢹͷৄࡉʢͦͷ2 Grafana + Lokiฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/12/130022 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯ؂ࢹͷৄࡉʢͦͷ3 Grafana + Tempoฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/15/203442