$30 off During Our Annual Pro Sale. View Details »

クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレースモニタリング

Shin Tanimoto
February 28, 2023

 クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレースモニタリング

Java Developers Summit (2023-02-28)

Shin Tanimoto

February 28, 2023
Tweet

More Decks by Shin Tanimoto

Other Decks in Programming

Transcript

  1. Ϋϥ΢υωΠςΟϒ࣌୅ͷ

    ίϯςφ؀ڥʹ͓͚ΔJavaΞϓϦέʔγϣϯͷ

    ϝτϦΫεɾϩάɾτϨʔεϞχλϦϯά
    Everforth / Acroquest Technology / JJUG


    Shin Tanimoto (@cero_t)

    View Slide

  2. ࣗݾ঺հ
    • ୩ຊ ৺ (Shin Tanimoto)


    • 2ࣾͰಇ͘ΤϯδχΞ


    • גࣜձࣾEverforthʢCTOʣ


    • Acroquest Technologyגࣜձࣾ


    • ೔ຊJavaϢʔβʔάϧʔϓʢ୅දʣ


    • Twitter : @cero_t

    View Slide

  3. ࠓ೔ͷϝΠϯςʔϚ

    View Slide

  4. ྩ࿨ʹ͓͚ΔΦϒβόϦςΟ
    ͷΞʔΩςΫνϟʹ͍ͭͯ

    View Slide

  5. ϞχλϦϯάͷ3େཁૉ
    • ϝτϦΫε


    • ϩά


    • τϨʔε

    View Slide

  6. ϞχλϦϯάͷ3େཁૉ
    • ϝτϦΫε : ࣌ܥྻͷ਺஋৘ใ


    • ϩά : λΠϜελϯϓ͖ͭͷจࣈྻ


    • τϨʔε : ݺͼग़͠ͷه࿥

    View Slide

  7. ϝτϦΫε
    • ࣌ܥྻͷ਺஋৘ใ


    • ྫ


    • OS : CPU࢖༻཰ɺϝϞϦ࢖༻ྔʢ཰ʣɺσΟεΫ࢖༻ྔʢ཰ʣ


    • JavaVM : ώʔϓ࢖༻ྔʢ཰ʣɺεϨου਺ɺίωΫγϣϯϓʔϧ਺


    • App : ϦΫΤετ਺ɺΤϥʔ਺ʢ཰ʣ

    View Slide

  8. ϝτϦΫε
    • αϯϓϦϯάϕʔε


    • ͦͷॠؒͷ஋͔ɺूܭͨ͠஋Λऔಘ͢Δ


    • CPU࢖༻཰ : ʮඵؒʯͷ࢖༻཰ͷฏۉ͸Կ%͔


    • ϝϞϦ࢖༻ྔ : ʮ͍·ʯ͸ԿMB࢖͍ͬͯΔ͔


    • GCճ਺ : ʮ͍··ͰʯʹԿճGCΛߦ͔ͬͨ


    • αϯϓϦϯάִ͕ؒখ͍͞΄Ͳਫ਼౓্͕͕Δ͕ɺσʔλྔ͕૿͑Δ

    View Slide

  9. ϝτϦΫε
    • ฏ੒ͷϞχλϦϯά


    • OSϨϕϧͷϝτϦΫε͚ͩऔΓ͕ͪ


    • JavaϨϕϧͷϝτϦΫεΛऔ͍ͬͯΕ͹େͨ͠΋ͷ


    • ΞϓϦέʔγϣϯϨϕϧͷϝτϦΫεΛऔΕΔΑ͏ʹ͢Δͷ͸

    ૬౰ͳ΍Γ͜Έ


    • ྩ࿨Ͱ͸͢΂ͯऔΔ΂͠

    View Slide

  10. ϩά
    • ࣌ࠁΛ൐͏จࣈྻ


    • ྫ


    • ΞϓϦέʔγϣϯϩάɺOSͷϩάɺGCϩάɺ

    ϑϨʔϜϫʔΫ΍ϛυϧ΢ΣΞͷϩά


    • ॲཧΛߦͬͨɺΤϥʔ͕ى͖ͨɺԿΒ͔ͷෆ౎߹ͳ͜ͱ͕ى͖ͨ


    • ͦͷ࣌ͷίϯςΩετ৘ใʢม਺஋΍ఆ਺஋ͳͲʣ

    View Slide

  11. ϩά
    • Πϕϯτϕʔε


    • ࣄ͕ى͖ͨ࣌ʹه࿥͢Δ


    • αϯϓϦϯάϕʔεͱ͸ҟͳΓɺ͢΂ͯه࿥͞ΕΔ


    • จࣈྻͳͷͰϝτϦΫεʹൺ΂ͯαΠζ͕େ͖͍


    • 1ͭͷϦΫΤετʹΑͬͯۀ຿ଆͷσʔλετΞʹॻ͖ࠐ·ΕΔ৘ใྔΑΓ

    ϩάͷํ͕େ͖͘ͳΔ͜ͱ͕͋Δ


    • ΞϓϦέʔγϣϯͷσʔλετΞΑΓ΋ϩάͷσʔλετΞͷํ͕

    ߴੑೳɾେ༰ྔͳ΋ͷΛٻΊΒΕΔ͜ͱ͕͋Δ

    View Slide

  12. ϩά
    • ฏ੒ͷϞχλϦϯά


    • ೔͝ͱʹzipͯ͠ωοτϫʔΫετϨʔδʹஔ͖͕ͪ


    • ඞཁͳ࣌ʹμ΢ϯϩʔυͯ͠։͔͘ɺviͰݟʹߦ͖͕ͪ


    • ʮERRORʯͱ͍͏ϝοηʔδ͕͋Ε͹ύτϥϯϓ͕ճΓ͕ͪ


    • ྩ࿨ͷϞχλϦϯάͰ͸ৗʹݕࡧͰ͖Δঢ়ଶʹ͢Δ΂͠


    • ͦͯ͠໰୊͕ݟ͔ͭͬͨΒSlackͳͲʹ௨஌͢Δ΂͠

    View Slide

  13. τϨʔε
    • ݺͼग़͠ͷه࿥


    • ෳ਺ͷΞϓϦέʔγϣϯϓϩηεΛލ͍ͩελοΫτϨʔεͷΑ͏ͳ΋ͷ


    • ͍ͭɺ୭͕୭ΛݺΜͰ͔ɺͲΕ͘Βֻ͍͔͔ͬͨɺ੒ޭ͔ࣦͨ͠ഊ͔ͨ͠


    • ΠϕϯτϕʔεͰऔಘ


    • ͢΂ͯऔಘ͢Δ͔ɺҰఆͷׂ߹Ͱऔಘ͢Δ͔͸ઃఆՄೳ

    View Slide

  14. τϨʔε
    • ελοΫτϨʔε


    • Javaϓϩηε಺͚ͩͰ؆ܿ͢Δݺͼग़͠֊૚


    • ओʹྫ֎ൃੜ࣌ʹऔಘ͢Δ


    • ͦΕͳΒෳ਺ΞϓϦέʔγϣϯΛލ͍ͩτϨʔε΋औΓ͍ͨΑͶ


    • τϨʔε͸ɺ໰୊ൃੜ࣌ͷ֬ೝ΍ɺϘτϧωοΫݕग़ʹར༻Ͱ͖Δ

    View Slide

  15. τϨʔε
    • ฏ੒ͷϞχλϦϯά


    • τϨʔε͸औΒͳ͍ or औΕͳ͍


    • ྩ࿨Ͱ͸औΓ·͠ΐ͏

    View Slide

  16. Spring Boot 3.0࣌୅ͷ

    ϞχλϦϯά

    View Slide

  17. σϞ

    View Slide

  18. σϞͰ࢖ͬͨιʔείʔυ
    • https://github.com/cero-t/spring-store-2022


    • ࣮ߦखॱ΍ઃఆͷৄࡉͳ಺༰͸ϒϩάʹهࡌ


    • ຊεϥΠυͷ࠷ऴϖʔδʹ·ͱΊ·͢

    View Slide

  19. ΞϓϦέʔγϣϯͷߏ੒
    BFF
    Payment

    Service
    Order

    Service
    Stock

    Service
    Delivery

    Service
    Item

    Service
    Cart

    Service
    Queue
    ECαΠτͷόοΫΤϯυΛΠϝʔδ

    View Slide

  20. ར༻ٕͨ͠ज़ελοΫ
    • ϚΠΫϩαʔϏεΞϓϦέʔγϣϯ


    • Java 17


    • Spring Boot 3.0 (without Spring Cloud)


    • Spring Boot Web


    • Micrometer


    • OpenTelemetry

    View Slide

  21. ར༻ٕͨ͠ज़ελοΫ
    • GrafanaελοΫ


    • Grafana : ϞχλϦϯάͷUI


    • Prometheus : ϝτϦΫεͷऩूͱσʔλετΞ


    • Loki : ϩάͷσʔλετΞ


    • Tempo : τϨʔεͷσʔλετΞ

    View Slide

  22. ϞχλϦϯάͷߏ੒
    Java

    Process
    Grafana
    Tempo
    Prometheus
    Loki
    औಘͯ͠ՄࢹԽ
    ϝτϦΫε
    ϩά
    τϨʔε
    Java

    Process
    Java

    Process
    Spring Boot

    Application

    View Slide

  23. Spring Boot + PrometheusͰ

    ϝτϦΫεऩू

    View Slide

  24. Spring BootͰϝτϦΫεऩू
    • Spring BootΞϓϦέʔγϣϯͷϝτϦΫε৘ใΛWeb APIͱͯ͠

    ެ։ͯ͠ɺPrometheus͔Βఆظతʹऔಘͯ͠ऩूɾอଘ͢Δ


    • Grafana͕Prometheus͔ΒϝτϦΫεΛऔಘͯ͠ՄࢹԽ͢Δ
    Spring Boot

    Application
    ϝτϦΫε
    औಘAPI Prometheus
    ఆظతʹऔಘ
    Grafana
    औಘͯ͠ՄࢹԽ

    View Slide

  25. Spring BootͰϝτϦΫεऩू
    • ϝτϦΫεऔಘͷWeb APIΛެ։͢Δํ๏


    • Spring BootΞϓϦέʔγϣϯͷdependencyʹ

    ActuatorͱMicrometerΛ௥Ճ͢Δ


    • application.propertiesʹͯActuatorͷΤϯυϙΠϯτΛެ։͢Δ

    View Slide

  26. ActuatorͱMicrometerΛ௥Ճ
    • pom.xml ͷdependencyʹ࣍ͷ2ͭΛ௥Ճ


    • Actuator : spring-boot-starter-actuator


    • Spring Bootͷઃఆ΍ϝτϦΫεΛWeb APIͱͯ͠ެ։͢ΔͨΊͷϥΠϒϥϦ


    • Micrometer : micrometer-registry-prometheus


    • OS΍JavaVMɺSpring BootͷϝτϦΫε৘ใΛऔಘ͢ΔͨΊͷϥΠϒϥϦ


    • “-prometheus” ͸Prometheus͔ΒΞΫηε͢ΔͨΊͷΤϯυϙΠϯτΛ

    ActuatorΛ༻͍ͯެ։͢Δɻ


    • “-datadog” ͱ͍͏ϥΠϒϥϦ͸ఆظతʹDataDogʹϝτϦΫεΛૹ৴͢ΔͳͲ

    ϥΠϒϥϦʹΑͬͯڍಈ͕ҟͳΔ

    View Slide

  27. ActuatorͱMicrometerΛ௥Ճ
    • pom.xml ͷdependencyʹ࣍ͷ2ͭΛ௥Ճ



    org.springframework.boot


    spring-boot-starter-actuator


    runtime








    io.micrometer


    micrometer-registry-prometheus


    runtime



    View Slide

  28. ActuatorͷΤϯυϙΠϯτΛެ։
    • application.properties ʹͯɺMicrometerʹؔ͢Δ

    ActuatorͷΤϯυϙΠϯτΛެ։͢Δ
    management.endpoints.web.exposure.include=*


    management.metrics.distribution.percentiles-histogram.http.server.requests=true


    management.metrics.tags.application=${spring.application.name}


    # Micrometerʹؔ͢ΔϝτϦΫε͚ͩެ։͢Δ৔߹ʹ͸࣍ͷઃఆͰ΋ྑ͍

    # management.endpoints.web.exposure.include=prometheus

    View Slide

  29. ActuatorͱMicrometerΛ௥Ճ
    • http://(ΞϓϦέʔγϣϯͷΞυϨε)/actuator/prometheus ʹ

    ΞΫηε͢ΔͱɺϝτϦΫεҰཡΛऔಘͰ͖Δ
    process_
    fi
    les_max_
    fi
    les{application="bff",} 10240.0


    process_uptime_seconds{application="bff",} 15057.411


    jvm_threads_peak_threads{application="bff",} 43.0


    jvm_threads_states_threads{application="bff",state="new",} 0.0


    jvm_threads_states_threads{application="bff",state="runnable",} 11.0


    jvm_threads_states_threads{application="bff",state="terminated",} 0.0


    jvm_threads_states_threads{application="bff",state="waiting",} 12.0


    jvm_threads_states_threads{application="bff",state="timed-waiting",} 9.0


    jvm_threads_states_threads{application="bff",state="blocked",} 0.0

    View Slide

  30. σϞ

    View Slide

  31. PrometheusͰϝτϦΫεऩू
    • ʢGrafana΍Prometheusࣗମͷߏஙʹ͍ͭͯ͸ׂѪʣ


    • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ

    ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ

    View Slide

  32. PrometheusͰϝτϦΫεऩू
    • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ

    ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ
    scrape_con
    fi
    gs:


    - job_name: "apps"


    metrics_path: "/actuator/prometheus"


    static_con
    fi
    gs:


    - targets:


    [


    "host.docker.internal:9000",


    "host.docker.internal:9001",


    "host.docker.internal:9002",


    "host.docker.internal:9010"


    ]

    View Slide

  33. Ϋϥ΢υωΠςΟϒ࣌୅ʹ

    ΞϓϦέʔγϣϯͷΞυϨεΛ

    ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ

    View Slide

  34. Ϋϥ΢υωΠςΟϒ࣌୅ʹ

    ΞϓϦέʔγϣϯͷΞυϨεΛ

    ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ
    ͖ͪΜͱޙͰղઆ͠·͢

    View Slide

  35. σϞ

    View Slide

  36. GrafanaʹμογϡϘʔυΛ௥Ճ
    • Grafana͸ίϛϡχςΟ͕࡞੒ͨ͠μογϡϘʔυΛެ։͍ͯ͠Δ


    • https://grafana.com/grafana/dashboards/


    • JVM (Micrometer) - 4701 ͕Φεεϝ

    View Slide

  37. σϞ

    View Slide

  38. Prometheus͸pullܕ
    • ΞϓϦέʔγϣϯଆ͔ΒσʔλΛૹΔ΋ͷΛʮPushܕʯͱݺͿ


    • ΞϓϦέʔγϣϯʹऔΓʹߦ͘΋ͷΛʮPullܕʯͱݺͿ


    • Prometheus͸ʮPullܕʯͰ͋ΓϝτϦΫεऩूͷϛυϧ΢ΣΞͱ

    ͯ͠͸ΘΓͱ௝͍͠෦ྨʢ*ݸਓͷײ૝Ͱ͢ʣ


    • PullܕͰ΋෼ࢄΞϓϦέʔγϣϯͷϝτϦΫεΛ໰୊ͳ͘

    ऩूͰ͖Δʢޙ൒Ͱղઆ͠·͢ʣ

    View Slide

  39. Prometheus͸pullܕ
    • Pullܕͷར఺


    • ؂ࢹର৅ͱͳΔΞϓϦέʔγϣϯ͕໌֬Ͱ͋ΔͨΊ

    ʮϝτϦΫεΛऔಘͰ͖ͳ͔ͬͨʯͱ͍͏໰୊Λݕग़͠΍͍͢


    • PushܕͰ͸ʮૹΒΕͯ͜ͳ͔ͬͨʯ = ʮه࿥͠ͳ͍ʯ͚ͩͱ

    ͳΓɺͦΕΛ໰୊ͱͯ͠ݕग़͢Δ͜ͱ͕೉͍͠


    • ؂ࢹର৅͕ଟ͘ͳͬͨࡍʹɺPushܕͰ͸ϝτϦΫεऩूαʔό͕ύϯΫͯ͠

    ͠·͏Մೳੑ͕͋Δ͕ɺPullܕͳΒऩूස౓Λ੍ޚ͢Δ͜ͱ͕Ͱ͖Δʮ͔΋ʯ

    View Slide

  40. Spring Boot + LokiͰ

    ϩάऩू

    View Slide

  41. Spring BootͰϩάऩू
    • Spring BootΞϓϦέʔγϣϯͷϩάΛPromtailͰఆظతʹऩूͯ͠

    Lokiʹૹ৴͢ΔʢPullܕʣ


    • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ
    Spring Boot

    Application
    ϩά

    ϑΝΠϧ
    Loki
    ఆظతʹऩू
    Grafana
    औಘͯ͠ՄࢹԽ
    Promtail
    ૹ৴

    View Slide

  42. Ͱ΋΋͏ɺϩάϑΝΠϧͱ͔

    ࢖͍ͨ͘ͳ͍Ͱ͢ΑͶ

    View Slide

  43. Spring BootͰϩάऩू
    • Spring BootΞϓϦέʔγϣϯͷϩάΛLoki4JͷLogback AppenderΛ

    ༻͍ͯఆظతʹLokiʹૹ৴͢ΔʢPushܕʣ


    • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ
    Spring Boot

    Application
    Loki4J

    Appender
    Loki
    ఆظతʹૹ৴
    Grafana
    औಘͯ͠ՄࢹԽ

    View Slide

  44. Loki4JͷLogback AppenderΛ௥Ճ
    • pom.xml ͷdependencyʹloki-logback-appenderΛ௥Ճ


    • ϑΝΠϧ΍ίϯιʔϧͰ͸ͳ͘LokiʹϩάΛૹΔΑ͏ͳ

    LogbackͷAppender



    com.github.loki4j


    loki-logback-appender

    1.3.2


    runtime



    View Slide

  45. LogbackͷઃఆϑΝΠϧΛ࡞੒
    • LogbackͷઃఆϑΝΠϧΛ࡞੒͠ɺAppenderʹLoki4jAppenderΛ

    ࢦఆ͢Δʢlogback-loki.xmlʣ






    http://localhost:3100/loki/api/v1/push











    app=${appName},host=${HOSTNAME},traceID=%X{traceId:-NONE},level=%level








    ${FILE_LOG_PATTERN}





    true






    View Slide

  46. LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ
    • application.properties ʹͯɺ࡞੒ͨ͠LogbackͷઃఆϑΝΠϧΛ

    ༗ޮʹ͢Δ


    • ͍ͭͰʹɺޙͰτϨʔεऩूͷ࣌ʹ࢖͏τϨʔεID΋

    ϩάʹग़ΔΑ͏ʹ͓ͯ͘͠
    logging.pattern.level=%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]


    logging.con
    fi
    g=classpath:logback-loki.xml

    View Slide

  47. LokiͰϩάऩू
    • ʢLokiࣗମͷߏஙʹ͍ͭͯ͸ׂѪʣ


    • LokiΛىಈ͢Ε͹ΞϓϦέʔγϣϯଆ͔ΒϩάΛPush͢ΔͷͰ

    Lokiଆʹ͸ಛʹඞཁͳઃఆͳͲͳ͠

    View Slide

  48. σϞ

    View Slide

  49. ϩά͸push͔pull͔
    • ྺ࢙తʹϩά͸PullܕͰऩू


    • ϑΝΠϧʹग़ྗ͠ɺόονͰωοτϫʔΫετϨʔδʹҠಈͤ͞Δ


    • Fluentd΍LogstashɺFilebeatͳͲͰऩूͯ͠ElasticsearchʹૹΔ


    • ϩάΛPushܕʹ͢Δͷ͸ෆ҆


    • ϩά͸ʮγεςϜͷෆਖ਼ऴྃ௚લͷ࠷ظͷ੠ʯ


    • ΞϓϦέʔγϣϯϓϩηε͕ࢮΜͰ΋ϩά͚ͩ͸࢒͍ͨ͠


    • PushܕͰ͸ड͚औΔଆͷϛυϧ΢ΣΞ͕ࢭ·Δͱϩά͕ࣦΘΕΔ

    View Slide

  50. ͦΜͳʹॏཁͳ৘ใΛ

    ͳͥϩάͳͲͱ͍͏ᐆດͳํ๏Ͱ

    ࢒ͦ͏ͱ͢Δͷ͔ʁ

    View Slide

  51. ϩάͷ֓೦Λݟ௚͢΂͖
    • ఏݴ : ϩάͱ͍͏֓೦ͦͷ΋ͷΛݟ௚͢΂͖


    • ϩά͸ॏཁͱ͞ΕΔΘΓʹݫີʹઃܭ͞Εͳ͍ࣄ͕ଟ͍


    • ํ਑͚ܾͩΊͯޙ͸։ൃऀ೚ͤʢͦͷΘΓʹɺ༗༻Ͱͳ͚Ε͹ౖΒΕΔʣ


    • ॏཁͳ৘ใͱɺͦ͏Ͱͳ͍৘ใ͕ࡶଟʹࠞࡏ͢Δ


    • ຊ౰ʹॏཁͳ΋ͷ͸ɺϩάͰ͸ͳ͘ۀ຿σʔλͱͯ͠ଊ͑Δ΂͖


    • ຊ౰ʹॏཁͳ৘ใ͸Կ͔Λਫ਼ࠪ͠ɺݫີʹઃܭͨ͠͏͑Ͱ

    ৴པੑͷߴ͍σʔλετΞʹه࿥͢Δ


    • ϩάͱ͍͏࢓૊ΈʹཔΔ΋ͷ͸ʮ͋ͬͨΒخ͍͠ʯʹཹΊΔ΂͖

    View Slide

  52. ϩά͸push͔pull͔
    • ͦͷ্Ͱɺϩά͸Pushܕ͕ྑ͍ͷ͔ɺPullܕ͕ྑ͍ͷ͔


    • Pullܕͷํ͕ॊೈੑ͕ߴ͍


    • ऩूͯ͠Ճ޻͢Δͱ͍͏ϓϩηεΛܦΔ͜ͱ͕Ͱ͖Δ


    • ͦͷաఔʹKafka΍RabbitMQΛڬΜͰෛՙΛ෼ࢄͰ͖Δ


    • Pushܕͷํ͕༰қ


    • ϩʔΧϧPCͰ࣮ߦ͢Δࡍʹ΋ɺϩάϑΝΠϧΛग़ྗ͠ͳͯ͘ࡁΉ


    • Ճ޻͕ඞཁͳ͍ʢ΋͘͠͸JSONܗࣜͰग़ྗ͢Δʣ৔߹΍

    ϩάऩूαʔόʹߴෛՙֻ͕͔ΔΑ͏ͳ݅਺Ͱ͸ͳ͍৔߹ͳͲͰ͸༗ޮ

    View Slide

  53. Spring Boot + TempoͰ

    τϨʔεऩू

    View Slide

  54. Spring BootͰτϨʔεऩू
    • τϨʔεऩूͷͨΊʹ͸࣍ͷ2͕ͭඞཁ


    • ᶃτϨʔεIDͷ఻೻


    • ʮͲͷϦΫΤετʹඥ෇͘ॲཧ͔ʯʮ୭͔Βݺ͹Ε͔ͨʯͱ͍͏
    ৘ใΛΞϓϦέʔγϣϯؒͰ఻೻ͤ͞Δ


    • ᶄτϨʔε৘ใͷ஝ੵ


    • τϨʔε৘ใΛσʔλετΞʹૹͬͯ஝ੵ͢Δ

    View Slide

  55. Spring BootͰτϨʔεIDͷ఻೻
    • Spring BootΞϓϦέʔγϣϯͷؒͷ௨৴ʢHTTP΍AMQPʣͷ

    ϔομʹτϨʔεIDΛೖΕͯૹ৴ɺड৴͢Δ
    Spring Boot

    Application1
    Spring Boot

    Application2
    Spring Boot

    Application4
    Spring Boot

    Application3
    traceparent

    = req001-01
    traceparent

    = req001-02
    traceparent

    = req001-01

    View Slide

  56. Spring BootͰτϨʔεIDͷ఻೻
    • τϨʔεIDͷૹड৴ʹ͸Spring Boot 2.x·Ͱ͸Spring Cloud Sleuth
    Λ༻͍͍͕ͯͨɺSpring Boot 3.0͔ΒMicrometer Tracing͕

    ར༻Ͱ͖ΔΑ͏ʹͳͬͨ
    Spring Boot

    Application
    Micrometer

    Tracing
    RestTemplate
    Spring Boot

    Application
    Micrometer

    Tracing
    Rest
    Controller
    HTTPϔομʹ

    τϨʔεIDͷ෇༩

    ʢࣗಈʣ
    HTTPϔομ͔Β

    τϨʔεIDͷऔಘ

    ʢࣗಈʣ

    View Slide

  57. Micrometer Tracingͷ௥Ճ
    • pom.xml ͷdependencyʹ micrometer-tracing-bridge-otel ͔

    -brave Λ௥Ճ͢ΔʢͲͪΒͰ΋ৼΔ෣͍ʹҧ͍͸ͳ͍ʣ


    • Spring BootͷRestTemplate΍Spring Cloud StreamΛ༻͍ͨ௨৴Ͱ

    τϨʔεIDͷ఻೻͕ߦΘΕΔʢࣗಈʣ



    io.micrometer


    micrometer-tracing-bridge-otel


    runtime



    View Slide

  58. Spring BootͰτϨʔεऩू
    • Spring BootΞϓϦέʔγϣϯͷτϨʔε৘ใͷه࿥Λ

    Open Telemetry͔BraveͷϥΠϒϥϦΛ༻͍ͯTempoʹૹ৴͢Δ


    • Grafana͕Tempo͔ΒτϨʔε৘ใΛऔಘͯ͠ՄࢹԽ͢Δ
    Spring Boot

    Application
    Open

    Telemetry
    Tempo
    ఆظతʹૹ৴
    Grafana
    औಘͯ͠ՄࢹԽ

    View Slide

  59. Open Telemetryͷ௥Ճ
    • Spring Boot͔ΒτϨʔε৘ใΛૹ৴͢Δํ๏


    • pom.xml ͷdependencyʹ opentelemetry-exporter-zipkin ͔

    zipkin-reporter-brave Λ௥Ճ͢ΔʢৼΔ෣͍ʹҧ͍͸ͳ͍ʣ


    • ZipkinʹτϨʔε৘ใΛૹΔϥΠϒϥϦ

    ʢTempo͕Zipkinޓ׵APIΛඋ͍͑ͯΔͨΊར༻Ͱ͖Δʣ



    io.opentelemetry


    opentelemetry-exporter-zipkin


    runtime



    View Slide

  60. LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ
    • application.properties ʹͯɺτϨʔε৘ใΛͲΕ͘Β͍ͷׂ߹Ͱ
    TempoʹૹΔ͔Λઃఆ͢ΔʢσϑΥϧτ͸10%ʣ


    • ։ൃத΍σϞͰ͸100%ʹͨ͠ํ͕ྑ͍


    • ӡ༻தͷΤϥʔൃੜ࣌ʹ༨ͣ͞τϨʔεΛऔಘ͢ΔͨΊʹ͸

    100%ʹઃఆ͓ͯ͘͠ͱྑ͍
    management.tracing.sampling.probability=1.0

    View Slide

  61. TempoͰτϨʔεऩू
    • ʢTempoࣗମͷߏஙʹ͍ͭͯ͸ׂѪʣ


    • TempoͷઃఆϑΝΠϧʹͯɺZipkinޓ׵APIΛ༗ޮʹ͢Δ
    server:


    http_listen_port: 3200


    distributor:


    receivers:


    zipkin:

    View Slide

  62. σϞ

    View Slide

  63. τϨʔε͸pushͷΈ
    • Zipkinʢޓ׵APIؚΉʣ͸τϨʔε৘ใΛड͚औΔͨΊͷAPIΛ

    ެ։͓ͯ͠Γ֤ΞϓϦέʔγϣϯ͔Βͦ͜ʹτϨʔε৘ใΛૹΔ

    ʮPushܕʯͷΞʔΩςΫνϟ͕࠾༻͞Ε͍ͯΔ


    • ͍ͣΕPullܕͷΤʔδΣϯτ͕ग़ͯ͘Δ͔΋஌Εͳ͍

    View Slide

  64. Spring Boot 3.0 + k8sͷ

    ϞχλϦϯά

    View Slide

  65. k8s্ͰPullܕΛӡ༻͢Δ՝୊
    • PrometheusͰ͸ϞχλϦϯάର৅ͷΞυϨεΛྻڍͨ͠


    • k8sͷΑ͏ʹಈతʹϞχλϦϯάର৅͕૿ݮɾมԽ͢Δ৔߹ʹ͸

    ͲͷΑ͏ʹऩू͢Δͷ͔ʁ

    View Slide

  66. αʔϏεσΟεΧόϦ
    • ΞϓϦέʔγϣϯͷϧʔςΟϯά΍ϩʔυόϥϯγϯάͷͨΊʹ

    ༻͍ΒΕΔαʔϏεσΟεΧόϦͷ࢓૊ΈΛϞχλϦϯάͰ΋ར༻
    ͢Δ


    • k8sͰ͸CoreDNSΛ༻͍ͨαʔϏεσΟεΧόϦΛߦ͏

    View Slide

  67. αʔϏεσΟεΧόϦ
    • 20ඵͰ෼͔ΔαʔϏεσΟεΧόϦ
    Spring Boot

    Application1
    Service

    Registry
    Spring Boot

    Application1
    Spring Boot

    Application1
    Spring Boot

    Application1
    ᶃΞυϨεͷొ࿥ ᶄݺͼग़͍ͨ͠αʔϏεͷΞυϨεΛऔಘ
    ᶅݺͼग़͠ Spring Boot

    Application2

    View Slide

  68. αʔϏεσΟεΧόϦ
    • αʔϏεσΟεΧόϦΛར༻ͨ͠ϞχλϦϯά
    Spring Boot

    Application1 Prometheus


    Promtail
    Service

    Registry
    Spring Boot

    Application1
    Spring Boot

    Application1
    Spring Boot

    Application
    ᶃΞυϨεͷొ࿥ ᶄϞχλϦϯά͍ͨ͠αʔϏεͷΞυϨεΛऔಘ
    ᶅݺͼग़͠

    View Slide

  69. PrometheusͷαʔϏεσΟεΧόϦઃఆ
    • PrometheusͷઃఆϑΝΠϧʹͯɺkubernetes_sd_con
    fi
    g ͰαʔϏεσΟεΧόϦΛઃఆ͢Δ
    scrape_con
    fi
    gs:


    - job_name: "apps"


    metrics_path: "/actuator/prometheus"


    kubernetes_sd_con
    fi
    g:


    - role: pod

    relabel_con
    fi
    gs:


    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]


    action: keep


    regex: true


    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]


    action: replace


    target_label: __metrics_path__


    regex: (.+)


    - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]


    action: replace


    target_label: __address__


    regex: (.+?)(?::\d+)?;(\d+)


    replacement: $1:$2
    ଞʹ΋৭ʑͱઃఆ͕ඞཁ͔ͩΒ

    Prometheus΍Promtail͸

    helmͰΠϯετʔϧ͢Δͷ͕Φεεϝ

    View Slide

  70. PrometheusͷαʔϏεσΟεΧόϦઃఆ
    • ΞϓϦέʔγϣϯΛσϓϩΠ͢ΔϚχϑΣετϑΝΠϧʹͯ

    annotationsͰϙʔτ൪߸΍ActuatorͷύεΛࢦఆ͢Δ
    spec:


    template:


    metadata:


    labels:


    app: bff


    annotations:


    prometheus.io/scrape: "true"


    prometheus.io/port: "9000"


    prometheus.io/path: /actuator/prometheus

    View Slide

  71. σϞ

    View Slide

  72. ίϯςφ࣌୅Ͱ΋PullͰେৎ෉
    • ϚΠΫϩαʔϏεͷΑ͏ʹಈతʹ૿ݮ͢ΔΞϓϦέʔγϣϯ܈Λ

    ϞχλϦϯά͢ΔࡍɺαʔϏεσΟεΧόϦͷ࢓૊Έͱซ༻͢Ε͹

    PullܕͰ΋ಛʹ໰୊ͳ͘ϞχλϦϯά؀ڥΛߏஙͰ͖Δ

    View Slide

  73. ·ͱΊ

    View Slide

  74. ·ͱΊ
    • ϝτϦΫεɺϩάɺτϨʔε͸औΕΔ͚ͩऔΓ·͠ΐ͏


    • GrafanaελοΫͰ͋ΔɺGrafanaɺPrometheusɺLokiɺTempoͰ

    ·ͱΊͯϞχλϦϯάͰ͖Δ


    • Spring Boot 3.0Ͱ͸ϞχλϦϯάपΓͷϥΠϒϥϦͳͲ΋੔ཧ͞Ε

    GrafanaελοΫͳͲͰϞχλϦϯά͠΍͘͢ͳ͍ͬͯΔ


    • ϩά͔ΒτϨʔε΁ͷભҠͳͲɺ͜ΕΒͷ3ͭΛͦΕͧΕ

    ભҠͰ͖ΔΑ͏ʹ͢ΔτϨϯυ͕͋Δ

    View Slide

  75. ·ͱΊ
    • PushܕɺPullܕͷ͍ͣΕͰ΋໰୊ͳ͘σʔλΛऩूͰ͖Δ


    • ૯ͯ͡ɺPushܕͷํ͕༰қɺPullܕͷํ͕ॊೈͱͳ͍ͬͯΔ


    • DataDog΍AWS CloudWatchͳͲ͸جຊతʹPushܕ


    • ΞϓϦέʔγϣϯଆͷιʔείʔυ΍ઃఆϑΝΠϧ͕

    Πϯϑϥʹґଘ͠ͳ͍Α͏ཹҙ͢Δͱ։ൃ͠΍͍͢Α

    View Slide

  76. Resources
    • ιʔείʔυ


    • https://github.com/cero-t/spring-store-2022


    • υΩϡϝϯτ


    • Spring Boot 3.0ΞϓϦέʔγϣϯΛGrafanaελοΫͰՄࢹԽͯ͠Έͨɻ


    • https://cero-t.hatenadiary.jp/entry/2023/01/10/163547


    • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯ؂ࢹͷৄࡉʢͦͷ1 Grafana + Prometheusฤʣ


    • https://cero-t.hatenadiary.jp/entry/2023/01/11/181715


    • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯ؂ࢹͷৄࡉʢͦͷ2 Grafana + Lokiฤʣ


    • https://cero-t.hatenadiary.jp/entry/2023/01/12/130022


    • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯ؂ࢹͷৄࡉʢͦͷ3 Grafana + Tempoฤʣ


    • https://cero-t.hatenadiary.jp/entry/2023/01/15/203442

    View Slide