Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JCConf 2020 Observing in Microservices

Shihyu Ho
November 18, 2020

JCConf 2020 Observing in Microservices

Lessons learned observing in Microservices

Shihyu Ho

November 18, 2020
Tweet

More Decks by Shihyu Ho

Other Decks in Programming

Transcript

  1. Lessons learned observing


    in Microservices


    Matt Ho

    View Slide

  2. SoftLeader RD
    Matt Ho
    https://github.com/shihyuho

    View Slide

  3. Microservices!

    View Slide

  4. Move from Spring Cloud


    to Kubernetes native!

    View Slide

  5. 好多 App, 平常沒事, But...

    View Slide

  6. Infrastructure
    0

    View Slide

  7. Tracing
    App
    App
    App
    container platform

    View Slide

  8. Monitoring
    App
    App
    App
    container platform
    cpu/memory/VM heap …
    cpu/memory/VM heap …

    View Slide

  9. Loggin
    g


    App
    App
    App
    container platform

    View Slide

  10. Kubernetes on-premises

    View Slide

  11. 原⽣的, ⼀裝好什麼都沒有!


    View Slide

  12. 不能客製呦!

    View Slide

  13. 不是在做比較

    View Slide

  14. Tracing
    1

    View Slide

  15. Using OpenTracing
    #

    View Slide

  16. github.com/opentracing-contrib/
    java-spring-cloud

    View Slide

  17. Put tracing in response
    #

    View Slide

  18. App
    filter
    response
    request
    api
    add trace info

    View Slide

  19. Add tracing in logger MDC
    #

    View Slide

  20. App A
    log
    MDC

    View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. View Slide

  26. Create your Spans
    #

    View Slide

  27. time
    Span A
    Span B Span C
    Span D
    外部呼叫
    trace

    View Slide

  28. View Slide

  29. Propagating messages
    #

    View Slide

  30. View Slide

  31. App A App B
    App C
    App D
    Queue
    Baggage

    View Slide

  32. App A App B
    App C
    App D
    REST
    gRPC
    Queue
    Enqueue
    Dequeue
    Baggage

    View Slide

  33. View Slide

  34. Monitoring
    2

    View Slide

  35. Prometheus & exporters
    #

    View Slide

  36. - node-exporter - Hardware/OS metric
    s


    - cAdvisor - Container metric
    s


    - kube-state-metrics - Kubernetes Object metrics
    環境類 exporters

    View Slide

  37. github.com/prometheus-operator/
    kube-prometheus

    View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. - blackbox - Probing over HTTP, HTTPS, DNS



    - micrometer.io - Java app metrics
    服務類 exporters

    View Slide

  42. View Slide

  43. View Slide

  44. https://grafana.com/grafana/
    dashboards

    View Slide

  45. Logging
    3

    View Slide

  46. Define logging format
    #

    View Slide

  47. JSON logging format

    View Slide

  48. github.com/logstash/
    logstash-logback-encoder

    View Slide

  49. Append app information
    #

    View Slide

  50. app_name, app_version…

    View Slide

  51. View Slide

  52. View Slide

  53. ES index strategy
    #

    View Slide

  54. By namespace and date

    View Slide

  55. View Slide

  56. Forwarding logs from { … }
    #

    View Slide

  57. Fluentd
    Forward
    Log format
    OCP
    DockerSwarm
    Forward
    Other Cluster

    View Slide

  58. App
    App
    App
    App
    App
    App

    View Slide

  59. 好多 App, 平常沒事, But...

    View Slide

  60. Thanks

    View Slide