JCConf 2020 Observing in Microservices

Lessons learned observing in Microservices

Transcript

1. Lessons learned observing
   
   
   in Microservices
   
   
   Matt Ho
   

   View Slide

2. SoftLeader RD
   Matt Ho
   https://github.com/shihyuho
   

   View Slide

3. Microservices!
   

   View Slide

4. Move from Spring Cloud
   
   
   to Kubernetes native!
   

   View Slide

5. 好多 App, 平常沒事, But...
   

   View Slide

6. Infrastructure
   0
   

   View Slide

7. Tracing
   App
   App
   App
   container platform
   

   View Slide

8. Monitoring
   App
   App
   App
   container platform
   cpu/memory/VM heap …
   cpu/memory/VM heap …
   

   View Slide

9. Loggin
   g
   
   
   App
   App
   App
   container platform
   

   View Slide

10. Kubernetes on-premises
    

    View Slide

11. 原⽣的, ⼀裝好什麼都沒有!
    
    
    

    View Slide

12. 不能客製呦!
    

    View Slide

13. 不是在做比較
    

    View Slide

14. Tracing
    1
    

    View Slide

15. Using OpenTracing
    #
    

    View Slide

16. github.com/opentracing-contrib/
    java-spring-cloud
    

    View Slide

17. Put tracing in response
    #
    

    View Slide

18. App
    filter
    response
    request
    api
    add trace info
    

    View Slide

19. Add tracing in logger MDC
    #
    

    View Slide

20. App A
    log
    MDC
    

    View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. Create your Spans
    #
    

    View Slide

27. time
    Span A
    Span B Span C
    Span D
    外部呼叫
    trace
    

    View Slide

28. View Slide

29. Propagating messages
    #
    

    View Slide

30. View Slide

31. App A App B
    App C
    App D
    Queue
    Baggage
    

    View Slide

32. App A App B
    App C
    App D
    REST
    gRPC
    Queue
    Enqueue
    Dequeue
    Baggage
    

    View Slide

33. View Slide

34. Monitoring
    2
    

    View Slide

35. Prometheus & exporters
    #
    

    View Slide

36. - node-exporter - Hardware/OS metric
    s
    
    
    - cAdvisor - Container metric
    s
    
    
    - kube-state-metrics - Kubernetes Object metrics
    環境類 exporters
    

    View Slide

37. github.com/prometheus-operator/
    kube-prometheus
    

    View Slide

38. View Slide

39. View Slide

40. View Slide

41. - blackbox - Probing over HTTP, HTTPS, DNS
    …
    
    
    - micrometer.io - Java app metrics
    服務類 exporters
    

    View Slide

42. View Slide

43. View Slide

44. https://grafana.com/grafana/
    dashboards
    

    View Slide

45. Logging
    3
    

    View Slide

46. Define logging format
    #
    

    View Slide

47. JSON logging format
    

    View Slide

48. github.com/logstash/
    logstash-logback-encoder
    

    View Slide

49. Append app information
    #
    

    View Slide

50. app_name, app_version…
    

    View Slide

51. View Slide

52. View Slide

53. ES index strategy
    #
    

    View Slide

54. By namespace and date
    

    View Slide

55. View Slide

56. Forwarding logs from { … }
    #
    

    View Slide

57. Fluentd
    Forward
    Log format
    OCP
    DockerSwarm
    Forward
    Other Cluster
    

    View Slide

58. App
    App
    App
    App
    App
    App
    

    View Slide

59. 好多 App, 平常沒事, But...
    

    View Slide

60. Thanks
    

    View Slide