Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Microservices Observability Zup Open Talks

Microservices Observability Zup Open Talks

Nesta talk falamos sobre Observability para microservices e como podemos fazer Canary Releases usando uns dos Pilares, as metricas

More Decks by Claudio Eduardo de Oliveira

Other Decks in Programming

Transcript

  1. Microservices
    Observability

    View Slide

  2. Hello!
    I am Cláudio Oliveira
    Technical Lead API Team
    Book Author
    @luizalabs
    Java, Golang, k8s & microservices
    2

    View Slide

  3. Agenda
    ● Metrics
    ● Distributed Tracing
    ● Logs
    ● Progressive Delivery
    ● Demos
    3

    View Slide

  4. 4

    View Slide

  5. Glossary
    5
    Telemetry
    How to collect data that will provide
    observability (sensors)
    Observability
    Monitoring, Alerting and Visualizations,
    Distributing tracing and Log Aggregation

    View Slide

  6. Glossary
    6
    Monitoring
    Is the practice of collecting signals,
    aggregating them, and matching them
    against some predefined criteria

    View Slide

  7. 7
    Microservices Drawbacks
    Sh*** happens

    View Slide

  8. Fallacies of Distributed
    Computing

    View Slide

  9. Fallacies of Distributed Computing
    9
    ● Network is Reliable
    ● Latency is Zero
    ● Bandwidth is Infinite

    View Slide

  10. 10
    ● Understand how microservices connect each other
    ● Network latencies can be a bottleneck (intense IPC
    ● Network can be unreliable
    ● Control the UP and Running instances
    ● Increase the non-functional requirements
    Microservices implies “some” challenges

    View Slide

  11. 11

    View Slide

  12. Metrics

    View Slide

  13. 13
    Metrics are the only way to get your
    job done

    View Slide

  14. 14
    RED pattern to monitor Services

    View Slide

  15. 15
    R - the number of request per second

    View Slide

  16. 16
    E - the number of failed requests per
    second

    View Slide

  17. 17
    D - distributions of the amount of
    time each request takes

    View Slide


  18. The benefits of treating each service
    the same, from a monitoring
    perspective, is scalability in your
    operations teams
    18

    View Slide

  19. Use case

    View Slide

  20. 20

    View Slide

  21. 21

    View Slide

  22. Distributed
    Tracing

    View Slide

  23. How it works???
    23
    ● Assign external Unique ID
    ● Passes it to all services that are involved
    ● Includes the Request ID in Log Messages
    ● Record times information e.g start and end
    time

    View Slide

  24. OpenTracing OpenTelemetry
    24
    ● Cloud Native Computing Foundation CNCF
    ● It standardizes the instrumentation of apps
    for distributed tracing

    View Slide

  25. OpenTracing OpenTelemetry
    Concepts
    25
    ● Trace tells the story of a transaction
    ● Span represents a single call
    ● Distributed Tracing systems collecting and
    we can see the graph in a nice interface

    View Slide

  26. 26

    View Slide

  27. Logs

    View Slide

  28. Use 5’s W!!!!
    28
    ● who
    ● what
    ● when
    ● where
    ● why

    View Slide

  29. Use severity correctly!!!
    29
    ● INFO
    ● DEBUG
    ● WARNING
    ● ERROR

    View Slide

  30. 30

    View Slide

  31. Aggregate Logs
    Microservices is distributed systems

    View Slide

  32. 32

    View Slide

  33. 33

    View Slide

  34. What are my
    opinions to get
    observability
    done???
    34

    View Slide

  35. There are two ways to solve
    this problem
    35

    View Slide

  36. Before Service Mesh

    View Slide

  37. 37

    View Slide

  38. 38
    Things to think about

    View Slide

  39. Concerns about observability in the
    app
    39
    ● Increase the size of application
    ● Configuration should be done inside the
    application
    ● It will consume the application resources
    ● More control to “customize” metrics and
    distributed tracing
    ● There is no sidecar involved

    View Slide

  40. After Service Mesh

    View Slide

  41. 41

    View Slide

  42. 42
    Things to think about

    View Slide

  43. Concerns about observability with
    sidecar
    43
    ● One more thing to care about
    ● Control Plane should configure the sidecars
    ● Not so intrusive
    ● The developers can focus on business rules
    ● It is a kind of industry standard today

    View Slide

  44. 44

    View Slide

  45. Progressive App
    Delivery with
    ArgoCD && Rollouts

    View Slide

  46. Progressive App Delivery
    46
    ● Rolling out new features gradually
    ● Avoid downtime as much as possible
    ● Stateless Application is mandatory
    ● The versions should be backwards
    compatible
    ● Blue-Green, Canary Release and others

    View Slide

  47. 47
    Argo CD is a declarative, GitOps
    continuous delivery tool for
    Kubernetes

    View Slide

  48. 48

    View Slide

  49. 49
    Argo Rollouts is a Kubernetes controller and set of CRDs which
    provide advanced deployment capabilities such as blue-green,
    canary, canary analysis, experimentation, and progressive
    delivery features to Kubernetes.

    View Slide

  50. 50
    But, How it connect it Observability
    stuff????

    View Slide

  51. 51

    View Slide

  52. 52
    It should be Automated

    View Slide

  53. With everything
    metrified we can
    automate release
    process
    53

    View Slide

  54. 54
    HTTP Calls with Status Code ~2.*
    should be more than 95%
    Release is good to go!!!
    Else
    Ohhh sh****!!!

    View Slide

  55. 55

    View Slide

  56. 56
    sum(irate(istio_requests_total{reporter="source",destination_service=~
    "bets-canary.istio.svc.cluster.local",response_code=~"2.*"[2m])) /
    sum(irate(istio_requests_total{reporter="source",destination_servi
    ce=~"bets-canary.istio.svc.cluster.local"[2m]))
    Prometheus Query

    View Slide

  57. 57

    View Slide

  58. Conclusions

    View Slide

  59. 59
    Follow the industry standards.
    Homemade solution is not a good
    way. Always

    View Slide

  60. 60
    You can start simple and then evolve
    step by step

    View Slide

  61. 61
    Microservices without observability
    (monitoring, distributed tracing and
    log aggregate) is the worst thing in
    the world

    View Slide

  62. 62
    Microservices is effective to delivery
    software frequently.
    But THINK seriously in Observability

    View Slide

  63. View Slide

  64. Thanks!
    Any questions?
    You can find me on
    twitter and linkedin
    ● @claudioed
    64

    View Slide