Microservices Observability Zup Open Talks

Nesta talk falamos sobre Observability para microservices e como podemos fazer Canary Releases usando uns dos Pilares, as metricas

Transcript

1. Microservices
   Observability
   

   View Slide

2. Hello!
   I am Cláudio Oliveira
   Technical Lead API Team
   Book Author
   @luizalabs
   Java, Golang, k8s & microservices
   2
   

   View Slide

3. Agenda
   ● Metrics
   ● Distributed Tracing
   ● Logs
   ● Progressive Delivery
   ● Demos
   3
   

   View Slide

4. View Slide

5. Glossary
   5
   Telemetry
   How to collect data that will provide
   observability (sensors)
   Observability
   Monitoring, Alerting and Visualizations,
   Distributing tracing and Log Aggregation
   

   View Slide

6. Glossary
   6
   Monitoring
   Is the practice of collecting signals,
   aggregating them, and matching them
   against some predefined criteria
   

   View Slide

7. 7
   Microservices Drawbacks
   Sh*** happens
   

   View Slide

8. Fallacies of Distributed
   Computing
   

   View Slide

9. Fallacies of Distributed Computing
   9
   ● Network is Reliable
   ● Latency is Zero
   ● Bandwidth is Infinite
   

   View Slide

10. 10
    ● Understand how microservices connect each other
    ● Network latencies can be a bottleneck (intense IPC
    ● Network can be unreliable
    ● Control the UP and Running instances
    ● Increase the non-functional requirements
    Microservices implies “some” challenges
    

    View Slide

11. View Slide

12. Metrics
    

    View Slide

13. 13
    Metrics are the only way to get your
    job done
    

    View Slide

14. 14
    RED pattern to monitor Services
    

    View Slide

15. 15
    R - the number of request per second
    

    View Slide

16. 16
    E - the number of failed requests per
    second
    

    View Slide

17. 17
    D - distributions of the amount of
    time each request takes
    

    View Slide

18. “
    The benefits of treating each service
    the same, from a monitoring
    perspective, is scalability in your
    operations teams
    18
    

    View Slide

19. Use case
    

    View Slide

20. View Slide

21. View Slide

22. Distributed
    Tracing
    

    View Slide

23. How it works???
    23
    ● Assign external Unique ID
    ● Passes it to all services that are involved
    ● Includes the Request ID in Log Messages
    ● Record times information e.g start and end
    time
    

    View Slide

24. OpenTracing OpenTelemetry
    24
    ● Cloud Native Computing Foundation CNCF
    ● It standardizes the instrumentation of apps
    for distributed tracing
    

    View Slide

25. OpenTracing OpenTelemetry
    Concepts
    25
    ● Trace tells the story of a transaction
    ● Span represents a single call
    ● Distributed Tracing systems collecting and
    we can see the graph in a nice interface
    

    View Slide

26. View Slide

27. Logs
    

    View Slide

28. Use 5’s W!!!!
    28
    ● who
    ● what
    ● when
    ● where
    ● why
    

    View Slide

29. Use severity correctly!!!
    29
    ● INFO
    ● DEBUG
    ● WARNING
    ● ERROR
    

    View Slide

30. View Slide

31. Aggregate Logs
    Microservices is distributed systems
    

    View Slide

32. View Slide

33. View Slide

34. What are my
    opinions to get
    observability
    done???
    34
    

    View Slide

35. There are two ways to solve
    this problem
    35
    

    View Slide

36. Before Service Mesh
    

    View Slide

37. View Slide

38. 38
    Things to think about
    

    View Slide

39. Concerns about observability in the
    app
    39
    ● Increase the size of application
    ● Configuration should be done inside the
    application
    ● It will consume the application resources
    ● More control to “customize” metrics and
    distributed tracing
    ● There is no sidecar involved
    

    View Slide

40. After Service Mesh
    

    View Slide

41. View Slide

42. 42
    Things to think about
    

    View Slide

43. Concerns about observability with
    sidecar
    43
    ● One more thing to care about
    ● Control Plane should configure the sidecars
    ● Not so intrusive
    ● The developers can focus on business rules
    ● It is a kind of industry standard today
    

    View Slide

44. View Slide

45. Progressive App
    Delivery with
    ArgoCD && Rollouts
    

    View Slide

46. Progressive App Delivery
    46
    ● Rolling out new features gradually
    ● Avoid downtime as much as possible
    ● Stateless Application is mandatory
    ● The versions should be backwards
    compatible
    ● Blue-Green, Canary Release and others
    

    View Slide

47. 47
    Argo CD is a declarative, GitOps
    continuous delivery tool for
    Kubernetes
    

    View Slide

48. View Slide

49. 49
    Argo Rollouts is a Kubernetes controller and set of CRDs which
    provide advanced deployment capabilities such as blue-green,
    canary, canary analysis, experimentation, and progressive
    delivery features to Kubernetes.
    

    View Slide

50. 50
    But, How it connect it Observability
    stuff????
    

    View Slide

51. View Slide

52. 52
    It should be Automated
    

    View Slide

53. With everything
    metrified we can
    automate release
    process
    53
    

    View Slide

54. 54
    HTTP Calls with Status Code ~2.*
    should be more than 95%
    Release is good to go!!!
    Else
    Ohhh sh****!!!
    

    View Slide

55. View Slide

56. 56
    sum(irate(istio_requests_total{reporter="source",destination_service=~
    "bets-canary.istio.svc.cluster.local",response_code=~"2.*"[2m])) /
    sum(irate(istio_requests_total{reporter="source",destination_servi
    ce=~"bets-canary.istio.svc.cluster.local"[2m]))
    Prometheus Query
    

    View Slide

57. View Slide

58. Conclusions
    

    View Slide

59. 59
    Follow the industry standards.
    Homemade solution is not a good
    way. Always
    

    View Slide

60. 60
    You can start simple and then evolve
    step by step
    

    View Slide

61. 61
    Microservices without observability
    (monitoring, distributed tracing and
    log aggregate) is the worst thing in
    the world
    

    View Slide

62. 62
    Microservices is effective to delivery
    software frequently.
    But THINK seriously in Observability
    

    View Slide

63. View Slide

64. Thanks!
    Any questions?
    You can find me on
    twitter and linkedin
    ● @claudioed
    64
    

    View Slide