Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ncorpore a Supernanny para seus microservices e entenda o que eles andam aprontando em produção

ncorpore a Supernanny para seus microservices e entenda o que eles andam aprontando em produção

Nesta palestra apresentamos conceitos de Observabilidade para Microservices fazendo uma analogia a algumas atividades da Supernanny

Transcript

  1. Autor: Claudio Oliveira | claudio.oliveira@sensedia.com Autor: Matheus Moraes | matheus.moraes@sensedia.com

    Data: 06/05/2019 Incorpore a Supernanny para seus microservices Entenda o que eles andam aprontando em produção
  2. ▰ Microservices ▰ North/South vs East/West Traffic ▰ Monitoring ▰

    Observability ◦ Logs ◦ Traces ◦ Metrics ▰ Kiali ▰ Demo AGENDA
  3. whoami I am Claudio de Oliveira Book Author, Speaker ,

    Software Architect and Developer @sensedia Spring, Java, Microservices and Docker enthusiast
  4. whoami I am Matheus Moraes Developer & Speaker @sensedia Java,

    NoSQL and Microservices enthusiast
  5. sensedia.com Microservices

  6. 1 The term "Microservice Architecture" ... there are certain common

    characteristics around organization around business capability... https:/ /martinfowler.com/articles/microservices.html
  7. Microservices architecture is not about technologies and frameworks is about

    How to Scale Business
  8. https://www.thoughtworks.com/pt/insights/blog/applying-conways-law-improve-your-software-development ‘ Conway’s law

  9. it means...

  10. Microservices are in general Distributed systems

  11. an user’s transaction happens in different services

  12. Usually these transactions happens over http:// protocol

  13. sensedia.com

  14. in the east/west direction, inside in our microservices infrastructure

  15. sensedia.com

  16. sensedia.com Our microsservices

  17. like children, our microservices can...

  18. they can go down

  19. do not scale

  20. they can get overloaded

  21. memory leak

  22. they may be unhealthy

  23. sensedia.com Monitoring

  24. “Monitoring is the practice of collecting signals, telemetry, traces, etc

    and aggregating them, and matching them against some pre-defined criteria of system states we should carefully watch. When we find that one of our signals has crossed a threshold and may be heading toward a known bad state, we take action to remedy the system - Christian Posta https://www.manning.com/books/istio-in-action
  25. Monitoring is a subset of Observability

  26. sensedia.com Observability

  27. The term Observability sprung around in the software community

  28. “Observability on the other hand supposes up front that our

    systems are highly unpredictable and we cannot know all of the possible failure modes up front https://www.manning.com/books/istio-in-action
  29. We need to collect much more data, even high-cardinality data

    like userIDs, requestIDs, source IPs, etc where the entire set could be exponentially large https://www.manning.com/books/istio-in-action “
  30. a user goes to pay for the items in their

    cart and experiences a 10s delay choosing a payment option. All of the pre-defined metric thresholds (disk usage, queue depth, machine health, etc) might be at acceptable levels https://www.manning.com/books/istio-in-action “
  31. With observability, we need fine-grained data “ https://www.manning.com/books/istio-in-action

  32. sensedia.com Pillars

  33. The 3 pillars of observability Do I have a problem?

    Dashboarding, Trending & Problem Detection Where exactly is my problem? Cross-Service Debug & Performance Optimization What is causing it? Root Cause & Forensics Logs Traces Metrics 1 2 3
  34. sensedia.com Logs

  35. Who What When Where Why INFO DEBUG WARNING ERROR

  36. sensedia.com [INFO ][2019-05-28 00:51:18][de4c1b04-9ca1][c.e.m.g.u.domain.service.PaymentsService] - finding user by id 140708

    [WARN ][2019-05-28 00:51:18][de4c1b04-9ca1][c.e.m.g.u.domain.service.PaymentsService] - user non-cached, calling user service [ERROR][2019-05-28 00:51:18][de4c1b04-9ca1][c.e.m.g.u.domain.service.PaymentsService] - error when calling /users/140708 org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND
  37. We need to aggregate the logs

  38. sensedia.com

  39. Do I have a problem? Dashboarding, Trending & Problem Detection

    Where exactly is my problem? Cross-Service Debug & Performance Optimization What is causing it? Root Cause & Forensics Logs Traces Metrics 1 2 3
  40. sensedia.com Distributed Tracing

  41. We need to understand the application's behavior and be able

    to troubleshoot problems. https://microservices.io/patterns/observability/distributed-tracing.html
  42. sensedia.com

  43. Assign external unique request id passes it to all services

    that are involved includes the request id in log messages Record times information e.g start and end time https://microservices.io/patterns/observability/distributed-tracing.html
  44. sensedia.com

  45. sensedia.com

  46. Solution should have minimal overhead External tool to analysis the

    data https://microservices.io/patterns/observability/distributed-tracing.html
  47. We need an instrumentation standard. Introducing…..

  48. Vendor-neutral APIs and instrumentation for distributed tracing #9 languages

  49. We need a tool to enable us to analyze data.

    Introducing...
  50. JAEGER Monitor and troubleshoot transactions in complex distributed systems https://www.jaegertracing.io/

  51. sensedia.com https://www.jaegertracing.io/

  52. sensedia.com

  53. sensedia.com

  54. Do I have a problem? Dashboarding, Trending & Problem Detection

    Where exactly is my problem? Cross-Service Debug & Performance Optimization What is causing it? Root Cause & Forensics Logs Traces Metrics 1 2 3
  55. sensedia.com Metrics

  56. Health Check ≠ Metrics

  57. sensedia.com payments_http_requests_total{env="prod",method="POST",code="200",type="infra"} 647.0 payments_http_requests_total{env="prod",method="POST",code="400",type="infra"} 74.0

  58. ↑ Cumulative, increasing metric payments_technology_total{env="prod",method="nfc",type="business"} 152.0 Counter ↑ ↓ Gauge

    Single metric that goes up or down payments_hikaricp_connections_active{env="prod",type="infra"} 4.0 Timer Samples and buckets observation payments_crypto_seconds_count{env="prod",type="infra"} 77.0 payments_crypto_seconds_sum{env="prod",type="infra"} 34.97
  59. sensedia.com

  60. store metrics is not the responsibility of microservices

  61. but …. Where is these metrics??? Prometheus

  62. Prometheus is an open source, metrics-based monitoring system

  63. sensedia.com

  64. sensedia.com

  65. We need dashboards more useful and beautiful than the previous

    one
  66. The open platform for beautiful analytics and monitoring

  67. sensedia.com

  68. Do I have a problem? Dashboarding, Trending & Problem Detection

    Where exactly is my problem? Cross-Service Debug & Performance Optimization What is causing it? Root Cause & Forensics Logs Traces Metrics 1 2 3
  69. sensedia.com

  70. Kiali project provides answers to the questions: What microservices are

    part of my Istio service mesh? How are they connected? How are they performing?
  71. sensedia.com

  72. sensedia.com

  73. sensedia.com DEMO

  74. sensedia.com

  75. sensedia.com

  76. sensedia.com

  77. and… the final Question..

  78. Where has Supernanny been working on?

  79. sensedia.com

  80. sensedia.com

  81. sensedia.com

  82. sensedia.com sensedia.com

  83. Kubernetes Prometheus Grafana Micrometer Metrics Fluentd Log Aggregator OpenTracing Jaeger

    Tracing Kiali Istio Sensedia API Platform