Incorpore a Supernanny para seus microservices e entenda o que andam aprontando em produção

Concepts, tools and best practices of observability in microservices architecture.

Transcript

1. Autor: Claudio Oliveira | claudio.oliveira@sensedia.com Autor: Matheus Moraes | matheus.moraes@sensedia.com Data: 06/05/2019 Incorpore a Supernanny para seus microservices Entenda o que eles andam aprontando em produção
2. ▰ Microservices ▰ North/South vs East/West Traffic ▰ Monitoring ▰ Observability ◦ Logs ◦ Traces ◦ Metrics ▰ Kiali ▰ Demo AGENDA
3. whoami I am Claudio de Oliveira Book Author, Speaker , Software Architect and Developer @sensedia Spring, Java, Microservices and Docker enthusiast
4. whoami I am Matheus Moraes Developer & Speaker @sensedia Java, NoSQL and Microservices enthusiast
5. sensedia.com Microservices
6. 1 The term "Microservice Architecture" ... there are certain common characteristics around organization around business capability... https:/ /martinfowler.com/articles/microservices.html
7. Microservices architecture is not about technologies and frameworks is about How to Scale Business
8. https://www.thoughtworks.com/pt/insights/blog/applying-conways-law-improve-your-software-development ‘ Conway’s law
9. it means...
10. Microservices are in general Distributed systems
11. an user’s transaction happens in different services
12. Usually these transactions happens over http:// protocol
13. sensedia.com
14. in the east/west direction, inside in our microservices infrastructure
15. sensedia.com
16. sensedia.com Our microsservices
17. like children, our microservices can...
18. they can go down
19. do not scale
20. they can get overloaded
21. memory leak
22. they may be unhealthy
23. sensedia.com Monitoring
24. “Monitoring is the practice of collecting signals, telemetry, traces, etc and aggregating them, and matching them against some pre-defined criteria of system states we should carefully watch. When we find that one of our signals has crossed a threshold and may be heading toward a known bad state, we take action to remedy the system - Christian Posta https://www.manning.com/books/istio-in-action
25. Monitoring is a subset of Observability
26. sensedia.com Observability
27. The term Observability sprung around in the software community
28. “Observability on the other hand supposes up front that our systems are highly unpredictable and we cannot know all of the possible failure modes up front https://www.manning.com/books/istio-in-action
29. We need to collect much more data, even high-cardinality data like userIDs, requestIDs, source IPs, etc where the entire set could be exponentially large https://www.manning.com/books/istio-in-action “
30. a user goes to pay for the items in their cart and experiences a 10s delay choosing a payment option. All of the pre-defined metric thresholds (disk usage, queue depth, machine health, etc) might be at acceptable levels https://www.manning.com/books/istio-in-action “
31. With observability, we need fine-grained data “ https://www.manning.com/books/istio-in-action
32. sensedia.com Pillars
33. The 3 pillars of observability Do I have a problem? Dashboarding, Trending & Problem Detection Where exactly is my problem? Cross-Service Debug & Performance Optimization What is causing it? Root Cause & Forensics Logs Traces Metrics 1 2 3
34. sensedia.com Logs
35. Who What When Where Why INFO DEBUG WARNING ERROR
36. sensedia.com [INFO ][2019-05-28 00:51:18][de4c1b04-9ca1][c.e.m.g.u.domain.service.PaymentsService] - finding user by id 140708 [WARN ][2019-05-28 00:51:18][de4c1b04-9ca1][c.e.m.g.u.domain.service.PaymentsService] - user non-cached, calling user service [ERROR][2019-05-28 00:51:18][de4c1b04-9ca1][c.e.m.g.u.domain.service.PaymentsService] - error when calling /users/140708 org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND
37. We need to aggregate the logs
38. sensedia.com
39. Do I have a problem? Dashboarding, Trending & Problem Detection Where exactly is my problem? Cross-Service Debug & Performance Optimization What is causing it? Root Cause & Forensics Logs Traces Metrics 1 2 3
40. sensedia.com Distributed Tracing
41. We need to understand the application's behavior and be able to troubleshoot problems. https://microservices.io/patterns/observability/distributed-tracing.html
42. sensedia.com
43. Assign external unique request id passes it to all services that are involved includes the request id in log messages Record times information e.g start and end time https://microservices.io/patterns/observability/distributed-tracing.html
44. sensedia.com
45. sensedia.com
46. Solution should have minimal overhead External tool to analysis the data https://microservices.io/patterns/observability/distributed-tracing.html
47. We need an instrumentation standard. Introducing…..
48. Vendor-neutral APIs and instrumentation for distributed tracing #9 languages
49. We need a tool to enable us to analyze data. Introducing...
50. JAEGER Monitor and troubleshoot transactions in complex distributed systems https://www.jaegertracing.io/
51. sensedia.com https://www.jaegertracing.io/
52. sensedia.com
53. sensedia.com
54. Do I have a problem? Dashboarding, Trending & Problem Detection Where exactly is my problem? Cross-Service Debug & Performance Optimization What is causing it? Root Cause & Forensics Logs Traces Metrics 1 2 3
55. sensedia.com Metrics
56. Health Check ≠ Metrics
57. sensedia.com payments_http_requests_total{env="prod",method="POST",code="200",type="infra"} 647.0 payments_http_requests_total{env="prod",method="POST",code="400",type="infra"} 74.0
58. ↑ Cumulative, increasing metric payments_technology_total{env="prod",method="nfc",type="business"} 152.0 Counter ↑ ↓ Gauge Single metric that goes up or down payments_hikaricp_connections_active{env="prod",type="infra"} 4.0 Timer Samples and buckets observation payments_crypto_seconds_count{env="prod",type="infra"} 77.0 payments_crypto_seconds_sum{env="prod",type="infra"} 34.97
59. sensedia.com
60. store metrics is not the responsibility of microservices
61. but …. Where is these metrics??? Prometheus
62. Prometheus is an open source, metrics-based monitoring system
63. sensedia.com
64. sensedia.com
65. We need dashboards more useful and beautiful than the previous one
66. The open platform for beautiful analytics and monitoring
67. sensedia.com
68. Do I have a problem? Dashboarding, Trending & Problem Detection Where exactly is my problem? Cross-Service Debug & Performance Optimization What is causing it? Root Cause & Forensics Logs Traces Metrics 1 2 3
69. sensedia.com
70. Kiali project provides answers to the questions: What microservices are part of my Istio service mesh? How are they connected? How are they performing?
71. sensedia.com
72. sensedia.com
73. sensedia.com DEMO
74. sensedia.com
75. sensedia.com
76. sensedia.com
77. and… the final Question..
78. Where has Supernanny been working on?
79. sensedia.com
80. sensedia.com
81. sensedia.com
82. sensedia.com sensedia.com
83. Kubernetes Prometheus Grafana Micrometer Metrics Fluentd Log Aggregator OpenTracing Jaeger Tracing Kiali Istio Sensedia API Platform