Cloud Run + Observability / Reliability @ KAUCHE

Transcript

1. Cloud Run + Observability / Reliability @ KAUCHE Yuki Ito (@mrno110) Cloud Run Casual Talk #2
2. KAUCHE Architect / Platform Team Yuki Ito @mrno110
4. Agenda ɾArchitecture ɾApproaches - Logs - Traces - Metrics - SLI / SLO
5. Architecture Run Tasks Pub/Sub Mobile App External Service Mobile API Web Hook API Job API Scheduler
6. Architecture
7. Observability our de fi nition of “observability” for software systems is a measure of how well you can understand and explain any state your system can get into, no matter how novel or bizarre. ... If you can understand any bizarre or novel state without needing to ship new code, you have observability. https://www.oreilly.com/library/view/observability-engineering/9781492076438/
8. Observability - Goal ✅ Enable new members to understand system states on Day 1.
9. Agenda ɾArchitecture ɾApproaches - Logs - Traces - Metrics - SLI / SLO
10. Logs • Request logs • Container logs Cloud Run generates two types of logs: https://cloud.google.com/run/docs/logging
11. Logs • Request logs • Container logs Cloud Run generates two types of logs: https://cloud.google.com/run/docs/logging
12. Logs Cloud Run generates Request Logs
13. Logs • Request logs • Container logs Cloud Run generates two types of logs: https://cloud.google.com/run/docs/logging
14. Logs Container (Application) Logs Structured Log
15. Logs Request Logs + Container Logs Correlate Logs https://cloud.google.com/run/docs/logging#correlate-logs
16. Logs Request Logs Container Logs Correlate Logs
17. Logs Correlate Logs
18. Logs { "message": "grpc request", "logger": "grpc.request_logger", "method": "/customer.v1.CustomerService/GetXXX", "level": "info", "timestamp": 1613885945098.689 "logging.googleapis.com/trace": "projects/.../traces/xxx", } https://cloud.google.com/logging/docs/structured-logging Container (Application) Logs
19. Agenda ɾArchitecture ɾApproaches - Logs - Traces - Metrics - SLI / SLO
20. Cloud Trace
21. OpenTelemetry OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. https://opentelemetry.io/
22. Trace Just tracing is not enough...
23. Trace Attributes
24. Trace Events
25. Trace Correlate Logs
26. Agenda ɾArchitecture ɾApproaches - Logs - Traces - Metrics - SLI / SLO
27. Metrics Cloud Run Logging Monitoring Metrics Log Log Based Metrics
28. Metrics Log Based Metrics
29. Metrics Cloud Run OpenTelemetry Monitoring Metrics Metrics OpenTelemetry
30. Agenda ɾArchitecture ɾApproaches - Log - Trace - Metrics - SLI / SLO
31. Cloud Monitoring
32. Cloud Monitoring PromQL
33. Agenda ɾArchitecture ɾApproaches - Log - Trace - Metrics - SLI / SLO