Cloud Run + Observability / Reliability @ KAUCHE

Transcript

1. Cloud Run + Observability / Reliability
   
   
   @ KAUCHE
   Yuki Ito (@mrno110)
   Cloud Run Casual Talk #2
   

   View Slide

2. KAUCHE
   
   
   Architect / Platform Team
   Yuki Ito
   
   
   @mrno110
   

   View Slide

3. View Slide

4. Agenda
   ɾArchitecture
   
   
   ɾApproaches
   
   
   - Logs
   
   
   - Traces
   
   
   - Metrics
   
   
   - SLI / SLO
   

   View Slide

5. Architecture
   Run
   Tasks
   Pub/Sub
   Mobile App External Service
   Mobile API Web Hook API Job API
   Scheduler
   

   View Slide

6. Architecture
   

   View Slide

7. Observability
   our de
   fi
   nition of “observability” for
   software systems is a measure of how
   well you can understand and explain
   any state your system can get into, no
   matter how novel or bizarre.
   
   
   ...
   
   
   If you can understand any bizarre or
   novel state without needing to ship
   new code, you have observability.
   https://www.oreilly.com/library/view/observability-engineering/9781492076438/
   

   View Slide

8. Observability - Goal
   ✅ Enable new members
   
   
   to understand system states on Day 1.
   

   View Slide

9. Agenda
   ɾArchitecture
   
   
   ɾApproaches
   
   
   - Logs
   
   
   - Traces
   
   
   - Metrics
   
   
   - SLI / SLO
   

   View Slide

10. Logs
    • Request logs
    
    
    • Container logs
    Cloud Run generates two types of logs:
    https://cloud.google.com/run/docs/logging
    

    View Slide

11. Logs
    • Request logs
    
    
    • Container logs
    Cloud Run generates two types of logs:
    https://cloud.google.com/run/docs/logging
    

    View Slide

12. Logs
    Cloud Run generates Request Logs
    

    View Slide

13. Logs
    • Request logs
    
    
    • Container logs
    Cloud Run generates two types of logs:
    https://cloud.google.com/run/docs/logging
    

    View Slide

14. Logs
    Container (Application) Logs
    Structured Log
    

    View Slide

15. Logs
    Request Logs
    
    
    +
    
    
    Container Logs
    Correlate Logs
    https://cloud.google.com/run/docs/logging#correlate-logs
    

    View Slide

16. Logs
    Request Logs
    Container Logs
    Correlate Logs
    

    View Slide

17. Logs
    Correlate Logs
    

    View Slide

18. Logs
    {
    "message": "grpc request",
    "logger": "grpc.request_logger",
    "method": "/customer.v1.CustomerService/GetXXX",
    "level": "info",
    "timestamp": 1613885945098.689
    "logging.googleapis.com/trace": "projects/.../traces/xxx",
    }
    https://cloud.google.com/logging/docs/structured-logging
    Container (Application) Logs
    

    View Slide

19. Agenda
    ɾArchitecture
    
    
    ɾApproaches
    
    
    - Logs
    
    
    - Traces
    
    
    - Metrics
    
    
    - SLI / SLO
    

    View Slide

20. Cloud Trace
    

    View Slide

21. OpenTelemetry
    OpenTelemetry is a collection of tools,
    APIs, and SDKs. Use it to instrument,
    generate, collect, and export telemetry
    data (metrics, logs, and traces) to help
    you analyze your software’s
    performance and behavior.
    https://opentelemetry.io/
    

    View Slide

22. Trace
    Just tracing is not enough...
    

    View Slide

23. Trace
    Attributes
    

    View Slide

24. Trace
    Events
    

    View Slide

25. Trace
    Correlate Logs
    

    View Slide

26. Agenda
    ɾArchitecture
    
    
    ɾApproaches
    
    
    - Logs
    
    
    - Traces
    
    
    - Metrics
    
    
    - SLI / SLO
    

    View Slide

27. Metrics
    Cloud Run Logging Monitoring
    Metrics
    Log
    Log Based Metrics
    

    View Slide

28. Metrics
    Log Based Metrics
    

    View Slide

29. Metrics
    Cloud Run OpenTelemetry Monitoring
    Metrics
    Metrics
    OpenTelemetry
    

    View Slide

30. Agenda
    ɾArchitecture
    
    
    ɾApproaches
    
    
    - Log
    
    
    - Trace
    
    
    - Metrics
    
    
    - SLI / SLO
    

    View Slide

31. Cloud Monitoring
    

    View Slide

32. Cloud Monitoring
    PromQL
    

    View Slide

33. Agenda
    ɾArchitecture
    
    
    ɾApproaches
    
    
    - Log
    
    
    - Trace
    
    
    - Metrics
    
    
    - SLI / SLO
    

    View Slide

34. View Slide