Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Run + Observability / Reliability @ KAUCHE

Yuki Ito
April 19, 2023

Cloud Run + Observability / Reliability @ KAUCHE

Yuki Ito

April 19, 2023
Tweet

More Decks by Yuki Ito

Other Decks in Technology

Transcript

  1. Cloud Run + Observability / Reliability


    @ KAUCHE
    Yuki Ito (@mrno110)
    Cloud Run Casual Talk #2

    View Slide

  2. KAUCHE


    Architect / Platform Team
    Yuki Ito


    @mrno110

    View Slide

  3. View Slide

  4. Agenda
    ɾArchitecture


    ɾApproaches


    - Logs


    - Traces


    - Metrics


    - SLI / SLO

    View Slide

  5. Architecture
    Run
    Tasks
    Pub/Sub
    Mobile App External Service
    Mobile API Web Hook API Job API
    Scheduler

    View Slide

  6. Architecture

    View Slide

  7. Observability
    our de
    fi
    nition of “observability” for
    software systems is a measure of how
    well you can understand and explain
    any state your system can get into, no
    matter how novel or bizarre.


    ...


    If you can understand any bizarre or
    novel state without needing to ship
    new code, you have observability.
    https://www.oreilly.com/library/view/observability-engineering/9781492076438/

    View Slide

  8. Observability - Goal
    ✅ Enable new members


    to understand system states on Day 1.

    View Slide

  9. Agenda
    ɾArchitecture


    ɾApproaches


    - Logs


    - Traces


    - Metrics


    - SLI / SLO

    View Slide

  10. Logs
    • Request logs


    • Container logs
    Cloud Run generates two types of logs:
    https://cloud.google.com/run/docs/logging

    View Slide

  11. Logs
    • Request logs


    • Container logs
    Cloud Run generates two types of logs:
    https://cloud.google.com/run/docs/logging

    View Slide

  12. Logs
    Cloud Run generates Request Logs

    View Slide

  13. Logs
    • Request logs


    • Container logs
    Cloud Run generates two types of logs:
    https://cloud.google.com/run/docs/logging

    View Slide

  14. Logs
    Container (Application) Logs
    Structured Log

    View Slide

  15. Logs
    Request Logs


    +


    Container Logs
    Correlate Logs
    https://cloud.google.com/run/docs/logging#correlate-logs

    View Slide

  16. Logs
    Request Logs
    Container Logs
    Correlate Logs

    View Slide

  17. Logs
    Correlate Logs

    View Slide

  18. Logs
    {
    "message": "grpc request",
    "logger": "grpc.request_logger",
    "method": "/customer.v1.CustomerService/GetXXX",
    "level": "info",
    "timestamp": 1613885945098.689
    "logging.googleapis.com/trace": "projects/.../traces/xxx",
    }
    https://cloud.google.com/logging/docs/structured-logging
    Container (Application) Logs

    View Slide

  19. Agenda
    ɾArchitecture


    ɾApproaches


    - Logs


    - Traces


    - Metrics


    - SLI / SLO

    View Slide

  20. Cloud Trace

    View Slide

  21. OpenTelemetry
    OpenTelemetry is a collection of tools,
    APIs, and SDKs. Use it to instrument,
    generate, collect, and export telemetry
    data (metrics, logs, and traces) to help
    you analyze your software’s
    performance and behavior.
    https://opentelemetry.io/

    View Slide

  22. Trace
    Just tracing is not enough...

    View Slide

  23. Trace
    Attributes

    View Slide

  24. Trace
    Events

    View Slide

  25. Trace
    Correlate Logs

    View Slide

  26. Agenda
    ɾArchitecture


    ɾApproaches


    - Logs


    - Traces


    - Metrics


    - SLI / SLO

    View Slide

  27. Metrics
    Cloud Run Logging Monitoring
    Metrics
    Log
    Log Based Metrics

    View Slide

  28. Metrics
    Log Based Metrics

    View Slide

  29. Metrics
    Cloud Run OpenTelemetry Monitoring
    Metrics
    Metrics
    OpenTelemetry

    View Slide

  30. Agenda
    ɾArchitecture


    ɾApproaches


    - Log


    - Trace


    - Metrics


    - SLI / SLO

    View Slide

  31. Cloud Monitoring

    View Slide

  32. Cloud Monitoring
    PromQL

    View Slide

  33. Agenda
    ɾArchitecture


    ɾApproaches


    - Log


    - Trace


    - Metrics


    - SLI / SLO

    View Slide

  34. View Slide