Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Run + Observability / Reliability @ KAUCHE

Yuki Ito
April 19, 2023

Cloud Run + Observability / Reliability @ KAUCHE

Yuki Ito

April 19, 2023
Tweet

More Decks by Yuki Ito

Other Decks in Technology

Transcript

  1. Cloud Run + Observability / Reliability


    @ KAUCHE
    Yuki Ito (@mrno110)
    Cloud Run Casual Talk #2

    View full-size slide

  2. KAUCHE


    Architect / Platform Team
    Yuki Ito


    @mrno110

    View full-size slide

  3. Agenda
    ɾArchitecture


    ɾApproaches


    - Logs


    - Traces


    - Metrics


    - SLI / SLO

    View full-size slide

  4. Architecture
    Run
    Tasks
    Pub/Sub
    Mobile App External Service
    Mobile API Web Hook API Job API
    Scheduler

    View full-size slide

  5. Architecture

    View full-size slide

  6. Observability
    our de
    fi
    nition of “observability” for
    software systems is a measure of how
    well you can understand and explain
    any state your system can get into, no
    matter how novel or bizarre.


    ...


    If you can understand any bizarre or
    novel state without needing to ship
    new code, you have observability.
    https://www.oreilly.com/library/view/observability-engineering/9781492076438/

    View full-size slide

  7. Observability - Goal
    ✅ Enable new members


    to understand system states on Day 1.

    View full-size slide

  8. Agenda
    ɾArchitecture


    ɾApproaches


    - Logs


    - Traces


    - Metrics


    - SLI / SLO

    View full-size slide

  9. Logs
    • Request logs


    • Container logs
    Cloud Run generates two types of logs:
    https://cloud.google.com/run/docs/logging

    View full-size slide

  10. Logs
    • Request logs


    • Container logs
    Cloud Run generates two types of logs:
    https://cloud.google.com/run/docs/logging

    View full-size slide

  11. Logs
    Cloud Run generates Request Logs

    View full-size slide

  12. Logs
    • Request logs


    • Container logs
    Cloud Run generates two types of logs:
    https://cloud.google.com/run/docs/logging

    View full-size slide

  13. Logs
    Container (Application) Logs
    Structured Log

    View full-size slide

  14. Logs
    Request Logs


    +


    Container Logs
    Correlate Logs
    https://cloud.google.com/run/docs/logging#correlate-logs

    View full-size slide

  15. Logs
    Request Logs
    Container Logs
    Correlate Logs

    View full-size slide

  16. Logs
    Correlate Logs

    View full-size slide

  17. Logs
    {
    "message": "grpc request",
    "logger": "grpc.request_logger",
    "method": "/customer.v1.CustomerService/GetXXX",
    "level": "info",
    "timestamp": 1613885945098.689
    "logging.googleapis.com/trace": "projects/.../traces/xxx",
    }
    https://cloud.google.com/logging/docs/structured-logging
    Container (Application) Logs

    View full-size slide

  18. Agenda
    ɾArchitecture


    ɾApproaches


    - Logs


    - Traces


    - Metrics


    - SLI / SLO

    View full-size slide

  19. OpenTelemetry
    OpenTelemetry is a collection of tools,
    APIs, and SDKs. Use it to instrument,
    generate, collect, and export telemetry
    data (metrics, logs, and traces) to help
    you analyze your software’s
    performance and behavior.
    https://opentelemetry.io/

    View full-size slide

  20. Trace
    Just tracing is not enough...

    View full-size slide

  21. Trace
    Attributes

    View full-size slide

  22. Trace
    Correlate Logs

    View full-size slide

  23. Agenda
    ɾArchitecture


    ɾApproaches


    - Logs


    - Traces


    - Metrics


    - SLI / SLO

    View full-size slide

  24. Metrics
    Cloud Run Logging Monitoring
    Metrics
    Log
    Log Based Metrics

    View full-size slide

  25. Metrics
    Log Based Metrics

    View full-size slide

  26. Metrics
    Cloud Run OpenTelemetry Monitoring
    Metrics
    Metrics
    OpenTelemetry

    View full-size slide

  27. Agenda
    ɾArchitecture


    ɾApproaches


    - Log


    - Trace


    - Metrics


    - SLI / SLO

    View full-size slide

  28. Cloud Monitoring

    View full-size slide

  29. Cloud Monitoring
    PromQL

    View full-size slide

  30. Agenda
    ɾArchitecture


    ɾApproaches


    - Log


    - Trace


    - Metrics


    - SLI / SLO

    View full-size slide