Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Run + Observability

Yuki Ito
February 26, 2021

Cloud Run + Observability

Yuki Ito

February 26, 2021
Tweet

More Decks by Yuki Ito

Other Decks in Technology

Transcript

  1. What is Cloud Run Cloud Run is a managed compute

    platform that enables you to run stateless containers that are invocable via web requests... Cloud Run is serverless: it abstracts away all infrastructure management... https://cloud.google.com/run/docs
  2. Architecture ✅ Using same API interceptors ✅ Managed by API

    De fi nitions ✅ Using same Monitoring environments
  3. Observability Put simply, our de fi nition of observability for

    software systems is a measure of how well you can understand and explain any state your system can get into, no matter how novel or bizarre. ... If you can understand that bizarre or novel state without shipping new code, then you have observability. https://www.oreilly.com/library/view/observability-engineering/9781492076438/
  4. Observability With observability tools, the best debugger on the team

    is typically the engineer who is most curious. https://www.oreilly.com/library/view/observability-engineering/9781492076438/
  5. Observability The best debugger is... Without Observability the person who

    has been there the longest With Observability the person who is most curious the issue
  6. Logging Structured logging In Cloud Logging, structured logs refer to

    log entries that use the jsonPayload fi eld to add structure to their payloads. https://cloud.google.com/logging/docs/structured-logging
  7. Logging Container (Application) logs { "message": "grpc request" , "logger":

    "grpc.request_logger" , "method": "/customer.v1.CustomerService/GetXXX" , "level": "info" , "timestamp": 1613885945098.689 } stdout
  8. Logging Let's talk about logging https://dave.cheney.net/2015/11/05/lets-talk-about-logging I believe that there

    are only two things you should log: 1. Things that developers care about when they are developing or debugging software. 2. Things that users care about when using your software. Obviously these are debug and info levels, respectively.
  9. Logging go-logr / logr https://github.com/go-logr/logr type Logger interface { Info(msg

    string, keysAndValues ...interface{} ) Error(err error, msg string, keysAndValues ...interface{} ) //... }
  10. Logging go-logr / logr https://github.com/go-logr/logr type Logger interface { Info(msg

    string, keysAndValues ...interface{} ) Error(err error, msg string, keysAndValues ...interface{} ) //... }
  11. Logging err := xxxRepository.Get(ctx, xxx ) if err != nil

    { return "", fmt.Errorf("failed to get xxx: %w", err ) } Wrapping Errors (Go 1.13 ~) https://blog.golang.org/go1.13-errors error: failed to vereify access token: auth-client: failed to verify token: failed to get xxx: failed to access to the database due to ...
  12. Logging Container (Application) logs { "message": "grpc request" , "logger":

    "grpc.request_logger" , "method": "/customer.v1.CustomerService/GetXXX" , "level": "info" , "timestamp": 1613885945098.68 9 "logging.googleapis.com/trace": "projects/.../traces/xxx" , } https://cloud.google.com/logging/docs/structured-logging
  13. Trace OpenTelemetry OpenTelemetry is a collection of tools, APIs, and

    SDKs. You use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis in order to understand your software's performance and behavior. https://opentelemetry.io/
  14. Trace googleapis / google-cloud-go package trac e import ( "go.opencensus.io/trace"

    // .. . ) func StartSpan(ctx context.Context, name string) context.Context { ctx, _ = trace.StartSpan(ctx, name ) return ct x }
  15. Trace open-telemetry / opentelemetry-go GoogleCloudPlatform / opentelemetry-operations-go (Cloud Trace Exporter)

    + https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/main/exporter/trace/README.md
  16. Monitoring Dashboard Monitoring Query Language https://cloud.google.com/blog/products/management-tools/introducing-monitoring-query-language-or-mql MQL represents a decade

    of learnings and improvements on Google’s internal metric query language. The same language that powers advanced querying for internal Google production users, is now available to Google Cloud users as well.
  17. Monitoring Dashboard fetch cloud_run_revisio n | { t_error : metric

    'logging.googleapis.com/user/CloudRunCustomerAPIErrorLogs ' | align delta( ) ; t_all : metric 'logging.googleapis.com/user/CloudRunCustomerAPIAllLogs ' | align delta() } | outer_join [0 ] | valu e [c'int lit *' : if(t_all.value.CloudRunCustomerAPIAllLogs == 0, 100.0 , (100 * (1 - (t_error.value.CloudRunCustomerAPIErrorLog s / t_all.value.CloudRunCustomerAPIAllLogs))))] Monitoring Query Language Reference: https://cloud.google.com/monitoring/mql/reference
  18. Monitoring Dashboard fetch cloud_run_revisio n | { t_error : metric

    'logging.googleapis.com/user/CloudRunCustomerAPIErrorLogs ' | align delta( ) ; t_all : metric 'logging.googleapis.com/user/CloudRunCustomerAPIAllLogs ' | align delta() } | outer_join [0 ] | valu e [c'int lit *' : if(t_all.value.CloudRunCustomerAPIAllLogs == 0, 100.0 , (100 * (1 - (t_error.value.CloudRunCustomerAPIErrorLog s / t_all.value.CloudRunCustomerAPIAllLogs))))] Monitoring Query Language Log-based metrics Reference: https://cloud.google.com/monitoring/mql/reference
  19. Architecture ✅ Using same API interceptors ✅ Managed by API

    De fi nitions ✅ Using same Monitoring environments