Tweet
Tweet

Slide 1

Slide 1 text

1 Cloud Run Reliability/Observability at ソウゾウ Ryuzo Yamamoto Cloud Run Casual Talk！#2

Slide 2

Slide 2 text

2 山本 竜三 自己紹介 @dragon3 Software Engineer Lead Architect / SRE at Souzoh in Fukuoka

Slide 3

Slide 3 text

3 ソウゾウ / メルカリShops

Slide 4

Slide 4 text

4 ● Architecture, Tech Stack ● Observability ○ Logs, Metrics, Traces ● Reliability ○ SLOs & Monitors as Code Agenda

Slide 5

Slide 5 text

5 Architecture Next.js Cloud Run GraphQL Cloud Run imgproxy Cloud Run microservice Cloud Run microservice Cloud Run Cloud Storage Cloud Load Balancing Cloud SQL Memorystore Cloud Run (70~ services) microservice(s) Cloud Run

Slide 6

Slide 6 text

6 Tech Stack ● Monorepo ○ Go, TypeScript, Python, Java ○ 70~ microservices ● Bazel, Turborepo ● GraphQL / gRPC ● Serverless (Cloud Run) ● PostgreSQL, Redis ● Cloud PubSub, Tasks, Workflows, Scheduler, VertexAI

Slide 7

Slide 7 text

7 ● Architecture, Tech Stack ● Observability ○ Logs, Metrics, Traces ● Reliability ○ SLOs & Monitors as Code Agenda

Slide 8

Slide 8 text

8 ● Logs ○ JSON structured logging ○ Cloud Logging -> BigQuery ● Metrics ○ Log-based Metrics ○ Cloud Monitoring -> Datadog ● Traces ○ OpenTelemetry -> Datadog Observability

Slide 9

Slide 9 text

9 Observability - Logs microservice Cloud Run container log STDOUT / STDERR Logging BigQuery { "message": "failed to say hello", "something_id": "xxxxxxxx" "serviceContext": { "version": "1.0.1", "service": "echo" }, "metadata": { "user-agent": "graphql/1.0.0 grpc-node-js/1.7.3", } } Sink

Slide 10

Slide 10 text

10 Observability - Metrics microservice Cloud Run container log STDOUT / STDERR Logging { "message": "grpc: finished server unary /echo.EchoService/Hello", "grpc": { "type": "unary", "kind": "server", "latency": 0.002360152, "code": "OK", "method": "Hello", "service": "echo.EchoService" }, "serviceContext": { "version": "1.0.1", "service": "echo" }, "metadata": { "user-agent": "graphql/1.0.0 grpc-node-js/1.7.3", } } Log-based Metrics Monitoring Log-based Metrics + Other GCP Metrics gRPC interceptor

Slide 11

Slide 11 text

11 Observability - Traces Next.js Cloud Run GraphQL Cloud Run microservice Cloud Run microservice Cloud Run microservice(s) Cloud Run datadog-agent Cloud Run OTLP (gRPC)

Slide 12

Slide 12 text

12 ● Architecture, Tech Stack ● Observability ○ Logs, Metrics, Traces ● Reliability ○ SLOs & Monitors as Code Agenda

Slide 13

Slide 13 text

13 Reliability - SLOs & Monitors as Code ● SLO をすべての gRPC method 毎に設定 ○ Availability (e.g. 99.9% / 30 days) ○ Latency (e.g. p95 100ms) ● 設定の自動化 ○ protobuf plugin + Terraform module ● Multiwindow, Multi-Burn-Rate Alerts ○ https://docs.datadoghq.com/monitors/service_level_objectives/burn_rate/ ○ https://sre.google/workbook/alerting-on-slos/#6-multiwindow-multi-burn-rate-alerts

Slide 14

Slide 14 text

14 Reliability - SLOs & Monitors as Code ... rpc Hello(HelloRequest) returns (HelloResponse) { option (extension.v2.method_monitoring) = { availability: { goal: 99.5 } latency: { threshold_ms: 100 percentile: 95 } }; } ... Terraform configuration protoc apply by CI (GitHub Actions) SLO monitors

Slide 15

Slide 15 text

15 ● Architecture, Tech Stack ● Observability ○ Logs, Metrics, Traces ● Reliability ○ SLOs & Monitors as Code Wrap Up