Slide 1
Slide 1 text
1
Cloud Run Reliability/Observability
at ソウゾウ
Ryuzo Yamamoto
Cloud Run Casual Talk！#2
Slide 2
Slide 2 text
2
山本 竜三
自己紹介
@dragon3
Software Engineer
Lead Architect / SRE at Souzoh
in Fukuoka
Slide 3
Slide 3 text
3
ソウゾウ / メルカリShops
Slide 4
Slide 4 text
4
● Architecture, Tech Stack
● Observability
○ Logs, Metrics, Traces
● Reliability
○ SLOs & Monitors as Code
Agenda
Slide 5
Slide 5 text
5
Architecture
Next.js
Cloud Run
GraphQL
Cloud Run
imgproxy
Cloud Run
microservice
Cloud Run
microservice
Cloud Run
Cloud
Storage
Cloud Load
Balancing
Cloud SQL
Memorystore
Cloud Run
(70~ services)
microservice(s)
Cloud Run
Slide 6
Slide 6 text
6
Tech Stack
● Monorepo
○ Go, TypeScript, Python, Java
○ 70~ microservices
● Bazel, Turborepo
● GraphQL / gRPC
● Serverless (Cloud Run)
● PostgreSQL, Redis
● Cloud PubSub, Tasks, Workflows, Scheduler, VertexAI
Slide 7
Slide 7 text
7
● Architecture, Tech Stack
● Observability
○ Logs, Metrics, Traces
● Reliability
○ SLOs & Monitors as Code
Agenda
Slide 8
Slide 8 text
8
● Logs
○ JSON structured logging
○ Cloud Logging -> BigQuery
● Metrics
○ Log-based Metrics
○ Cloud Monitoring -> Datadog
● Traces
○ OpenTelemetry -> Datadog
Observability
Slide 9
Slide 9 text
9
Observability - Logs
microservice
Cloud Run
container
log
STDOUT / STDERR
Logging BigQuery
{
"message": "failed to say hello",
"something_id": "xxxxxxxx"
"serviceContext": {
"version": "1.0.1",
"service": "echo"
},
"metadata": {
"user-agent": "graphql/1.0.0 grpc-node-js/1.7.3",
}
}
Sink
Slide 10
Slide 10 text
10
Observability - Metrics
microservice
Cloud Run
container
log
STDOUT / STDERR
Logging
{
"message": "grpc: finished server unary /echo.EchoService/Hello",
"grpc": {
"type": "unary",
"kind": "server",
"latency": 0.002360152,
"code": "OK",
"method": "Hello",
"service": "echo.EchoService"
},
"serviceContext": {
"version": "1.0.1",
"service": "echo"
},
"metadata": {
"user-agent": "graphql/1.0.0 grpc-node-js/1.7.3",
}
}
Log-based
Metrics
Monitoring
Log-based Metrics
+ Other GCP Metrics
gRPC interceptor
Slide 11
Slide 11 text
11
Observability - Traces
Next.js
Cloud Run
GraphQL
Cloud Run
microservice
Cloud Run
microservice
Cloud Run
microservice(s)
Cloud Run
datadog-agent
Cloud Run
OTLP
(gRPC)
Slide 12
Slide 12 text
12
● Architecture, Tech Stack
● Observability
○ Logs, Metrics, Traces
● Reliability
○ SLOs & Monitors as Code
Agenda
Slide 13
Slide 13 text
13
Reliability - SLOs & Monitors as Code
● SLO をすべての gRPC method 毎に設定
○ Availability (e.g. 99.9% / 30 days)
○ Latency (e.g. p95 100ms)
● 設定の自動化
○ protobuf plugin + Terraform module
● Multiwindow, Multi-Burn-Rate Alerts
○ https://docs.datadoghq.com/monitors/service_level_objectives/burn_rate/
○ https://sre.google/workbook/alerting-on-slos/#6-multiwindow-multi-burn-rate-alerts
Slide 14
Slide 14 text
14
Reliability - SLOs & Monitors as Code
...
rpc Hello(HelloRequest) returns (HelloResponse) {
option (extension.v2.method_monitoring) = {
availability: {
goal: 99.5
}
latency: {
threshold_ms: 100
percentile: 95
}
};
}
...
Terraform
configuration
protoc
apply by CI
(GitHub Actions)
SLO monitors
Slide 15
Slide 15 text
15
● Architecture, Tech Stack
● Observability
○ Logs, Metrics, Traces
● Reliability
○ SLOs & Monitors as Code
Wrap Up