Slide 1
Slide 1 text
Cloud Run
+
Observability
GCPUG Tokyo
Observability February 2021
Yuki Ito
Slide 2
Slide 2 text
Merpay / Mercari
Architect Team
Microservices Platform
Yuki Ito
X Asia
Kauche
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
Agenda
• Architecture
• Observability
• Goal
• Approaches
•Logging
•Trace
•Monitoring Dashboard
Slide 5
Slide 5 text
Agenda
• Architecture
• Observability
• Goal
• Approaches
•Logging
•Trace
•Monitoring Dashboard
Slide 6
Slide 6 text
What is Cloud Run
Cloud Run is a managed compute platform
that enables you to run stateless
containers that are invocable via web
requests...
Cloud Run is serverless: it abstracts away all
infrastructure management...
https://cloud.google.com/run/docs
Slide 7
Slide 7 text
What is Cloud Run
on GKE (Anthos)
+
Fully Managed
Slide 8
Slide 8 text
What is Cloud Run
on GKE (Anthos)
+
Fully Managed
Slide 9
Slide 9 text
Architecture
Run
Scheduler
Pub/Sub
Mobile App External Service
Web Hook API
Customer API Job API
Slide 10
Slide 10 text
Key Concepts
• Everything runs on Run
• Everything is API
Slide 11
Slide 11 text
Key Concepts
Run
e.g.) vs Cloud Functions Trigger
Pub/Sub Functions
Run Firestore Functions
Slide 12
Slide 12 text
Architecture
Run
Scheduler
Pub/Sub
Mobile App External Service
Web Hook API
Customer API Job API
Slide 13
Slide 13 text
Architecture
✅ Using same API interceptors
✅ Managed by API De
fi
nitions
✅ Using same Monitoring environments
Slide 14
Slide 14 text
Architecture
࣮ફ Cloud Run
https://speakerdeck.com/110y/introduce-cloud-run
https://www.youtube.com/watch?v=s_Y3dsUrux4
Slide 15
Slide 15 text
Agenda
• Architecture
• Observability
• Goal
• Approaches
•Logging
•Trace
•Monitoring Dashboard
Slide 16
Slide 16 text
Observability
What is
Observability?
🤔
Slide 17
Slide 17 text
Observability
https://www.oreilly.com/library/view/observability-engineering/9781492076438/
Slide 18
Slide 18 text
Observability
Put simply, our de
fi
nition of observability
for software systems is a measure of how
well you can understand and explain any
state your system can get into, no matter
how novel or bizarre.
...
If you can understand that bizarre or
novel state without shipping new code,
then you have observability.
https://www.oreilly.com/library/view/observability-engineering/9781492076438/
Slide 19
Slide 19 text
Observability
With observability tools,
the best debugger on the
team is typically the
engineer who is most
curious.
https://www.oreilly.com/library/view/observability-engineering/9781492076438/
Slide 20
Slide 20 text
Observability
The best debugger is...
Without Observability
the person who has been there the longest
With Observability
the person who is most curious the issue
Slide 21
Slide 21 text
Goal
✅ Can understand unknown state
without shipping new code
✅ Democratizing
Slide 22
Slide 22 text
Ultimate Goal
✅ Making new comers be able to
understand system states at Day 1.
Slide 23
Slide 23 text
Approaches
• Logging
• Trace
• Monitoring Dashboard
Slide 24
Slide 24 text
Approaches
• Logging
• Trace
• Monitoring Dashboard
Slide 25
Slide 25 text
Logging
• Request logs
• Container logs
https://cloud.google.com/run/docs/logging
Cloud Run has two types of logs:
Slide 26
Slide 26 text
Logging
• Request logs
• Container logs
https://cloud.google.com/run/docs/logging
Cloud Run has two types of logs:
automatically sent to Cloud Logging
Slide 27
Slide 27 text
Logging
• Request logs
• Container logs
https://cloud.google.com/run/docs/logging
Cloud Run has two types of logs:
automatically sent to Cloud Logging
Slide 28
Slide 28 text
Logging
Cloud Run generates Request Logs
Slide 29
Slide 29 text
Logging
Cloud Run generates Request Logs
Not enough...
Slide 30
Slide 30 text
Logging
• Request logs
• Container logs
https://cloud.google.com/run/docs/logging
Cloud Run has two types of logs:
automatically sent to Cloud Logging
Slide 31
Slide 31 text
Logging
Container (Application) logs
Slide 32
Slide 32 text
Logging
Container (Application) logs
Structured Log
Slide 33
Slide 33 text
Logging
Structured logging
In Cloud Logging,
structured logs refer to log
entries that use the
jsonPayload
fi
eld to add
structure to their payloads.
https://cloud.google.com/logging/docs/structured-logging
Slide 34
Slide 34 text
Logging
Container (Application) logs
{
"message": "grpc request"
,
"logger": "grpc.request_logger"
,
"method": "/customer.v1.CustomerService/GetXXX"
,
"level": "info"
,
"timestamp": 1613885945098.689
}
stdout
Slide 35
Slide 35 text
Logging
Container (Application) logs
Structured Log
Slide 36
Slide 36 text
Logging
Structured logging
go-logr / logr
https://github.com/go-logr/logr
Slide 37
Slide 37 text
Logging
Let's talk about logging
https://dave.cheney.net/2015/11/05/lets-talk-about-logging
I believe that there are only two things you should log:
1. Things that developers care about when they are developing or
debugging software.
2. Things that users care about when using your software.
Obviously these are debug and info levels, respectively.
Slide 38
Slide 38 text
Logging
go-logr / logr
https://github.com/go-logr/logr
type Logger interface
{
Info(msg string, keysAndValues ...interface{}
)
Error(err error, msg string, keysAndValues ...interface{}
)
//...
}
Slide 39
Slide 39 text
Logging
go-logr / logr
https://github.com/go-logr/logr
type Logger interface
{
Info(msg string, keysAndValues ...interface{}
)
Error(err error, msg string, keysAndValues ...interface{}
)
//...
}
Slide 40
Slide 40 text
Logging
err := xxxRepository.Get(ctx, xxx
)
if err != nil
{
return "", fmt.Errorf("failed to get xxx: %w", err
)
}
Wrapping Errors (Go 1.13 ~)
https://blog.golang.org/go1.13-errors
error: failed to vereify access token: auth-client:
failed to verify token: failed to get xxx: failed to
access to the database due to ...
Slide 41
Slide 41 text
AD
https://gihyo.jp/magazine/SD/archive/2021/202101
Slide 42
Slide 42 text
Logging
go-logr / logr
https://github.com/go-logr/logr
Slide 43
Slide 43 text
Logging
go-logr / logr
https://github.com/go-logr/logr#why-structured-logging
Slide 44
Slide 44 text
Logging
Container (Application) logs
Structured Log
Slide 45
Slide 45 text
Logging
Request Logs
+
Container Logs
https://cloud.google.com/run/docs/logging#correlate-logs
Correlating
Slide 46
Slide 46 text
Logging
Correlating
Slide 47
Slide 47 text
Logging
Correlating
Request Logs
Container Logs
Slide 48
Slide 48 text
Logging
Correlating
Slide 49
Slide 49 text
Logging
Container (Application) logs
{
"message": "grpc request"
,
"logger": "grpc.request_logger"
,
"method": "/customer.v1.CustomerService/GetXXX"
,
"level": "info"
,
"timestamp": 1613885945098.68
9
"logging.googleapis.com/trace": "projects/.../traces/xxx"
,
}
https://cloud.google.com/logging/docs/structured-logging
Slide 50
Slide 50 text
Logging
Container
X-Cloud-Trace-Context: projects/.../traces/xxx
Header
Slide 51
Slide 51 text
Logging
Correlating
Slide 52
Slide 52 text
Logging
Correlating
Request Logs
Container Logs
Slide 53
Slide 53 text
Approaches
• Logging
• Trace
• Monitoring Dashboard
Slide 54
Slide 54 text
Trace
Cloud Trace
Slide 55
Slide 55 text
Trace
OpenTelemetry
OpenTelemetry is a collection of tools, APIs, and
SDKs. You use it to instrument, generate, collect, and
export telemetry data (metrics, logs, and traces) for
analysis in order to understand your software's
performance and behavior.
https://opentelemetry.io/
Slide 56
Slide 56 text
Trace
https://medium.com/opentelemetry/opentelemetry-speci
fi
cation-v1-0-0-tracing-edition-72dd08936978
🎉
OpenTelemetry Speci
fi
cation
v1.0.0, Tracing Edition
Slide 57
Slide 57 text
Trace
open-telemetry / opentelemetry-go
Slide 58
Slide 58 text
Trace
open-telemetry / opentelemetry-go
Slide 59
Slide 59 text
Trace
googleapis / google-cloud-go
package trac
e
import
(
"go.opencensus.io/trace"
// ..
.
)
func StartSpan(ctx context.Context, name string) context.Context
{
ctx, _ = trace.StartSpan(ctx, name
)
return ct
x
}
Slide 60
Slide 60 text
Trace
open-telemetry / opentelemetry-go
Slide 61
Slide 61 text
Trace
OpenTelemetry
Slide 62
Slide 62 text
Trace
OpenCensus
Slide 63
Slide 63 text
Trace
OpenTelemetry + OpenCensus (Bridge)
Slide 64
Slide 64 text
Trace
open-telemetry / opentelemetry-go
GoogleCloudPlatform / opentelemetry-operations-go
(Cloud Trace Exporter)
+
https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/main/exporter/trace/README.md
Slide 65
Slide 65 text
Trace
Slide 66
Slide 66 text
Trace
Just tracing is not enough...
Slide 67
Slide 67 text
Trace
Attributes
Slide 68
Slide 68 text
Trace
Attributes
Slide 69
Slide 69 text
Trace
Attributes
Slide 70
Slide 70 text
Trace
Events
Slide 71
Slide 71 text
Trace
Events
Slide 72
Slide 72 text
Trace
Events
Slide 73
Slide 73 text
Trace
Events
Slide 74
Slide 74 text
Approaches
• Logging
• Trace
• Monitoring Dashboard
Slide 75
Slide 75 text
Monitoring Dashboard
Cloud Monitoring
Slide 76
Slide 76 text
Monitoring Dashboard
Cloud Monitoring
https://cloud.google.com/blog/products/management-tools/cloud-monitoring-improves-custom-dashboard-creation
Slide 77
Slide 77 text
Monitoring Dashboard
Slide 78
Slide 78 text
Monitoring Dashboard
Markdown Text
Slide 79
Slide 79 text
Monitoring Dashboard
Slide 80
Slide 80 text
Monitoring Dashboard
Slide 81
Slide 81 text
Monitoring Dashboard
Cloud Logging
https://cloud.google.com/logging/docs/release-notes#January_14_2021
Slide 82
Slide 82 text
Monitoring Dashboard
Slide 83
Slide 83 text
Monitoring Dashboard
SLOs
Slide 84
Slide 84 text
Monitoring Dashboard
Cloud Run Logging Monitoring
Metrics
Log
Slide 85
Slide 85 text
Monitoring Dashboard
Log Based Metrics
Slide 86
Slide 86 text
Monitoring Dashboard
SLO: Success Rate
Slide 87
Slide 87 text
Monitoring Dashboard
Slide 88
Slide 88 text
Monitoring Dashboard
MQL
Slide 89
Slide 89 text
Monitoring Dashboard
Monitoring Query Language
https://cloud.google.com/blog/products/management-tools/introducing-monitoring-query-language-or-mql
MQL represents a decade of learnings
and improvements on Google’s internal
metric query language. The same
language that powers advanced querying
for internal Google production users, is
now available to Google Cloud users as
well.
Slide 90
Slide 90 text
Monitoring Dashboard
SLO: Success Rate
Slide 91
Slide 91 text
Monitoring Dashboard
fetch cloud_run_revisio
n
| { t_error
:
metric 'logging.googleapis.com/user/CloudRunCustomerAPIErrorLogs
'
| align delta(
)
; t_all
:
metric 'logging.googleapis.com/user/CloudRunCustomerAPIAllLogs
'
| align delta()
}
| outer_join [0
]
| valu
e
[c'int lit *'
:
if(t_all.value.CloudRunCustomerAPIAllLogs == 0, 100.0
,
(100 * (1
-
(t_error.value.CloudRunCustomerAPIErrorLog
s
/ t_all.value.CloudRunCustomerAPIAllLogs))))]
Monitoring Query Language
Reference: https://cloud.google.com/monitoring/mql/reference
Slide 92
Slide 92 text
Monitoring Dashboard
fetch cloud_run_revisio
n
| { t_error
:
metric 'logging.googleapis.com/user/CloudRunCustomerAPIErrorLogs
'
| align delta(
)
; t_all
:
metric 'logging.googleapis.com/user/CloudRunCustomerAPIAllLogs
'
| align delta()
}
| outer_join [0
]
| valu
e
[c'int lit *'
:
if(t_all.value.CloudRunCustomerAPIAllLogs == 0, 100.0
,
(100 * (1
-
(t_error.value.CloudRunCustomerAPIErrorLog
s
/ t_all.value.CloudRunCustomerAPIAllLogs))))]
Monitoring Query Language
Log-based metrics
Reference: https://cloud.google.com/monitoring/mql/reference
Slide 93
Slide 93 text
Monitoring Dashboard
Cloud Run Logging Monitoring
Slack
Metrics
Log
Alert
👨💻
Dashboard
Slide 94
Slide 94 text
Approaches
• Logging
• Trace
• Monitoring Dashboard
Slide 95
Slide 95 text
Architecture
✅ Using same API interceptors
✅ Managed by API De
fi
nitions
✅ Using same Monitoring environments