Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RPC Metrics at Google

E7526ec3e801f8ba99f6746498a154a6?s=47 JBD
August 09, 2018
Programming
2
260

RPC Metrics at Google

E7526ec3e801f8ba99f6746498a154a6?s=128

JBD

August 09, 2018
Tweet

More Decks by JBD

See All by JBD
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
1
860
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
1
1.4k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
5
1.2k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
8
1.7k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
3
1.1k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
1
140
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
160
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
2
890
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
3.7k

Other Decks in Programming

See All in Programming
4fe56d2e6ecf4cbde3ddbf6ef54fc568?s=48 kairiyasumatsu
0
190
9f96c3fd49e6a294e521175f2ad80b71?s=48 usamomokawa
1
290
7eca3e06edc3c69004aaf57de51900c2?s=48 ajstarks
3
3.3k
D76231a2114896dfcc7b79ac69558b79?s=48 yosuke_furukawa
PRO
3
680
422d6810de47c88b0df5132771e67b81?s=48 michaeldyrynda
0
180
9ebd4f7e6dfec9dbaa4fe161022cd496?s=48 n0mimono
0
230
270b0c7883545117a9a618dc7ca7cc83?s=48 xrdnk
1
180
9242ec5423930ab9ab881ce07f4d4fcf?s=48 e99h2121
0
1.4k
B32c18d03687577950da29542d0a9180?s=48 kurumai
1
630
82d6167c4d14393c2e20b37a74b363c5?s=48 kasacchiful
0
200
29dd31db419a2e56cf6a8a11f9de98ee?s=48 masakazoo
2
470
Aedb2d72d275bbb27a7419e22ea6f70e?s=48 yutafujii0
1
310

Featured

See All Featured
Be9caeb9d4ef9944d151af909063ed6e?s=48 samlambert
238
10k
7653c7c449918ff6542880a8c284f5e7?s=48 jponch
106
5.3k
5b6a2bd86ef643786a381a212e0ef2f1?s=48 geoffreycrofte
41
1.4k
E4f5d494ebdc9e7cce1aecf3ce3e8bc1?s=48 shpigford
167
19k
7edec06b5435e0dac07a353b2cc26253?s=48 geeforr
335
29k
91cefdf141106fa10674aae74ce05890?s=48 yeseniaperezcruz
306
32k
C4de88ea47538e4594750e3861a341c8?s=48 brettharned
96
3.3k
465724d73fe3a92c0879fdfb43a3a6f3?s=48 philhawksworth
196
17k
32de0bd2ba869609d26fd052a4622778?s=48 cromwellryan
107
6.5k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
498
130k
Af6a12710770ad9a7cfd08c97efd40d3?s=48 pedronauck
654
120k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
56
5.6k

Transcript

  1. RPC Metrics at Google JBD, Google (@rakyll)

  2. gRPC Metrics at Google JBD, Google (@rakyll)

  3. Request Metrics at Google JBD, Google (@rakyll)

  4. @rakyll "100% is the wrong reliability target for basically everything."

    -- Benjamin Treynor Sloss, VP of Engineering, Google

  5. @rakyll "A service is available if users cannot tell that

    there was an outage."

  6. @rakyll Principled way of saying what level of downtime is

    acceptable. • Error rate • Latency expectations SLOs

  7. @rakyll Analytics frontend server Authentication Reporting Users ... Spanner Blob

    Store

  8. @rakyll Questions infra teams want to ask: • Are we

    meeting the SLO for the other team? • What’s the impact of a product on infra? • How much do we need to scale up if product grows 10%?

  9. @rakyll High-Cardinality Breaking down the metrics data...

  10. @rakyll Query the collected data in various ways: • Latency

    distribution for RPCs originated at Google Analytics. • Requests take took more than 100ms for the customer #123. • Compare the request latency initiated at web vs mobile frontend.

  11. @rakyll Analytics frontend server Authentication Reporting Users ... Spanner Blob

    Store originator=analytics; ...

  12. @rakyll Blob store read errors by originator

  13. @rakyll Dynamically choose aggregation (split between recording and aggregation)

  14. @rakyll Exemplars

  15. @rakyll /rpz and /statz

  16. @rakyll http://server:7777/debug/rpcz

  17. @rakyll Export? Monarch, Prometheus, and more.

  18. @rakyll import “cloud.google.com/go/pubsub”

  19. @rakyll +

  20. Thank you! JBD, Google jbd@google.com @rakyll