Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RPC Metrics at Google

E7526ec3e801f8ba99f6746498a154a6?s=47 JBD
August 09, 2018
Programming
1
190

RPC Metrics at Google

E7526ec3e801f8ba99f6746498a154a6?s=128

JBD

August 09, 2018
Tweet

More Decks by JBD

See All by JBD
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
5
1.2k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
7
1.5k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
2
1.1k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
120
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
110
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
2
870
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
3.6k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
100
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
2
1.5k

Other Decks in Programming

See All in Programming
51105482dc2e3d01b3ba64c9751d75c6?s=48 showwin
0
140
0ed10d1154c696886ca483fe827cb299?s=48 thatjeffsmith
0
490
2a67fe83a37381d8b464eff2837d3f1d?s=48 yagitatsu
3
1.7k
347a1b64f3a6e38981bc99b53919e2b1?s=48 azdaroth
0
150
1b838da2065660793d5b26f2cdc32de7?s=48 pyama86
2
270
7c9b8b368924556d8642bdaed3ded1f5?s=48 danilop
0
200
15934fa2aa7b2ce21f091e9b7cffa856?s=48 manfredsteyer
PRO
0
110
Af9618883e81d7685d2b34e14f85ae1d?s=48 466548
0
160
2102a6b8760bd6f57f672805723dd83a?s=48 line_developers_tw
0
590
4047c64e3a1e2f81addd4ba675ddc451?s=48 zsmb
2
110
3f309c992e2b1a5c3c014e63810a2f68?s=48 viteinfinite
0
210
2102a6b8760bd6f57f672805723dd83a?s=48 line_developers_tw
0
580

Featured

See All Featured
A0517b08532aec8ab2aae3f65d40ad86?s=48 hannesfritz
27
930
1f74b13f1e5c6c69cb5d7fbaabb1e2cb?s=48 sferik
610
54k
2f5463832ccb768ccb4a1ca3607c27ef?s=48 jacobian
255
20k
5f83ef806155b403c3393160ce51f955?s=48 jlugia
216
16k
F57e2136eb9895eb95ff5dc8a1c02677?s=48 zakiwarfel
88
3.3k
39488f9d172ab92fd352f2cd7b73258d?s=48 mza
80
4.1k
E4f5d494ebdc9e7cce1aecf3ce3e8bc1?s=48 shpigford
368
42k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
27
1.9k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
168
7.2k
245cee81a9c424266e5e401d844ea881?s=48 lara
16
2.6k
4262b9cfacfb6b2c77f23e1d0310cd3e?s=48 myddelton
109
11k
C0390e8b11079f8761c0cd3274ed61cb?s=48 productmarketing
5
660

Transcript

  1. RPC Metrics at Google JBD, Google (@rakyll)

  2. gRPC Metrics at Google JBD, Google (@rakyll)

  3. Request Metrics at Google JBD, Google (@rakyll)

  4. @rakyll "100% is the wrong reliability target for basically everything."

    -- Benjamin Treynor Sloss, VP of Engineering, Google

  5. @rakyll "A service is available if users cannot tell that

    there was an outage."

  6. @rakyll Principled way of saying what level of downtime is

    acceptable. • Error rate • Latency expectations SLOs

  7. @rakyll Analytics frontend server Authentication Reporting Users ... Spanner Blob

    Store

  8. @rakyll Questions infra teams want to ask: • Are we

    meeting the SLO for the other team? • What’s the impact of a product on infra? • How much do we need to scale up if product grows 10%?

  9. @rakyll High-Cardinality Breaking down the metrics data...

  10. @rakyll Query the collected data in various ways: • Latency

    distribution for RPCs originated at Google Analytics. • Requests take took more than 100ms for the customer #123. • Compare the request latency initiated at web vs mobile frontend.

  11. @rakyll Analytics frontend server Authentication Reporting Users ... Spanner Blob

    Store originator=analytics; ...

  12. @rakyll Blob store read errors by originator

  13. @rakyll Dynamically choose aggregation (split between recording and aggregation)

  14. @rakyll Exemplars

  15. @rakyll /rpz and /statz

  16. @rakyll http://server:7777/debug/rpcz

  17. @rakyll Export? Monarch, Prometheus, and more.

  18. @rakyll import “cloud.google.com/go/pubsub”

  19. @rakyll +

  20. Thank you! JBD, Google jbd@google.com @rakyll