RPC Metrics at Google

E7526ec3e801f8ba99f6746498a154a6?s=47 JBD
August 09, 2018
Programming
1
180

RPC Metrics at Google

E7526ec3e801f8ba99f6746498a154a6?s=128

JBD

August 09, 2018
Tweet

More Decks by JBD

See All by JBD
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
5
1.2k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
7
1.4k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
2
1.1k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
95
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
110
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
2
850
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
3.5k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
96
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
2
1.4k

Other Decks in Programming

See All in Programming
35761e3936deba2f8189c2d20982c771?s=48 getify
1
430
3115a782126be714b5f94d24073c957d?s=48 oracle4engineer
0
250
7c79a6c043825af5c538fb996430607c?s=48 peacemaker07
2
220
2607469010bfd7a0f0bd7546328cb01d?s=48 uakihir0
1
100
Da1305e5281b5208da85d14a356d01f3?s=48 taka1068
5
230
0157a830bfaa0c5426422c45aa0b2637?s=48 bo0km4n
3
440
C4b9a181d7fb82642b4c8c1df6ae2b87?s=48 louvre2489
1
160
4ccaadf34668c439a607c8b6083ba3b2?s=48 peaceiris
0
350
5ad0bb268dd2605b09165d464308cb7e?s=48 igaiga
2
120
Fd3442482c4a9b2d2353448a6e379823?s=48 matsu_chara
2
410
8284465a94bbdf1ea82cf1a67d55f447?s=48 kilometer
1
140
15934fa2aa7b2ce21f091e9b7cffa856?s=48 manfredsteyer
1
200

Featured

See All Featured
9fa239b3154a0169d99ab7bf1422dd12?s=48 marcelosomers
220
14k
Aff5641764408271f7bc398f2097edd0?s=48 denniskardys
219
110k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
165
6.8k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
23
1.6k
7cbd3f5f922d798cc312eaf596de7ae6?s=48 iamctodd
10
1.4k
E76911cbe088e5b850d966de3fc7435b?s=48 robhawkes
52
2.7k
F383c6a4dc55e331bbe2987b622cee6b?s=48 stephaniewalter
256
10k
C0b42577cfc2b321be3474618107e933?s=48 samanthasiow
53
6k
32de0bd2ba869609d26fd052a4622778?s=48 cromwellryan
100
5.7k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
775
240k
5b6a2bd86ef643786a381a212e0ef2f1?s=48 geoffreycrofte
11
270
027d1ebf66cc039a0bd3b55eeadbe75d?s=48 sachag
448
35k

Transcript

  1. RPC Metrics at Google JBD, Google (@rakyll)

  2. gRPC Metrics at Google JBD, Google (@rakyll)

  3. Request Metrics at Google JBD, Google (@rakyll)

  4. @rakyll "100% is the wrong reliability target for basically everything."

    -- Benjamin Treynor Sloss, VP of Engineering, Google

  5. @rakyll "A service is available if users cannot tell that

    there was an outage."

  6. @rakyll Principled way of saying what level of downtime is

    acceptable. • Error rate • Latency expectations SLOs

  7. @rakyll Analytics frontend server Authentication Reporting Users ... Spanner Blob

    Store

  8. @rakyll Questions infra teams want to ask: • Are we

    meeting the SLO for the other team? • What’s the impact of a product on infra? • How much do we need to scale up if product grows 10%?

  9. @rakyll High-Cardinality Breaking down the metrics data...

  10. @rakyll Query the collected data in various ways: • Latency

    distribution for RPCs originated at Google Analytics. • Requests take took more than 100ms for the customer #123. • Compare the request latency initiated at web vs mobile frontend.

  11. @rakyll Analytics frontend server Authentication Reporting Users ... Spanner Blob

    Store originator=analytics; ...

  12. @rakyll Blob store read errors by originator

  13. @rakyll Dynamically choose aggregation (split between recording and aggregation)

  14. @rakyll Exemplars

  15. @rakyll /rpz and /statz

  16. @rakyll http://server:7777/debug/rpcz

  17. @rakyll Export? Monarch, Prometheus, and more.

  18. @rakyll import “cloud.google.com/go/pubsub”

  19. @rakyll +

  20. Thank you! JBD, Google jbd@google.com @rakyll