RPC Metrics at Google

Transcript

1. 1.

   RPC Metrics at Google JBD, Google (@rakyll)

2. 2.

   gRPC Metrics at Google JBD, Google (@rakyll)

3. 3.

   Request Metrics at Google JBD, Google (@rakyll)

4. 4.

   @rakyll "100% is the wrong reliability target for basically everything."

   -- Benjamin Treynor Sloss, VP of Engineering, Google
5. 5.

   @rakyll "A service is available if users cannot tell that

   there was an outage."
6. 6.

   @rakyll Principled way of saying what level of downtime is

   acceptable. • Error rate • Latency expectations SLOs
7. 7.

   @rakyll Analytics frontend server Authentication Reporting Users ... Spanner Blob

   Store
8. 8.

   @rakyll Questions infra teams want to ask: • Are we

   meeting the SLO for the other team? • What’s the impact of a product on infra? • How much do we need to scale up if product grows 10%?
9. 9.

   @rakyll High-Cardinality Breaking down the metrics data...

10. 10.

    @rakyll Query the collected data in various ways: • Latency

    distribution for RPCs originated at Google Analytics. • Requests take took more than 100ms for the customer #123. • Compare the request latency initiated at web vs mobile frontend.
11. 11.

    @rakyll Analytics frontend server Authentication Reporting Users ... Spanner Blob

    Store originator=analytics; ...
12. 12.

    @rakyll Blob store read errors by originator

13. 13.

    @rakyll Dynamically choose aggregation (split between recording and aggregation)

14. 14.

    @rakyll Exemplars

15. 15.

    @rakyll /rpz and /statz

16. 16.

    @rakyll http://server:7777/debug/rpcz

17. 17.

    @rakyll Export? Monarch, Prometheus, and more.

18. 18.

    @rakyll import “cloud.google.com/go/pubsub”

19. 19.

    @rakyll +

20. 20.

    Thank you! JBD, Google jbd@google.com @rakyll