Keep watching and extending features of gRPC

mercari.go #5

Transcript

1. Keep watching and extending features of gRPC mercari.go #5
2. @kazegusuri • Merpay Backend Engineer • Architect Team 2015/11 2017/01 2018/01 SRE & Platform Platform Payment & Architect
3. Objective of this talk Just I talk about gRPC 01 Tell features of gRPC Just I talk about gRPC 03 02
4. ⚠Disclaimers⚠ • The contents are not for beginners of gRPC • I am talking forever once started. You will fall asleep. • The details are not described in this slide, just I talk ◦ You are lucky! or unlucky for boring time • This slide was originally created for internal meet up in merpay ◦ for mercari employees
5. My information source of gRPC • github.com/grpc/proposal ◦ gRFC • github.com/grpc/grpc-go ◦ read almost all PRs • github.com/grpc/grpc ◦ no longer read • Twitter ◦ search grpc!!
6. gRFC • Design proposals for substantial feature changes • Public disucssion not only inside Google ◦ Protobuf is still Google product.. not community • Looks effective to keep consistent features in all language impls. • Recommend to read merged docs for everyone
7. Recommended documentations • github.com/grpc/grpc-go/Documentation ◦ Docs for how to use features in grpc-go ◦ Actively updated recently • github.com/grpc/proposal ◦ gRFC for new features • github.com/grpc/grpc/doc ◦ Useful specs still exist ◦ not active for spec in favor of gRFC?
8. gRPC history • 2015/02/25 Opensourced • 2016/01/11 (First use in Mercari) • 2016/07/26 (Protobuf3 v3.0.0) • 2016/08/23 Generally available • 2017/03/01 Became a CNCF project
9. Features of grpc-go Now I start to talk about new features of grpc-go for each releases
10. Before v1.0(GA) • 2016/01/11 (First use in Mercari) • 2016/01/26 Compression (deprecated) • 2016/04/19 Server Interceptor • 2016/04/23 (GoCon 2016 Spring) • 2016/06/01 Balancer V1 (deprecated) • 2016/06/29 Server Reflection • 2016/07/27 Graceful Shutdown • 2016/09/03 Client Interceptor
11. v1.0 2016/07/12 (~v1.2) • stats 2017/01/10 ◦ grpc.StatsHandler() ◦ Various stats provided per requests/connections ◦ OpenCensus uses this feature for instrumentation • service config 2016/12/20 ◦ grpc.WithServiceConfig() (deprecated) ◦ services/methods configurations dynamically pupulated • tap 2016/11/08 ◦ grpc.InTapHandle() ◦ called in very early stage after connected ◦ I don’t know the use case...
12. v1.2 2017/03/21 • keepalive ◦ grpc.KeepaliveParams(), grpc.KeepaliveEnforcementPolicy() ◦ Read grpc-go/Documentation/keepalive.md v1.3 2017/04/29 • status ◦ Introduced status package, which responsible for handling gRPC errors ◦ Read grpc-go/Documentation/rpc-errors.md
13. v1.4 2017/06/08 • CallOption ◦ credentials.PerRPCCredentials ◦ Set credential(metadata) per request ◦ Read grpc-go/Documentation/grpc-auth-support.md • CallOption ◦ grpc.Max[Send,Recv]MsgSize, grpc.MaxCall[Send,Recv]MsgSize ◦ Limit max message size in both client/server side v1.5 2017/07/20 • No remarkable features
14. v1.6 2017/08/31 • status ◦ details support in server side • DNS resolver (deprecated) ◦ legacy resolver API called naming v1.7 2017/10/12 • balancer, resolver ◦ V2? API by gRFC L9 ◦ Registration API for balancer, resolver ◦ Resolver is used depending on scheme of an address in grpc.DialContext
15. v1.8 2017/11/27 • client-side retry ◦ transparent retry in client by gRFC A6 ◦ retry to send a request when failure without sending any requests • encoding ◦ Registration API for grpc.Compressor interface for compress/decompress a message ◦ Now server respects encoding/accept-encoding automatically! ◦ Read grpc-go/Documentation/compression.md v1.9 2018/01/03 • No remarkable features
16. v1.10 2018/02/16 • encoding ◦ Registration API for grpc.Codec interface for marshal/unmarshal a message ◦ Now we can change a codec per call, not per connection(DialOption)! ◦ Read grpc-go/Documentation/encoding.md v1.11 2018/03/28 • credentials/alts ◦ ALTS: Application Layer Transport Security
17. v1.12 2018/05/09 • stickness ◦ requested server is determined by sessionid value of metadata • channelz ◦ gRFC A14 ◦ expose internal details about connections v1.13 2018/06/20 • No remarkable features
18. v1.14 2018/08/01 • client-side retry ◦ gRFC A6 ◦ client library handles retry by ServiceConfig per Method/StatusCode • stickness ◦ Reverted!!! ◦ re-considering to implement balancer/resolver-base v1.15 2018/09/15 • No remarkable features
19. v1.16 2018/10/24 • healthcheck ◦ Watch method in server • channelz ◦ added tracing event by gRFC A3 v1.17 2018/12/05 • binarylog ◦ gRFC A16 ◦ official logging mechanism for message,metadata as protobuf • healthcheck ◦ client-side health check by gRFC A17
20. Extend the features • interceptor ◦ github.com/mercari/go-grpc-interceptor ▪ no longer used… ◦ internal usecase ▪ DataDog for instrumentation ▪ Notification with Sentry in case of unexpected errors like Unknown/Internal ◦ middlewares for gRPC to resolve common use cases • resolver ◦ custom resolver for kubernetes specific named k8sdns ◦ official dns resolver is not suitable for our use case
21. Extend the features • reflection ◦ kazegusuri/grpcurl ◦ Easy to call gRPC method like curl without grpc-gateway • channelz ◦ kazegusuri/channelzcli ◦ To observe state of gRPC connections for debugging
22. gRPC with protoc plugin • kazegusuri/go-proto-zap-marshaler ◦ generate go function to log proto message with zap • kazegusuri/go-proto-anonymizer ◦ generate go function to filter credentials or secrets fields in message
23. Extend the features (idea) • service config ◦ clients can know configs like retry policy, timeout ◦ difficult to use DNS, how to provide…? e.g. custom resolver • encoding (codec) ◦ CSV/TSV codec…! ◦ message passthrough gRPC proxy without any marshaling/unmarshaling • stickness ◦ use custom balancer instead • binarylog ◦ consider to use application log if better than interceptor