Jfokus 2019: Adopting gRPC at Spotify

After many years of relying on our own Spotify-proprietary message protocol, we have decided to adopt gRPC as our new default way of communication between the thousands of services that make up Spotify's backend.

This presentation will cover a real-world case study of migrating to gRPC, including various challenges and opportunities encountered along the way. In the pursuit of architecting reliable service communication, Matthias will cover the details in how his team go about changing such a fundamental piece of infrastructure and will walk through some of gRPC's most powerful features.

Transcript

2. -- someone at KubeCon 2018
6. ~2500 services ~1000 developers ~250 teams Java, Python, ...
9. Written in 2012 Based on ZeroMQ JSON or protobuf payload Not a RPC framework
13. 404 Not Found
17. Cloud Native
18. 1. HTTP/2 based 2. Binary protocol 3. Strongly typed service and message definition (Protobuf) 4. Encryption
21. Java, Golang, Python, Ruby, Dart, PHP, Node.js, Objective-C, C#, C++
22. syntax = "proto3"; package spotify.metadata.v1; option java_package = "com.spotify.metadata.v1" option java_multiple_files = true; option java_outer_classname = "MetadataProto"; // Interface exported by the server. service Metadata { rpc GetMetadata(SongId) returns (SongMetadata) {} } message SongId { int32 id = 1; } message SongMetadata { int32 id = 1; string name = 2; string artist = 3; string album = 4; }
23. public class MetadataService extends MetadataGrpc.MetadataImplBase { // [...] @Override public void getMetadata(SongId songId, StreamObserver<SongMetadata> response) { LOG.info("Received getMetadata request"); response.onNext(store.searchMetadata(songId) .orElse(EMPTY_METADATA)); response.onCompleted(); } }
24. func main() { ctx, cancel := context.WithTimeout( context.Background(), time.Second) defer cancel() ) conn, err := grpc.Dial("nls://metadata") if err != nil { log.Fatalf("couldn't connect to grpc server: %v", err) } defer conn.Close() client := pb.NewMetadataServiceClient(conn) r, err := client.Metdata(ctx, &pb.SongId{ Id: "42" }, ) if err != nil { log.Printf("metadata request failed: %v\n", err) return } fmt.Println(r.Response) }
26. 1. Embrace the proto 2. Shared repo for all protos 3. Version on the proto level
27. github.com/uber/prototool
29. Deadlines
30. A common RPC
31. Retries • Transparent • Configurable
32. Hedging
34. Retry throttling
36. Load Balancing • Client-side • Proxy • Lookaside
37. Load Balancing lookaside
40. 2874 Services 1341 HTTP 983 Hermes 76 gRPC
44. code generation
45. code generation Resiliency patterns
46. code generation Resiliency patterns Tracing
47. code generation Resiliency patterns Tracing Istio
52. 1. Add a new gRPC API 2. Move clients to new API 3. Remove old API
53. 1. Add a new gRPC API 2. Move clients to new API 3. Remove old API this is hard!
58. -- someone at KubeCon 2018