gRPC and Protobufs - JSCamp Romania

My talk that I gave on gRPC and Protocol Buffers at JSCamp Romania.

Transcript

1. gRPC and Protobufs Iheanyi Ekechukwu
2. Iheanyi Ekechukwu Software Engineer @ DigitalOcean @kwuchu
3. What are protobufs?
4. Protobufs
5. Protocol Buffers
6. Language-agnostic method for serializing structured data.
7. How do they work?
8. Defining a Message
9. // contact.proto syntax = "proto3"; message Contact { string name = 1; string email = 2; repeated PhoneNumber phone_numbers = 3; PhoneNumber home = 4; PhoneNumber mobile = 5; PhoneNumber work = 6; } message PhoneNumber { enum PhoneType { MOBILE = 0; HOME = 1; WORK = 2; } string number = 1; PhoneType type = 2; }
10. message Contact
11. string name = 1;
12. double float int32 int64 uint32 uint64 sint32 sint64 fixed32 fixed64 sfixed32 sfixed64 bool string
13. repeated PhoneNumber phone_numbers = 3;
14. message PhoneNumber { enum PhoneType { MOBILE = 0; HOME = 1; WORK = 2; } string number = 1; PhoneType type = 2; }
15. enum PhoneType { MOBILE = 0; HOME = 1; WORK = 2; }
16. Compiled to a binary wire format
17. Compiling
18. protoc -I=. --go_out=. --js_out=import_style=commonjs,binary:. contact.proto
19. Generates two files: contact.pb.go and contact_pb.js
20. package contact import proto "github.com/golang/protobuf/proto" import fmt "fmt" import math "math" // other code omitted for brevity type Contact struct { Name string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` Email string `protobuf:"bytes,2,opt,name=email" json:"email,omitempty"` PhoneNumbers []*PhoneNumber `protobuf:"bytes, 3,rep,name=phone_numbers,json=phoneNumbers" json:"phone_numbers,omitempty"` Home *PhoneNumber `protobuf:"bytes,4,opt,name=home" json:"home,omitempty"` Mobile *PhoneNumber `protobuf:"bytes,5,opt,name=mobile" json:"mobile,omitempty"` Work *PhoneNumber `protobuf:"bytes,6,opt,name=work" json:"work,omitempty"` }
21. func init() { proto.RegisterFile("contact.proto", fileDescriptor0) } var fileDescriptor0 = []byte{ // 249 bytes of a gzipped FileDescriptorProto 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x4d, 0xce, 0xcf, 0x2b, 0x49, 0x4c, 0x2e, 0xd1, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0x87, 0x72, 0x95, 0x3e, 0x30, 0x72, 0xb1, 0x3b, 0x43, 0xd8, 0x42, 0x42, 0x5c, 0x2c, 0x79, 0x89, 0xb9, 0xa9, 0x12, 0x8c, 0x0a, 0x8c, 0x1a, 0x9c, 0x41, 0x60, 0xb6, 0x90, 0x08, 0x17, 0x6b, 0x6a, 0x6e, 0x62, 0x66, 0x8e, 0x04, 0x13, 0x58, 0x10, 0xc2, 0x11, 0xb2, 0xe4, 0xe2, 0x2d, 0xc8, 0xc8, 0xcf, 0x4b, 0x8d, 0xcf, 0x2b, 0xcd, 0x4d, 0x4a, 0x2d, 0x2a, 0x96, 0x60, 0x56, 0x60, 0xd6, 0xe0, 0x36, 0x12, 0xd1, 0x83, 0xd9, 0x12, 0x00, 0x92, 0xf5, 0x03, 0x4b, 0x06, 0xf1, 0x14, 0x20, 0x38, 0xc5, 0x42, 0x1a, 0x5c, 0x2c, 0x19, 0xf9, 0xb9, 0xa9, 0x12, 0x2c, 0x0a, 0x8c, 0x38, 0x75, 0x80, 0x55, 0x08, 0xe9, 0x70, 0xb1, 0xe5, 0xe6, 0x27, 0x65, 0xe6, 0xa4, 0x4a, 0xb0, 0xe2, 0x51, 0x0b, 0x55, 0x03, 0x32, 0xb7, 0x3c, 0xbf, 0x28, 0x5b, 0x82, 0x0d, 0x9f, 0xb9, 0x20, 0x15, 0x4a, 0x6d, 0x8c, 0x5c, 0xdc, 0x48, 0xa2, 0x42, 0x62, 0x5c, 0x6c, 0x10, 0x6f, 0x40, 0x3d, 0x0e, 0xe5, 0x09, 0x19, 0x71, 0xb1, 0x94, 0x54, 0x16, 0xa4, 0x82, 0x7d, 0xce, 0x67, 0x24, 0x87, 0xcd, 0x44, 0x08, 0x3b, 0xa4, 0xb2, 0x20, 0x35, 0x08, 0xac, 0x56, 0x49, 0x9b, 0x8b, 0x13, 0x2e, 0x24, 0xc4, 0xc5, 0xc5, 0xe6, 0xeb, 0xef, 0xe4, 0xe9, 0xe3, 0x2a, 0xc0, 0x20, 0xc4, 0xc1, 0xc5, 0xe2, 0xe1, 0xef, 0xeb, 0x2a, 0xc0, 0x08, 0x62, 0x85, 0xfb, 0x07, 0x79, 0x0b, 0x30, 0x25, 0xb1, 0x81, 0xe3, 0xc2, 0x18, 0x10, 0x00, 0x00, 0xff, 0xff, 0x9b, 0xec, 0x7d, 0xa3, 0x9c, 0x01, 0x00, 0x00, }
22. const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package type PhoneNumber_PhoneType int32 const ( PhoneNumber_MOBILE PhoneNumber_PhoneType = 0 PhoneNumber_HOME PhoneNumber_PhoneType = 1 PhoneNumber_WORK PhoneNumber_PhoneType = 2 ) var PhoneNumber_PhoneType_name = map[int32]string{ 0: "MOBILE", 1: "HOME", 2: "WORK", } var PhoneNumber_PhoneType_value = map[string]int32{ "MOBILE": 0, "HOME": 1, "WORK": 2, } func (x PhoneNumber_PhoneType) String() string { return proto.EnumName(PhoneNumber_PhoneType_name, int32(x)) } func (PhoneNumber_PhoneType) EnumDescriptor() ([]byte, []int) { return fileDescriptor0, []int{1, 0} } type Contact struct { Name string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` Email string `protobuf:"bytes,2,opt,name=email" json:"email,omitempty"` PhoneNumbers []*PhoneNumber `protobuf:"bytes,3,rep,name=phone_numbers,json=phoneNumbers" json:"phone_numbers,omitempty"` Home *PhoneNumber `protobuf:"bytes,4,opt,name=home" json:"home,omitempty"` Mobile *PhoneNumber `protobuf:"bytes,5,opt,name=mobile" json:"mobile,omitempty"` Work *PhoneNumber `protobuf:"bytes,6,opt,name=work" json:"work,omitempty"` } func (m *Contact) Reset() { *m = Contact{} } func (m *Contact) String() string { return proto.CompactTextString(m) } func (*Contact) ProtoMessage() {} func (*Contact) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{0} } func (m *Contact) GetPhoneNumbers() []*PhoneNumber { if m != nil { return m.PhoneNumbers } return nil } func (m *Contact) GetHome() *PhoneNumber { if m != nil { return m.Home } return nil } func (m *Contact) GetMobile() *PhoneNumber { if m != nil { return m.Mobile } return nil } func (m *Contact) GetWork() *PhoneNumber { if m != nil { return m.Work } return nil } type PhoneNumber struct { Number string `protobuf:"bytes,1,opt,name=number" json:"number,omitempty"` Type PhoneNumber_PhoneType `protobuf:"varint,2,opt,name=type,enum=contact.PhoneNumber_PhoneType" json:"type,omitempty"` } func (m *PhoneNumber) Reset() { *m = PhoneNumber{} } func (m *PhoneNumber) String() string { return proto.CompactTextString(m) } func (*PhoneNumber) ProtoMessage() {} func (*PhoneNumber) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{1} }
23. Also, getters and setters are defined for the message’s attributes
24. // contact_pb.js // code omitted for brevity /** * optional string name = 1; * @return {string} */ proto.contact.Contact.prototype.getName = function() { return /** @type {string} */ (jspb.Message.getFieldWithDefault(thi 1, "")); }; /** @param {string} value */ proto.contact.Contact.prototype.setName = function(value) { jspb.Message.setField(this, 1, value); };
25. Why use them?
26. Harder Simpler.
27. Clear definition of the data model.
28. Language Agnostic.
29. Better.
30. Faster.
31. In comparison to XML/ JSON…
32. 3 to 10 times smaller
33. 20 to 100 times faster.
34. Stronger.
35. Strict Contract
36. Type definitions are also good.
37. Simpler.
38. Simpler. Better.
39. Simpler. Better. Faster.
40. Simpler. Better. Faster. Stronger.
42. Let’s talk gRPC.
43. What is gRPC?
44. RPC framework by Google that extends protocol buffers.
45. Why use it?
46. Uses HTTP/2 under the hood.
47. SSL/TLS for secure communication between server and clients.
48. Deadlines/timeouts can be specified on both the server and the client.
49. Cancellable RPCs
50. Anybody here ever been burned by JSON changes?
52. Safe (to an extent), to rename fields.
53. Did I mention clients are backwards/forward-compatible?
55. Old client binaries ignore new fields added.
56. All the benefits of protocol buffers.
57. Stronger contracts.
58. Faster & Better Performance
59. Language Agnosticism for server and clients.
60. Great for microservices.
61. So, how does it work?
62. Let’s build a phonebook service and CLI.
63. First, define the service
64. syntax = "proto3"; package api; service PhoneBook {}
65. Let’s add an RPC method to create a contact.
66. syntax = "proto3"; package api; service PhoneBook {}
67. syntax = "proto3"; package api; service PhoneBook { rpc CreateContact(CreateContactReq) returns (CreateContactRes); } message CreateContactReq { string name = 1; string email = 2; repeated PhoneNumber phone_numbers = 3; PhoneNumber home = 4; PhoneNumber mobile = 5; PhoneNumber work = 6; } message CreateContactRes { Contact contact = 1; }
68. Compile the protobuf with the gRPC plugin.
69. protoc -I=. --proto_path=${GOPATH}/src:. --go_out=plugins=grpc:. \ --js_out=import_style=commonjs,binary:. api.proto
70. Generates api.pb.go and api_pb.js, like before, with some differences.
71. // api/api.pb.go package api // Some code omitted for brevity // Server API for PhoneBook service type PhoneBookServer interface { CreateContact(context.Context, *CreateContactReq) (*CreateContactRes, error) } func RegisterPhoneBookServer(s *grpc.Server, srv PhoneBookServer) { s.RegisterService(&_PhoneBook_serviceDesc, srv) }
72. The client also gets defined within these files.
73. type PhoneBookClient interface { CreateContact(ctx context.Context, in *CreateContactReq, opts ...grpc.CallOption) (*CreateContactRes, error) } type phoneBookClient struct { cc *grpc.ClientConn } func NewPhoneBookClient(cc *grpc.ClientConn) PhoneBookClient { return &phoneBookClient{cc} }
74. Let’s implement the CreateContact method on the Go server.
75. // server/server.go package server import ( "sync" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" oldctx "golang.org/x/net/context" api "github.com/iheanyi/grpc-phonebook/api" ) // Use in-memory DB for simplicity type server struct { contactsByNameMu sync.RWMutex contactsByName map[string]*api.Contact }
76. func (svc *server) CreateContact(ctx oldctx.Context, req *api.CreateContactReq) (*api.CreateContactRes, error) { if len(req.Name) == 0 { return nil, status.Errorf(codes.InvalidArgument, "name cannot be blank") } contact := &api.Contact{ Name: req.Name, Email: req.Email, PhoneNumbers: req.PhoneNumbers, Home: req.Home, Mobile: req.Mobile, Work: req.Work, } svc.contactsByName[contact.Name] = contact res := &api.CreateContactRes{ Contact: contact, } return res, nil }
77. Register the service with a gRPC Server
78. // server/server.go func New() api.PhoneBookServer { contactsByName := make(map[string]*api.Contact) return &server{ contactsByName: contactsByName, } }
79. // cmd/pbsrv/main.go package main import ( "log" "net" "github.com/iheanyi/grpc-phonebook/api" "github.com/iheanyi/grpc-phonebook/server" "google.golang.org/grpc" ) func main() { lis, err := net.Listen("tcp", ":50051") if err != nil { log.Fatalf("failed to listen: %v", err) } srv := grpc.NewServer() svc := server.New() api.RegisterPhoneBookServer(srv, svc) srv.Serve(lis) }
80. Done.
81. Now, let’s implement it in our Node.js client.
82. #!/usr/bin/env node // code omitted for brevity // cmd/pbctl/pbctl var PROTO_PATH = __dirname + '/../../api/api.proto'; var grpc = require('grpc'); var program = require('commander'); var path = require('path'); var api = grpc.load(path.resolve(PROTO_PATH)).api; var client = new api.PhoneBook('localhost:50051', grpc.credentials.createInsecure()); require('console.table'); var PhoneType = api.PhoneNumber.PhoneType; // end of file program.parse(process.argv);
83. program .command('create <name>’) .option('-e, --email <email>', "Contact's email") .option('-h, --home <home>', "Contact's home number") .option('-m, --mobile <mobile>', "Contact's mobile number") .option('-w, --work <work>', "Contact's work number")
84. program.action(function(name, options) { var request = new api.CreateContactReq(); var phoneArr = []; if (options.home) { var homeNum = new api.PhoneNumber(); homeNum.type = PhoneType.HOME; homeNum.number = options.home; phoneArr.push(homeNum); request.home = homeNum; } // Repeat logic for mobile/work numbers. // Omitted for brevity. request.name = name; request.email = options.email || ''; request.phone_numbers = phoneArr; client.createContact(request, function(err, response) { if (err) { console.error(err.message); return; } printContacts(formatContacts([response.contact])); }); })
85. Done. !
86. Let’s test it out.
87. First, let’s compile and run our server in one terminal.
88. go install ./cmd/pbsrv/... && pbsrv
89. Let’s try out our create command.
90. ./pbctl create “Iheanyi Ekechukwu” -h 1234567890 -e test@example.com
91. Our Output! name email home mobile work ----------------- ---------------- ---------- ------ ---- Iheanyi Ekechukwu test@example.com 1234567890
92. Nice!
93. And what if our input is invalid?
94. ./pbctl create "" -h 1234567890
95. Outputs to stderr name cannot be blank
97. But it doesn’t stop there, we could implement other methods too!
98. service PhoneBook { rpc CreateContact(CreateContactReq) returns (CreateContactRes); rpc ListContacts(ListContactsReq) returns (ListContactsRes); rpc DeleteContact(DeleteContactReq) returns (DeleteContactRes); rpc ShowContact(ShowContactReq) returns (ShowContactRes); rpc UpdateContact(UpdateContactReq) returns (UpdateContactRes); }
100. https://github.com/iheanyi/grpc-phonebook
101. Other Features of gRPC
102. Bring your own IDL.
103. Load Balancing
104. Interceptors (Go, Java)
105. Streaming (Client/Server/Bidirectional)
106. Client Streaming
107. Server Streaming
108. Bidirectional Streaming
109. Metadata
110. The OSS Ecosystem
111. https://github.com/grpc-ecosystem
112. gRPC enables new possibilities for building software.
113. gRPC + Mobile Applications
114. gRPC + Electron
115. gRPC Web
116. Further Reading • https://grpc.io - gRPC Website & Documentation • https://github.com/improbable-eng/grpc-web – Improbable’s gRPC Web implementation • https://developers.google.com/protocol-buffers/docs/ proto3 - Protocol Buffers Documentation • https://github.com/grpc-ecosystem – GitHub Organization with various tools that integrate with gRPC
117. Thank you! @kwuchu