gRPC and Protobufs - JSCamp Romania

Transcript

1. 1.

   gRPC and Protobufs Iheanyi Ekechukwu

2. 2.

   Iheanyi Ekechukwu Software Engineer @ DigitalOcean @kwuchu

3. 3.

   What are protobufs?

4. 4.

   Protobufs

5. 5.

   Protocol Buffers

6. 6.

   Language-agnostic method for serializing structured data.

7. 7.

   How do they work?

8. 8.

   Defining a Message

9. 9.

   // contact.proto syntax = "proto3"; message Contact { string name

   = 1; string email = 2; repeated PhoneNumber phone_numbers = 3; PhoneNumber home = 4; PhoneNumber mobile = 5; PhoneNumber work = 6; } message PhoneNumber { enum PhoneType { MOBILE = 0; HOME = 1; WORK = 2; } string number = 1; PhoneType type = 2; }
10. 10.

    message Contact

11. 11.

    string name = 1;

12. 12.

    double float int32 int64 uint32 uint64 sint32 sint64 fixed32 fixed64

    sfixed32 sfixed64 bool string
13. 13.

    repeated PhoneNumber phone_numbers = 3;

14. 14.

    message PhoneNumber { enum PhoneType { MOBILE = 0; HOME

    = 1; WORK = 2; } string number = 1; PhoneType type = 2; }
15. 15.

    enum PhoneType { MOBILE = 0; HOME = 1; WORK

    = 2; }
16. 16.

    Compiled to a binary wire format

17. 17.

    Compiling

18. 18.

    protoc -I=. --go_out=. --js_out=import_style=commonjs,binary:. contact.proto

19. 19.

    Generates two files: contact.pb.go and contact_pb.js

20. 20.

    package contact import proto "github.com/golang/protobuf/proto" import fmt "fmt" import math

    "math" // other code omitted for brevity type Contact struct { Name string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` Email string `protobuf:"bytes,2,opt,name=email" json:"email,omitempty"` PhoneNumbers []*PhoneNumber `protobuf:"bytes, 3,rep,name=phone_numbers,json=phoneNumbers" json:"phone_numbers,omitempty"` Home *PhoneNumber `protobuf:"bytes,4,opt,name=home" json:"home,omitempty"` Mobile *PhoneNumber `protobuf:"bytes,5,opt,name=mobile" json:"mobile,omitempty"` Work *PhoneNumber `protobuf:"bytes,6,opt,name=work" json:"work,omitempty"` }
21. 21.

    func init() { proto.RegisterFile("contact.proto", fileDescriptor0) } var fileDescriptor0 = []byte{

    // 249 bytes of a gzipped FileDescriptorProto 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x4d, 0xce, 0xcf, 0x2b, 0x49, 0x4c, 0x2e, 0xd1, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0x87, 0x72, 0x95, 0x3e, 0x30, 0x72, 0xb1, 0x3b, 0x43, 0xd8, 0x42, 0x42, 0x5c, 0x2c, 0x79, 0x89, 0xb9, 0xa9, 0x12, 0x8c, 0x0a, 0x8c, 0x1a, 0x9c, 0x41, 0x60, 0xb6, 0x90, 0x08, 0x17, 0x6b, 0x6a, 0x6e, 0x62, 0x66, 0x8e, 0x04, 0x13, 0x58, 0x10, 0xc2, 0x11, 0xb2, 0xe4, 0xe2, 0x2d, 0xc8, 0xc8, 0xcf, 0x4b, 0x8d, 0xcf, 0x2b, 0xcd, 0x4d, 0x4a, 0x2d, 0x2a, 0x96, 0x60, 0x56, 0x60, 0xd6, 0xe0, 0x36, 0x12, 0xd1, 0x83, 0xd9, 0x12, 0x00, 0x92, 0xf5, 0x03, 0x4b, 0x06, 0xf1, 0x14, 0x20, 0x38, 0xc5, 0x42, 0x1a, 0x5c, 0x2c, 0x19, 0xf9, 0xb9, 0xa9, 0x12, 0x2c, 0x0a, 0x8c, 0x38, 0x75, 0x80, 0x55, 0x08, 0xe9, 0x70, 0xb1, 0xe5, 0xe6, 0x27, 0x65, 0xe6, 0xa4, 0x4a, 0xb0, 0xe2, 0x51, 0x0b, 0x55, 0x03, 0x32, 0xb7, 0x3c, 0xbf, 0x28, 0x5b, 0x82, 0x0d, 0x9f, 0xb9, 0x20, 0x15, 0x4a, 0x6d, 0x8c, 0x5c, 0xdc, 0x48, 0xa2, 0x42, 0x62, 0x5c, 0x6c, 0x10, 0x6f, 0x40, 0x3d, 0x0e, 0xe5, 0x09, 0x19, 0x71, 0xb1, 0x94, 0x54, 0x16, 0xa4, 0x82, 0x7d, 0xce, 0x67, 0x24, 0x87, 0xcd, 0x44, 0x08, 0x3b, 0xa4, 0xb2, 0x20, 0x35, 0x08, 0xac, 0x56, 0x49, 0x9b, 0x8b, 0x13, 0x2e, 0x24, 0xc4, 0xc5, 0xc5, 0xe6, 0xeb, 0xef, 0xe4, 0xe9, 0xe3, 0x2a, 0xc0, 0x20, 0xc4, 0xc1, 0xc5, 0xe2, 0xe1, 0xef, 0xeb, 0x2a, 0xc0, 0x08, 0x62, 0x85, 0xfb, 0x07, 0x79, 0x0b, 0x30, 0x25, 0xb1, 0x81, 0xe3, 0xc2, 0x18, 0x10, 0x00, 0x00, 0xff, 0xff, 0x9b, 0xec, 0x7d, 0xa3, 0x9c, 0x01, 0x00, 0x00, }
22. 22.

    const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package

    type PhoneNumber_PhoneType int32 const ( PhoneNumber_MOBILE PhoneNumber_PhoneType = 0 PhoneNumber_HOME PhoneNumber_PhoneType = 1 PhoneNumber_WORK PhoneNumber_PhoneType = 2 ) var PhoneNumber_PhoneType_name = map[int32]string{ 0: "MOBILE", 1: "HOME", 2: "WORK", } var PhoneNumber_PhoneType_value = map[string]int32{ "MOBILE": 0, "HOME": 1, "WORK": 2, } func (x PhoneNumber_PhoneType) String() string { return proto.EnumName(PhoneNumber_PhoneType_name, int32(x)) } func (PhoneNumber_PhoneType) EnumDescriptor() ([]byte, []int) { return fileDescriptor0, []int{1, 0} } type Contact struct { Name string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` Email string `protobuf:"bytes,2,opt,name=email" json:"email,omitempty"` PhoneNumbers []*PhoneNumber `protobuf:"bytes,3,rep,name=phone_numbers,json=phoneNumbers" json:"phone_numbers,omitempty"` Home *PhoneNumber `protobuf:"bytes,4,opt,name=home" json:"home,omitempty"` Mobile *PhoneNumber `protobuf:"bytes,5,opt,name=mobile" json:"mobile,omitempty"` Work *PhoneNumber `protobuf:"bytes,6,opt,name=work" json:"work,omitempty"` } func (m *Contact) Reset() { *m = Contact{} } func (m *Contact) String() string { return proto.CompactTextString(m) } func (*Contact) ProtoMessage() {} func (*Contact) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{0} } func (m *Contact) GetPhoneNumbers() []*PhoneNumber { if m != nil { return m.PhoneNumbers } return nil } func (m *Contact) GetHome() *PhoneNumber { if m != nil { return m.Home } return nil } func (m *Contact) GetMobile() *PhoneNumber { if m != nil { return m.Mobile } return nil } func (m *Contact) GetWork() *PhoneNumber { if m != nil { return m.Work } return nil } type PhoneNumber struct { Number string `protobuf:"bytes,1,opt,name=number" json:"number,omitempty"` Type PhoneNumber_PhoneType `protobuf:"varint,2,opt,name=type,enum=contact.PhoneNumber_PhoneType" json:"type,omitempty"` } func (m *PhoneNumber) Reset() { *m = PhoneNumber{} } func (m *PhoneNumber) String() string { return proto.CompactTextString(m) } func (*PhoneNumber) ProtoMessage() {} func (*PhoneNumber) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{1} }
23. 23.

    Also, getters and setters are defined for the message’s attributes

24. 24.

    // contact_pb.js // code omitted for brevity /** * optional

    string name = 1; * @return {string} */ proto.contact.Contact.prototype.getName = function() { return /** @type {string} */ (jspb.Message.getFieldWithDefault(thi 1, "")); }; /** @param {string} value */ proto.contact.Contact.prototype.setName = function(value) { jspb.Message.setField(this, 1, value); };
25. 25.

    Why use them?

26. 26.

    Harder Simpler.

27. 27.

    Clear definition of the data model.

28. 28.

    Language Agnostic.

29. 29.

    Better.

30. 30.

    Faster.

31. 31.

    In comparison to XML/ JSON…

32. 32.

    3 to 10 times smaller

33. 33.

    20 to 100 times faster.

34. 34.

    Stronger.

35. 35.

    Strict Contract

36. 36.

    Type definitions are also good.

37. 37.

    Simpler.

38. 38.

    Simpler. Better.

39. 39.

    Simpler. Better. Faster.

40. 40.

    Simpler. Better. Faster. Stronger.

41. 41.

    None
42. 42.

    Let’s talk gRPC.

43. 43.

    What is gRPC?

44. 44.

    RPC framework by Google that extends protocol buffers.

45. 45.

    Why use it?

46. 46.

    Uses HTTP/2 under the hood.

47. 47.

    SSL/TLS for secure communication between server and clients.

48. 48.

    Deadlines/timeouts can be specified on both the server and the

    client.
49. 49.

    Cancellable RPCs

50. 50.

    Anybody here ever been burned by JSON changes?

51. 51.

    None
52. 52.

    Safe (to an extent), to rename fields.

53. 53.

    Did I mention clients are backwards/forward-compatible?

54. 54.

    None
55. 55.

    Old client binaries ignore new fields added.

56. 56.

    All the benefits of protocol buffers.

57. 57.

    Stronger contracts.

58. 58.

    Faster & Better Performance

59. 59.

    Language Agnosticism for server and clients.

60. 60.

    Great for microservices.

61. 61.

    So, how does it work?

62. 62.

    Let’s build a phonebook service and CLI.

63. 63.

    First, define the service

64. 64.

    syntax = "proto3"; package api; service PhoneBook {}

65. 65.

    Let’s add an RPC method to create a contact.

66. 66.

    syntax = "proto3"; package api; service PhoneBook {}

67. 67.

    syntax = "proto3"; package api; service PhoneBook { rpc CreateContact(CreateContactReq)

    returns (CreateContactRes); } message CreateContactReq { string name = 1; string email = 2; repeated PhoneNumber phone_numbers = 3; PhoneNumber home = 4; PhoneNumber mobile = 5; PhoneNumber work = 6; } message CreateContactRes { Contact contact = 1; }
68. 68.

    Compile the protobuf with the gRPC plugin.

69. 69.

    protoc -I=. --proto_path=${GOPATH}/src:. --go_out=plugins=grpc:. \ --js_out=import_style=commonjs,binary:. api.proto

70. 70.

    Generates api.pb.go and api_pb.js, like before, with some differences.

71. 71.

    // api/api.pb.go package api // Some code omitted for brevity

    // Server API for PhoneBook service type PhoneBookServer interface { CreateContact(context.Context, *CreateContactReq) (*CreateContactRes, error) } func RegisterPhoneBookServer(s *grpc.Server, srv PhoneBookServer) { s.RegisterService(&_PhoneBook_serviceDesc, srv) }
72. 72.

    The client also gets defined within these files.

73. 73.

    type PhoneBookClient interface { CreateContact(ctx context.Context, in *CreateContactReq, opts ...grpc.CallOption)

    (*CreateContactRes, error) } type phoneBookClient struct { cc *grpc.ClientConn } func NewPhoneBookClient(cc *grpc.ClientConn) PhoneBookClient { return &phoneBookClient{cc} }
74. 74.

    Let’s implement the CreateContact method on the Go server.

75. 75.

    // server/server.go package server import ( "sync" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" oldctx

    "golang.org/x/net/context" api "github.com/iheanyi/grpc-phonebook/api" ) // Use in-memory DB for simplicity type server struct { contactsByNameMu sync.RWMutex contactsByName map[string]*api.Contact }
76. 76.

    func (svc *server) CreateContact(ctx oldctx.Context, req *api.CreateContactReq) (*api.CreateContactRes, error) {

    if len(req.Name) == 0 { return nil, status.Errorf(codes.InvalidArgument, "name cannot be blank") } contact := &api.Contact{ Name: req.Name, Email: req.Email, PhoneNumbers: req.PhoneNumbers, Home: req.Home, Mobile: req.Mobile, Work: req.Work, } svc.contactsByName[contact.Name] = contact res := &api.CreateContactRes{ Contact: contact, } return res, nil }
77. 77.

    Register the service with a gRPC Server

78. 78.

    // server/server.go func New() api.PhoneBookServer { contactsByName := make(map[string]*api.Contact) return

    &server{ contactsByName: contactsByName, } }
79. 79.

    // cmd/pbsrv/main.go package main import ( "log" "net" "github.com/iheanyi/grpc-phonebook/api" "github.com/iheanyi/grpc-phonebook/server"

    "google.golang.org/grpc" ) func main() { lis, err := net.Listen("tcp", ":50051") if err != nil { log.Fatalf("failed to listen: %v", err) } srv := grpc.NewServer() svc := server.New() api.RegisterPhoneBookServer(srv, svc) srv.Serve(lis) }
80. 80.

    Done.

81. 81.

    Now, let’s implement it in our Node.js client.

82. 82.

    #!/usr/bin/env node // code omitted for brevity // cmd/pbctl/pbctl var

    PROTO_PATH = __dirname + '/../../api/api.proto'; var grpc = require('grpc'); var program = require('commander'); var path = require('path'); var api = grpc.load(path.resolve(PROTO_PATH)).api; var client = new api.PhoneBook('localhost:50051', grpc.credentials.createInsecure()); require('console.table'); var PhoneType = api.PhoneNumber.PhoneType; // end of file program.parse(process.argv);
83. 83.

    program .command('create <name>’) .option('-e, --email <email>', "Contact's email") .option('-h, --home

    <home>', "Contact's home number") .option('-m, --mobile <mobile>', "Contact's mobile number") .option('-w, --work <work>', "Contact's work number")
84. 84.

    program.action(function(name, options) { var request = new api.CreateContactReq(); var phoneArr

    = []; if (options.home) { var homeNum = new api.PhoneNumber(); homeNum.type = PhoneType.HOME; homeNum.number = options.home; phoneArr.push(homeNum); request.home = homeNum; } // Repeat logic for mobile/work numbers. // Omitted for brevity. request.name = name; request.email = options.email || ''; request.phone_numbers = phoneArr; client.createContact(request, function(err, response) { if (err) { console.error(err.message); return; } printContacts(formatContacts([response.contact])); }); })
85. 85.

    Done. !

86. 86.

    Let’s test it out.

87. 87.

    First, let’s compile and run our server in one terminal.

88. 88.

    go install ./cmd/pbsrv/... && pbsrv

89. 89.

    Let’s try out our create command.

90. 90.

    ./pbctl create “Iheanyi Ekechukwu” -h 1234567890 -e test@example.com

91. 91.

    Our Output! name email home mobile work ----------------- ---------------- ----------

    ------ ---- Iheanyi Ekechukwu test@example.com 1234567890
92. 92.

    Nice!

93. 93.

    And what if our input is invalid?

94. 94.

    ./pbctl create "" -h 1234567890

95. 95.

    Outputs to stderr name cannot be blank

96. 96.

    None
97. 97.

    But it doesn’t stop there, we could implement other methods

    too!
98. 98.

    service PhoneBook { rpc CreateContact(CreateContactReq) returns (CreateContactRes); rpc ListContacts(ListContactsReq) returns

    (ListContactsRes); rpc DeleteContact(DeleteContactReq) returns (DeleteContactRes); rpc ShowContact(ShowContactReq) returns (ShowContactRes); rpc UpdateContact(UpdateContactReq) returns (UpdateContactRes); }
99. 99.

    None
100. 100.

     https://github.com/iheanyi/grpc-phonebook

101. 101.

     Other Features of gRPC

102. 102.

     Bring your own IDL.

103. 103.

     Load Balancing

104. 104.

     Interceptors (Go, Java)

105. 105.

     Streaming (Client/Server/Bidirectional)

106. 106.

     Client Streaming

107. 107.

     Server Streaming

108. 108.

     Bidirectional Streaming

109. 109.

     Metadata

110. 110.

     The OSS Ecosystem

111. 111.

     https://github.com/grpc-ecosystem

112. 112.

     gRPC enables new possibilities for building software.

113. 113.

     gRPC + Mobile Applications

114. 114.

     gRPC + Electron

115. 115.

     gRPC Web

116. 116.

     Further Reading • https://grpc.io - gRPC Website & Documentation •

     https://github.com/improbable-eng/grpc-web – Improbable’s gRPC Web implementation • https://developers.google.com/protocol-buffers/docs/ proto3 - Protocol Buffers Documentation • https://github.com/grpc-ecosystem – GitHub Organization with various tools that integrate with gRPC
117. 117.

     Thank you! @kwuchu