GRPC with Golang and Python

GRPC Demo and Presentation for Medellin DevOps + Golang Medellin Meetups involving an AWS server running a GRPC and HTTP Server, a local GRPC Golang client on a computer, and a local GRPC Python client running on a Raspberry Pi which displays retrieved information on a SenseHat via GPIO.

Transcript

1. Santiago Martin Zubieta Ortiz Student at Universidad EAFIT Unemployed... for now :-P With Golang and Python.
2. http://www.grpc.io/about/ What’s it? “gRPC is a modern open source high performance RPC framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication. It is also applicable in last mile of distributed computing to connect devices, mobile applications and browsers to backend services.”
3. What’s it good for? The main usage scenarios: • Efficiently connecting polyglot services in microservices style architecture • Connecting mobile devices, browser clients to backend services • Generating efficient client libraries Core Features that make it awesome: • Idiomatic client libraries in 10 languages • Highly efficient on wire and with a simple service definition framework • Bi-directional streaming with http/2 based transport • Pluggable auth, tracing, load balancing and health checking http://www.grpc.io/docs/
4. http://www.grpc.io/docs/ Much languages! Such support. Very nice. phpthulu Y U NO SWIFT THO https://www.gopheracademy.com/ http://knowyourmeme.com/memes/gadsden-flag-don-t-tread-on-me http://lovecraft.wikia.com/wiki/Cthulhu
5. Works across languages and platforms To understand gRPC we need to understand first what protos are. http://www.grpc.io/ Though JSON can also be used.
6. Whats a “Proto” anyway? Think of a JSON but much more powerful. https://developers.google.com/protocol-buffers/ Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data. Structural definition, imported in both ends. Creation on Java side. Parsing on C++ side. $ protoc will automatically convert proto file to language-friendly ready-to-import equivalent.
7. A message, its what can be instantiated into a structure on any language. Defining this structure is like a neutral way of defining a structure with its properties, then the proto compiler will generate for each respective language a import-friendly, language specific version of that structure. So several languages can communicate using the same structures defined in the proto. Whole classes and services can be defined this way, all fulfilling the same contract. Also RPCs will be restricted to certain message types, unlike JSON where something with any structure can come in or go out without knowing before hand what’s it internal structure and having to validate the structure both when the server receives a request and when the client receives an answer. Proto3 Message Basics https://developers.google.com/protocol-buffers/docs/proto3
8. There’s a lot of types what a proto can contain and control over that. There’s enums, strings, booleans, floating point numbers of different precision, maps with both the keys and values defined, timestamps or durations, any kind of variable length binary object, and even lists of objects with known types. It can even contain other messages inside! Proto3 Message Basics https://developers.google.com/protocol-buffers/docs/proto3
9. https://github.com/google/protobuf/releases protoc --proto_path=IMPORT_PATH \ --cpp_out=DST_DIR \ --java_out=DST_DIR \ --python_out=DST_DIR \ --go_out=DST_DIR \ --ruby_out=DST_DIR \ --javanano_out=DST_DIR \ --objc_out=DST_DIR \ --csharp_out=DST_DIR \ path/to/file.proto The proto compiler will automatically transpile a .proto or several ones into files that can be imported from a program in many supported languages, ready to be imported, and its structs and services ready to be instantiated and implemented. $ brew install protobuf :-) (building from source is fine too) Proto3 Building Basics protoc -I=$SRC_DIR $SRC_DIR/{proto_name}.proto --go_out=$DST_DIR protoc -I protos/ protos/services.proto --go_out=plugins=grpc:protos $ python -m pip install grpcio $ python -m pip install grpcio-tools $ go get google.golang.org/grpc $ go get -u github.com/golang/protobuf/{proto,protoc-gen-go}
10. PhoneType enum only exists in scope of Person PhoneNumber message only exists in scope of Person message definition enum definition nested message definition nested A “list” of PhoneNumber messages A person has a name, id, email, and a list of phone numbers. An AddressBook has a list of people (each one an instance of Person), each of which has a Name, an Id, an Email, and a list of phones (each one an instance of PhoneNumber), each of which has a Number, and a PhoneType (MOBILE, HOME, WORK). A whole structure has been defined, ready to be used in any language! Proto3 Message Basics https://developers.google.com/protocol-buffers/docs/proto3
11. A bunch of code is generated from sample.proto into sample.pb.go! You shouldn’t modify this code, it has a lot of confusing stuff, many handlers, and definitions, but what protoc did was transpile the previous high level proto definition into the equivalent in the Go language. Go-Transpiled Proto
12. https://developers.google.com/protocol-buffers/docs/proto3 A service, contains a definition of an RPC, or several ones. All RPC defined within a service, must be implemented in whichever program imports this protobuffer file and instantiates a server based on this definition, so its like a contract. It works in terms of the name of the RPC, the kind of message it receives, and the kind of message it returns. This way, it knows beforehand what kind of structure must receive, and the caller has an assurance that either known structure will be returned, or an error for whatever cause. Proto3 Service Basics
13. When the built proto file is imported in its respective language, one can instantiate freely its structs/messages. But when a server is instantiated using the specification given in the protobuffer for a given service, then all RPCs must be implemented with the appropriate signature. A standard RPC server. The imported proto. Register{ServiceName}Server method is automatically generated for all {ServiceNames}. What this method does is bind RPCs from a given service (with a struct that implements them) on a generic GRPC server. The generic RPC server. A struct defined by us, which needs to implement the SearchService interface, that is, the defined RPCs. Service Implementation
14. ???????
15. Server Launch Service Interface Implementation Proto Definition main.go search.proto
16. !!!!!!!!
17. !!!!!!!!
18. Services are then a group of RPCs (in this case, SearchService, with Search and SearchAgain RPCs), and such services defined in the proto can be thought of as a contract, if you will create a server using any service, you must implement ALL the RPCs outlined in the service. Lets change to not match signature Lets erase this so its missing from the server implementation. Services Are Contracts Wrong type for Search method s/SearchRequest/SearchResponse Missing SearchAgain method
19. Imports the Go compiled version of the proto spec file in “protos”. Symmetrically to how a SearchService server is created, on the compiled proto there’s a automatically generated method to create a SearchService client, taking only a connection, which has been deferred to be closed at the end of runtime. A Client in Go Search RPC SearchRequest Message/Struct SearchResponse Message/Struct
20. Imports the same Proto file (a modification of original hello world) that the server imports, but “compiled” into Python. Symmetrically to how a SearchService server is created, on the compiled proto there’s a automatically generated method to create a SearchService client. A Client in Python Search RPC SearchRequest Message/Struct SearchResponse Message/Struct
21. Go Client vs Python Client Client creation Message/Struct RPC Calling
22. Creating a gRPC server is fairly easy with all these concepts in mind. Define services on a proto (just name, what comes in, and what goes out, implementation is on whoever builds the server) and adequate data structures for messages (with the field names, types, which can be many kinds of numbers, strings, enums, lists, and substructures), its a nice and strict contract. Then compile that proto with 1 line in terminal, and import the compiled proto in any program you want to use. On the server side, create a generic gRPC server, and as many times as you want you can with just one function bind a service (remember though, that you must implement all the RPCs defined in each service). On the client side, use “dial” to the service location (address and port), and create a “client” structure with that connection (one method), and then you can start calling RPC methods directly on the structure. No need to do HTTP requests, because most of the low level details are abstracted, content types (thats implicit in the proto structures), parsing or validating the structure fields (proto structures are strict with the types so no need for that, maybe only validate if its legal values, but thats on the logic, not the structure). RPC TL;DR
23. https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/AmazonWebservices_Logo.svg/2000px-AmazonWebservices_Logo.svg.png https://www.modmypi.com/raspberry-pi/breakout-boards/raspberry-pi-(official)/raspberry-pi-sense-hat http://as-images.apple.com/is/image/AppleInc/aos/published/images/M/AC/MACBOOKPRO/MACBOOKPRO?wid=1200&hei=630 https://moki.com/wp-content/uploads/2014/12/iOSAndroidDevices.png Registering Website HTTP server GRPC server GET “/” GET “/” POST “/register” { firstName: ??? lastName: ??? code: ??? } GRPC “GetCode” message CodeRequest message CodeResponse GRPC “GetStatus” message StatusRequest message StatusResponse https://github.com/zubie7a/grpc-demo
24. https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/AmazonWebservices_Logo.svg/2000px-AmazonWebservices_Logo.svg.png https://www.modmypi.com/raspberry-pi/breakout-boards/raspberry-pi-(official)/raspberry-pi-sense-hat http://as-images.apple.com/is/image/AppleInc/aos/published/images/M/AC/MACBOOKPRO/MACBOOKPRO?wid=1200&hei=630 https://moki.com/wp-content/uploads/2014/12/iOSAndroidDevices.png Registering Website GET “/” GET “/” POST “/register” GRPC “GetCode” message CodeRequest message CodeResponse Give One-Time Registration Code to potential attendant. Generate a One-Time registration code on the server, and retrieve it to give personally to a potential attendant. Constantly poll the status of the attendants (whole app state, like registered attendant count and their names) Via GPIO display on the SenseHat display related to the attendance status. Use a provided One-Time Registration Code to register with name and last name, if code is invalid or has already been used then it will return an error message. message StatusRequest message StatusResponse GRPC “GetStatus” 1 2 3 ∞ HTTP server GRPC server https://github.com/zubie7a/grpc-demo
25. 1/ Generate and Retrieve Registatrion Code. HTTP server GRPC server GRPC “GetCode” message CodeRequest message CodeResponse ServerSide ClientSide Client will call an RPC to generate a One-Time Registration Code. This code is a unique and random 32 bit integer number, which can be handed to someone to register to attend. Also, the number is temporarily stored on the server so that it knows that the code inputted by someone when registering is valid. https://github.com/zubie7a/grpc-demo github.com/zubie7a/grpc-demo/go_client/main.go GRPC Go Client Location {server_ip_address}:50001 GRPC Server URL
26. 1/ Give One-Time Registration Code to Attendant Via whatever means, a flyer/voucher, in person, in a front desk, etc, give a registration code to an attendant. Remind them that it is one-time use only, and that upon losing it it cannot be retrieved back. The code will be 979509393 Give One-Time Registration Code to Potential Attendant https://github.com/zubie7a/grpc-demo github.com/zubie7a/grpc-demo/go_client/main.go GRPC Go Client Location {server_ip_address}:50001 GRPC Server URL
27. 3/ Use One-Time Registration Code to Register Attendance Code: 979509393 HTTP server GRPC server GET “/” GET “/” POST “/register” { firstName: ??? lastName: ??? code: ??? } The potential attendants (those who were handed a code) will visit a WebSite in which they will register themselves with their names, and their unique code. If the code is invalid, or has already been used, then a series of errors will be presented, otherwise, registration will go on successfully. https://github.com/zubie7a/grpc-demo {server_ip_address}:8080 github.com/zubie7a/grpc-demo/main.go GRPC + HTTP Go Server Location HTTP Server URL
28. 3/ Use One-Time Registration Code to Register Attendance Any of the fields is missing, server will reply: Please fill all fields. https://github.com/zubie7a/grpc-demo
29. 3/ Use One-Time Registration Code to Register Attendance If it goes right: Successfully registered! https://github.com/zubie7a/grpc-demo
30. 3/ Use One-Time Registration Code to Register Attendance If the code has already been used: The code 979509393 has already been used. https://github.com/zubie7a/grpc-demo
31. 3/ Use One-Time Registration Code to Register Attendance If the code hasn’t been generated by the server: The code {code} is not a valid code. haha so funny this won’t work Lets generate a new code so this Bobby Tables can register https://github.com/zubie7a/grpc-demo
32. 3/ Use One-Time Registration Code to Register Attendance Three attendants have successfully registered, and their names have been listed on the DropDown menu. This has been done via HTTP requests, and its been kinda messy because every time we have to worry about particular details like POST/GET handlers, building a JSON with the correct structure, parsing it manually back and forth, etc. GRPC provides the appropriate abstractions so that we don’t have to worry about most of this, we get message structures ready and we know what goes out and what comes in, and its very extensible regarding auth, middlewares, load balancing, etc. I told you it wouldn’t work https://github.com/zubie7a/grpc-demo
33. Constantly poll the status of the attendants (whole app state, like registered attendant count and their names) Via GPIO display on the SenseHat display related to the attendance status. message StatusRequest message StatusResponse GRPC “GetStatus” ∞ HTTP server GRPC server ∞/ Constantly Poll Via RPC To Monitor Attendant Status https://www.modmypi.com/raspberry-pi/breakout-boards/raspberry-pi-(official)/raspberry-pi-sense-hat https://github.com/zubie7a/grpc-demo GRPC Python Client + SenseHat Location github.com/zubie7a/grpc-demo/python_client/sensehat_py
34. https://www.youtube.com/watch?v=DSW_CQc_Wi4 Thank you all for coming to the Microservices Architecture presentation by @c4milo and the GRPC presentation by @zubie7a Remember to check out @scaleconfco for announcements! https://github.com/zubie7a/grpc-demo