GRPC with Golang and Python

GRPC Demo and Presentation for Medellin DevOps + Golang Medellin Meetups involving an AWS server running a GRPC and HTTP Server, a local GRPC Golang client on a computer, and a local GRPC Python client running on a Raspberry Pi which displays retrieved information on a SenseHat via GPIO.

Transcript

1. Santiago Martin Zubieta Ortiz
   Student at Universidad EAFIT
   Unemployed... for now :-P
   With Golang
   and Python.
   

   View Slide

2. http://www.grpc.io/about/
   What’s it?
   “gRPC is a modern open source high performance RPC
   framework that can run in any environment. It can
   efficiently connect services in and across data centers
   with pluggable support for load balancing, tracing, health
   checking and authentication. It is also applicable in last
   mile of distributed computing to connect devices, mobile
   applications and browsers to backend services.”
   

   View Slide

3. What’s it good for?
   The main usage scenarios:
   • Efficiently connecting polyglot services in microservices style
   architecture
   • Connecting mobile devices, browser clients to backend services
   • Generating efficient client libraries
   Core Features that make it awesome:
   • Idiomatic client libraries in 10 languages
   • Highly efficient on wire and with a simple service definition framework
   • Bi-directional streaming with http/2 based transport
   • Pluggable auth, tracing, load balancing and health checking
   http://www.grpc.io/docs/
   

   View Slide

4. http://www.grpc.io/docs/
   Much languages!
   Such support.
   Very nice.
   phpthulu
   Y U NO SWIFT THO
   https://www.gopheracademy.com/
   http://knowyourmeme.com/memes/gadsden-flag-don-t-tread-on-me
   http://lovecraft.wikia.com/wiki/Cthulhu
   

   View Slide

5. Works across languages and platforms
   To understand gRPC we need to understand first what protos are.
   http://www.grpc.io/
   Though JSON can also be used.
   

   View Slide

6. Whats a “Proto” anyway?
   Think of a JSON but much more powerful.
   https://developers.google.com/protocol-buffers/
   Protocol buffers are a language-neutral, platform-neutral
   extensible mechanism for serializing structured data.
   Structural definition, imported in both ends.
   Creation on Java side. Parsing on C++ side.
   $ protoc will automatically convert proto file to language-friendly ready-to-import equivalent.
   

   View Slide

7. A message, its what can be instantiated into a structure on any language. Defining this
   structure is like a neutral way of defining a structure with its properties, then the proto compiler
   will generate for each respective language a import-friendly, language specific version of that
   structure. So several languages can communicate using the same structures defined in the
   proto. Whole classes and services can be defined this way, all fulfilling the same contract.
   Also RPCs will be restricted to certain message types, unlike JSON where something with any
   structure can come in or go out without knowing before hand what’s it internal structure and
   having to validate the structure both when the server receives a request and when the client
   receives an answer.
   Proto3 Message Basics
   https://developers.google.com/protocol-buffers/docs/proto3
   

   View Slide

8. There’s a lot of types what a proto can contain
   and control over that. There’s enums, strings,
   booleans, floating point numbers of different
   precision, maps with both the keys and values
   defined, timestamps or durations, any kind of
   variable length binary object, and even lists of
   objects with known types. It can even contain
   other messages inside!
   Proto3 Message Basics
   https://developers.google.com/protocol-buffers/docs/proto3
   

   View Slide

9. https://github.com/google/protobuf/releases
   protoc --proto_path=IMPORT_PATH \
   --cpp_out=DST_DIR \
   --java_out=DST_DIR \
   --python_out=DST_DIR \
   --go_out=DST_DIR \
   --ruby_out=DST_DIR \
   --javanano_out=DST_DIR \
   --objc_out=DST_DIR \
   --csharp_out=DST_DIR \
   path/to/file.proto
   The proto compiler will automatically
   transpile a .proto or several ones into
   files that can be imported from a program
   in many supported languages, ready to
   be imported, and its structs and services
   ready to be instantiated and implemented.
   $ brew install protobuf :-) (building from source is fine too)
   Proto3 Building Basics
   protoc -I=$SRC_DIR $SRC_DIR/{proto_name}.proto --go_out=$DST_DIR
   protoc -I protos/ protos/services.proto --go_out=plugins=grpc:protos
   $ python -m pip install grpcio
   $ python -m pip install grpcio-tools
   $ go get google.golang.org/grpc
   $ go get -u github.com/golang/protobuf/{proto,protoc-gen-go}
   

   View Slide

10. PhoneType enum only exists in scope of Person
    PhoneNumber message only exists in
    scope of Person
    message definition
    enum definition
    nested
    message definition
    nested
    A “list” of PhoneNumber
    messages
    A person has a name, id, email,
    and a list of phone numbers.
    An AddressBook has a list of people (each one an
    instance of Person), each of which has a Name, an Id,
    an Email, and a list of phones (each one an instance of
    PhoneNumber), each of which has a Number, and a
    PhoneType (MOBILE, HOME, WORK).
    A whole structure has been defined, ready to be
    used in any language!
    Proto3 Message Basics
    https://developers.google.com/protocol-buffers/docs/proto3
    

    View Slide

11. A bunch of code is generated from sample.proto into
    sample.pb.go! You shouldn’t modify this code, it has
    a lot of confusing stuff, many handlers, and
    definitions, but what protoc did was transpile the
    previous high level proto definition into the equivalent
    in the Go language.
    Go-Transpiled Proto
    

    View Slide

12. https://developers.google.com/protocol-buffers/docs/proto3
    A service, contains a definition of an RPC, or several ones. All RPC defined
    within a service, must be implemented in whichever program imports this
    protobuffer file and instantiates a server based on this definition, so its like a
    contract.
    It works in terms of the name of the RPC, the kind of message it receives,
    and the kind of message it returns. This way, it knows beforehand what kind
    of structure must receive, and the caller has an assurance that either known
    structure will be returned, or an error for whatever cause.
    Proto3 Service Basics
    

    View Slide

13. When the built proto file is imported in its respective language, one can
    instantiate freely its structs/messages. But when a server is instantiated
    using the specification given in the protobuffer for a given service, then all
    RPCs must be implemented with the appropriate signature.
    A standard RPC server.
    The imported proto.
    Register{ServiceName}Server method is automatically generated for all
    {ServiceNames}. What this method does is bind RPCs from a given
    service (with a struct that implements them) on a generic GRPC server.
    The generic RPC server. A struct defined by us, which needs
    to implement the SearchService
    interface, that is, the defined RPCs.
    Service Implementation
    

    View Slide

14. ???????
    

    View Slide

15. Server Launch
    Service Interface
    Implementation
    Proto Definition
    main.go
    search.proto
    

    View Slide

16. !!!!!!!!
    

    View Slide

17. !!!!!!!!
    

    View Slide

18. Services are then a group of RPCs (in this case, SearchService, with
    Search and SearchAgain RPCs), and such services defined in the proto
    can be thought of as a contract, if you will create a server using any
    service, you must implement ALL the RPCs outlined in the service.
    Lets change to not match signature
    Lets erase this so its missing from
    the server implementation.
    Services Are Contracts
    Wrong type for Search method s/SearchRequest/SearchResponse
    Missing SearchAgain method
    

    View Slide

19. Imports the Go compiled version of the
    proto spec file in “protos”.
    Symmetrically to how a SearchService server
    is created, on the compiled proto there’s a
    automatically generated method to create a
    SearchService client, taking only a
    connection, which has been deferred to be
    closed at the end of runtime.
    A Client in Go
    Search RPC
    SearchRequest Message/Struct
    SearchResponse
    Message/Struct
    

    View Slide

20. Imports the same Proto file (a modification of
    original hello world) that the server imports, but
    “compiled” into Python.
    Symmetrically to how a SearchService
    server is created, on the compiled proto
    there’s a automatically generated
    method to create a SearchService client.
    A Client in Python
    Search RPC
    SearchRequest Message/Struct
    SearchResponse
    Message/Struct
    

    View Slide

21. Go Client vs Python Client
    Client creation
    Message/Struct
    RPC Calling
    

    View Slide

22. Creating a gRPC server is fairly easy with all these concepts in mind. Define services
    on a proto (just name, what comes in, and what goes out, implementation is on
    whoever builds the server) and adequate data structures for messages (with the field
    names, types, which can be many kinds of numbers, strings, enums, lists, and
    substructures), its a nice and strict contract. Then compile that proto with 1 line in
    terminal, and import the compiled proto in any program you want to use.
    On the server side, create a generic gRPC server, and as many times as you want
    you can with just one function bind a service (remember though, that you must
    implement all the RPCs defined in each service).
    On the client side, use “dial” to the service location (address and port), and create a
    “client” structure with that connection (one method), and then you can start calling
    RPC methods directly on the structure.
    No need to do HTTP requests, because most of the low level details are abstracted,
    content types (thats implicit in the proto structures), parsing or validating the structure
    fields (proto structures are strict with the types so no need for that, maybe only
    validate if its legal values, but thats on the logic, not the structure).
    RPC TL;DR
    

    View Slide

23. https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/AmazonWebservices_Logo.svg/2000px-AmazonWebservices_Logo.svg.png
    https://www.modmypi.com/raspberry-pi/breakout-boards/raspberry-pi-(official)/raspberry-pi-sense-hat http://as-images.apple.com/is/image/AppleInc/aos/published/images/M/AC/MACBOOKPRO/MACBOOKPRO?wid=1200&hei=630
    https://moki.com/wp-content/uploads/2014/12/iOSAndroidDevices.png
    Registering Website
    HTTP server
    GRPC server
    GET “/” GET “/”
    POST “/register”
    {
    firstName: ???
    lastName: ???
    code: ???
    }
    GRPC “GetCode”
    message CodeRequest
    message CodeResponse
    GRPC “GetStatus”
    message StatusRequest
    message StatusResponse
    https://github.com/zubie7a/grpc-demo
    

    View Slide

24. https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/AmazonWebservices_Logo.svg/2000px-AmazonWebservices_Logo.svg.png
    https://www.modmypi.com/raspberry-pi/breakout-boards/raspberry-pi-(official)/raspberry-pi-sense-hat http://as-images.apple.com/is/image/AppleInc/aos/published/images/M/AC/MACBOOKPRO/MACBOOKPRO?wid=1200&hei=630
    https://moki.com/wp-content/uploads/2014/12/iOSAndroidDevices.png
    Registering Website
    GET “/” GET “/”
    POST “/register”
    GRPC “GetCode”
    message CodeRequest
    message CodeResponse
    Give One-Time Registration
    Code to potential attendant.
    Generate a One-Time registration
    code on the server, and retrieve it
    to
    give
    personally
    to
    a
    potential
    attendant.
    Constantly poll the status of the
    attendants (whole app state, like
    registered attendant count and
    their names)
    Via GPIO display on the SenseHat
    display related to the attendance
    status.
    Use a provided One-Time Registration Code
    to register with name and last name, if code
    is invalid or has already been used then it
    will return an error message.
    message StatusRequest
    message StatusResponse
    GRPC “GetStatus”
    1
    2
    3
    ∞
    HTTP server
    GRPC server
    https://github.com/zubie7a/grpc-demo
    

    View Slide

25. 1/ Generate and Retrieve Registatrion Code.
    HTTP server
    GRPC server
    GRPC “GetCode”
    message CodeRequest
    message CodeResponse
    ServerSide
    ClientSide
    Client will call an RPC to generate a
    One-Time Registration Code. This code
    is a unique and random 32 bit integer
    number, which can be handed to
    someone to register to attend. Also,
    the number is temporarily stored on
    the server so that it knows that the
    code inputted by someone when
    registering is valid.
    https://github.com/zubie7a/grpc-demo
    github.com/zubie7a/grpc-demo/go_client/main.go
    GRPC Go Client Location
    {server_ip_address}:50001
    GRPC Server URL
    

    View Slide

26. 1/ Give One-Time Registration Code to Attendant
    Via whatever means, a flyer/voucher, in person,
    in a front desk, etc, give a registration code to
    an attendant. Remind them that it is one-time
    use only, and that upon losing it it cannot be
    retrieved back. The code will be 979509393
    Give One-Time Registration Code to Potential Attendant
    https://github.com/zubie7a/grpc-demo
    github.com/zubie7a/grpc-demo/go_client/main.go
    GRPC Go Client Location
    {server_ip_address}:50001
    GRPC Server URL
    

    View Slide

27. 3/ Use One-Time Registration Code to Register Attendance
    Code: 979509393
    HTTP server
    GRPC server
    GET “/” GET “/”
    POST “/register”
    {
    firstName: ???
    lastName: ???
    code: ???
    }
    The potential attendants (those who were handed a code)
    will visit a WebSite in which they will register themselves
    with their names, and their unique code. If the code is
    invalid, or has already been used, then a series of errors
    will be presented, otherwise, registration will go on
    successfully.
    https://github.com/zubie7a/grpc-demo
    {server_ip_address}:8080
    github.com/zubie7a/grpc-demo/main.go
    GRPC + HTTP Go Server Location
    HTTP Server URL
    

    View Slide

28. 3/ Use One-Time Registration Code to Register Attendance
    Any of the fields is missing, server will reply: Please fill all fields.
    https://github.com/zubie7a/grpc-demo
    

    View Slide

29. 3/ Use One-Time Registration Code to Register Attendance
    If it goes right: Successfully registered!
    https://github.com/zubie7a/grpc-demo
    

    View Slide

30. 3/ Use One-Time Registration Code to Register Attendance
    If the code has already been used: The code 979509393 has already been used.
    https://github.com/zubie7a/grpc-demo
    

    View Slide

31. 3/ Use One-Time Registration Code to Register Attendance
    If the code hasn’t been generated by the server: The code {code} is not a valid code.
    haha so funny this won’t work
    Lets generate a new code so this Bobby Tables can register
    https://github.com/zubie7a/grpc-demo
    

    View Slide

32. 3/ Use One-Time Registration Code to Register Attendance
    Three attendants have successfully registered, and their names have been listed on the DropDown menu.
    This has been done via HTTP requests, and its been kinda messy because every time
    we have to worry about particular details like POST/GET handlers, building a JSON
    with the correct structure, parsing it manually back and forth, etc. GRPC provides the
    appropriate abstractions so that we don’t have to worry about most of this, we get
    message structures ready and we know what goes out and what comes in, and its
    very extensible regarding auth, middlewares, load balancing, etc.
    I told you it wouldn’t work
    https://github.com/zubie7a/grpc-demo
    

    View Slide

33. Constantly poll the status of the
    attendants (whole app state, like
    registered attendant count and
    their names)
    Via GPIO display on the SenseHat
    display related to the attendance
    status.
    message StatusRequest
    message StatusResponse
    GRPC “GetStatus”
    ∞
    HTTP server
    GRPC server
    ∞/ Constantly Poll Via RPC To Monitor Attendant Status
    https://www.modmypi.com/raspberry-pi/breakout-boards/raspberry-pi-(official)/raspberry-pi-sense-hat
    https://github.com/zubie7a/grpc-demo
    GRPC Python Client + SenseHat Location
    github.com/zubie7a/grpc-demo/python_client/sensehat_py
    

    View Slide

34. https://www.youtube.com/watch?v=DSW_CQc_Wi4
    Thank you all for coming to the Microservices Architecture presentation by @c4milo
    and the GRPC presentation by @zubie7a
    Remember to check
    out @scaleconfco for
    announcements!
    https://github.com/zubie7a/grpc-demo
    

    View Slide