How Checkr uses gRPC [RailsConf 2019]

How Checkr uses gRPC [RailsConf 2019]

Checkr’s mission is to build a fairer future by improving understanding of the past. We are based in San Francisco and Denver. We have found some limitations in only using JSON-based RESTful APIs to communicate between services. Moving to gRPC has allowed us to better document interfaces and enforce stronger boundaries between services. In this session we will share the lessons we have learned while incorporating gRPC into our application, walkthrough setting up a new rails project with gRPC support, and how we plan to expand our usage of gRPC in the future.

Ea01dcd1fcd0e3c2f54e82d5c1308a28?s=128

Paul Zaich

April 30, 2019
Tweet

Transcript

  1. 6.

    At Checkr, the number of internal endpoints and services has

    grown over the years Our monolith currently has 500+ endpoints and 20+ other production services in critical path
  2. 7.

    Checkr is an API first background check company We provide

    modern and compliant background checks for 1,000s of global enterprises and startups
  3. 9.

    This talk will cover: 1. What is gRPC? 2. Tradeoffs

    of using it vs REST 3. Walkthrough an example
  4. 11.

    “Remote Procedure Call (RPC) is a protocol that one program

    can use to request a service from a program located in another computer on a network without having to understand the network's details” - Wikipedia Service A Service B I want you to do a thing! Ok! Here is the result of doing the thing!
  5. 13.

    paths: /stocks/{stock_name}: get: description: | Return stock price operationId: findStock

    parameters: - name: stock_name in: path description: stock ticker symbol required: true schema: type: string responses: '200': description: Stock price content: application/json: schema: $ref: '#/components/schemas/Stock' Swagger
  6. 16.

    // gRPC IDL (Interface Definition Language) service Stocks { rpc

    getStockPrice (StockPriceRequest) returns (StockPriceResponse); } message StockPriceRequest { string stockName = 1; } message StockPriceResponse { string stockName = 1; float price = 2; } gRPC
  7. 17.

    Overview IDL Interface Definition Language: Detailed description of your service

    Implementation How you actually implement the IDL Protocol How the implementations talk to each other Documentation How you communicate the IDL/implementations to your team
  8. 18.

    REST SOAP gRPC IDL OpenAPI / Swagger WSDL Proto Definition

    Implementation Swagger Codegen / HTTParty Savon / Zeep gRPC Codegen Protocol JSON / XML HTTP XML HTTP Protocol Buffer HTTP/2 Documentation Swagger UI / Slate wsdl2html protoc-doc-gen
  9. 29.
  10. 34.

    Service A Service B @#$%&! Service C Service D Service

    E Service F Service G Service H Service I Service J
  11. 36.

    // Like a schema.rb for your services service Stocks {

    rpc getStockPrice (StockPriceRequest) returns (StockPriceResponse); } message StockPriceRequest { string stockName = 1; } message StockPriceResponse { string stockName = 1; float price = 2; } gRPC
  12. 37.

    IDL Repo service Stocks { rpc getStockPrice (StockPriceRequest) returns (StockPriceResponse);

    } message StockPriceRequest { string stockName = 1; } message StockPriceResponse { string stockName = 1; float price = 2; }
  13. 41.

    Service A Python Client Service B Ruby Server I want

    to call getStockPrice StockPriceRequest StockPriceResponse
  14. 42.

    IDL Repo gRPC Tooling Ruby Code Server create_table :users do

    ... rails db:migrate class User < ApplicationRecord @user.save! Mapping these concepts to stuff we already do!
  15. 44.

    Continue to use JSON Definitely switch to gRPC Time Checkr’s

    journey to gRPC Over time we started to feel pain with all of our endpoints and services. We wanted to move to more microservices so our platform team started to introduce gRPC into critical path services
  16. 45.

    Continue to use JSON Definitely switch to gRPC Mythical point

    that for sure means you should switch everything to gRPC right now Checkr’s journey to gRPC
  17. 46.

    Continue to use JSON Definitely switch to gRPC Gray Area

    In reality it looks like this Checkr’s journey to gRPC
  18. 47.

    Continue to use JSON Definitely switch to gRPC This is

    where Checkr is today Checkr’s journey to gRPC
  19. 51.

    service NameMatcher { rpc match (MatchRequest) returns (MatchResponse); } message

    PersonName { string first_name = 1; string middle_name = 2; string last_name = 3; } message MatchRequest { PersonName name_a = 1; PersonName name_b = 2; } message MatchResponse { bool match = 1; float confidence = 2; } Service definition
  20. 52.

    My New Feature GitHub CircleCI Ruby (via Gemfury) Python Javascript

    PR is reviewed on GitHub and merged into IDL repo’s master branch An update is proposed to the IDL to support new functionality Builds and publishes language specific libraries HTML Documentation
  21. 53.
  22. 57.

    class NameMatcherService < Idl::NameMatcher::Service def match(match_request) # compare names Idl::NameMatcher::MatchResponse.new(

    match: result.match, confidence: result.confidence ) end end A simple grpc Ruby server
  23. 59.

    # app/rpc/match_controller.rb class MatchController < ::Gruf::Controllers::Base bind Idl::NameMatcher::Service def match

    message = request.message # compare names here... Idl::NameMatcher::MatchResponse.new( match: result.match, confidence: result.confidence ) end end $ bundle exec gruf Serve gPRC in Rails Open Source Gem
  24. 63.

    # client.rb > name.first = "Obi" NoMethodError: undefined method `first='

    for #<Idl::PersonName:0x000055cda27e2af0> from (irb):16:in `method_missing' Message definitions in Ruby
  25. 64.

    # client.rb > Idl::PersonName.new("Obi Wan Kenobi") ArgumentError: Unknown field name

    'name' in initialization map entry. > Idl::PersonName.new(['Obi','Wan', 'Kenobi']) ArgumentError: Expected hash arguments. Message definitions in Ruby
  26. 65.

    # client.rb > name_a = Idl::PersonName.new(first_name: 'Obi', middle_name: 'Wan', last_name:

    'Kenobi') > name_b = Idl::PersonName.new(first_name: 'Ben', last_name: 'Kenobi') Message definitions in Ruby
  27. 67.

    # client.rb > response = client.match(name_a: name_a, name_b: name_b) >

    response.match # true > response.confidence # 99.1 gRPC request
  28. 68.

    name = Idl::PersonName.new(first_name: 'Obi-wan', last_name: 'Kenobi') encoded_name = Idl::PersonName.encode(name) AsyncEnqueur.publish(queue:

    'name-analyzer', encoded_name) class AsyncWorker def perform(encoded_name) name = Idl::PersonName.decode(encoded_name) # do work end end Bonus: Use Definitions for all service communication
  29. 73.

    # server.rb errors = { middle_names: ['Must be present'] }

    raise GRPC::BadStatus.new(GRPC::Core::StatusCodes::INVALID_ARGUMENT, errors.to_json) Error messages
  30. 74.

    # server.rb validation_error = IDL::ValidationError.new( name: 'middle_names', message: 'must be

    present.' ) raise GRPC::BadStatus.new( GRPC::Core::StatusCodes::INVALID_ARGUMENT, 'Validation failed', IDL::ValidationError.encode(validation_error) ) Error messages
  31. 77.
  32. 81.

    Gruf: Streamlined gRPC integration with Rails by BigCommerce https://github.com/bigcommerce/gruf gRPC:

    Open-source universal RPC framework by Google https://grpc.io/ Protoc Gen Doc: Generate documentation from protocol definitions https://github.com/pseudomuto/protoc-gen-doc Shout out to open source!