Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Checkr uses gRPC [RailsConf 2019]

How Checkr uses gRPC [RailsConf 2019]

Checkr’s mission is to build a fairer future by improving understanding of the past. We are based in San Francisco and Denver. We have found some limitations in only using JSON-based RESTful APIs to communicate between services. Moving to gRPC has allowed us to better document interfaces and enforce stronger boundaries between services. In this session we will share the lessons we have learned while incorporating gRPC into our application, walkthrough setting up a new rails project with gRPC support, and how we plan to expand our usage of gRPC in the future.

Ea01dcd1fcd0e3c2f54e82d5c1308a28?s=128

Paul Zaich

April 30, 2019
Tweet

Transcript

  1. How Checkr uses gRPC RailsConf 2019

  2. Audience Survey

  3. Have you ever worked with an API without documentation?

  4. Have you ever worked with an API with inaccurate documentation?

  5. Have you ever worked with an API that returns inconsistent

    types?
  6. At Checkr, the number of internal endpoints and services has

    grown over the years Our monolith currently has 500+ endpoints and 20+ other production services in critical path
  7. Checkr is an API first background check company We provide

    modern and compliant background checks for 1,000s of global enterprises and startups
  8. Paul Zaich Ben Jacobson

  9. This talk will cover: 1. What is gRPC? 2. Tradeoffs

    of using it vs REST 3. Walkthrough an example
  10. What is RPC?

  11. “Remote Procedure Call (RPC) is a protocol that one program

    can use to request a service from a program located in another computer on a network without having to understand the network's details” - Wikipedia Service A Service B I want you to do a thing! Ok! Here is the result of doing the thing!
  12. <?xml version="1.0"?> <soap:Envelope> <soap:Header> </soap:Header> <soap:Body> <m:GetStockPrice> <m:StockName>GOOG</m:StockName> </m:GetStockPrice> </soap:Body>

    </soap:Envelope> SOAP
  13. paths: /stocks/{stock_name}: get: description: | Return stock price operationId: findStock

    parameters: - name: stock_name in: path description: stock ticker symbol required: true schema: type: string responses: '200': description: Stock price content: application/json: schema: $ref: '#/components/schemas/Stock' Swagger
  14. But what is gRPC?

  15. Open source framework by Google to help make RPC, in

    any language, easy
  16. // gRPC IDL (Interface Definition Language) service Stocks { rpc

    getStockPrice (StockPriceRequest) returns (StockPriceResponse); } message StockPriceRequest { string stockName = 1; } message StockPriceResponse { string stockName = 1; float price = 2; } gRPC
  17. Overview IDL Interface Definition Language: Detailed description of your service

    Implementation How you actually implement the IDL Protocol How the implementations talk to each other Documentation How you communicate the IDL/implementations to your team
  18. REST SOAP gRPC IDL OpenAPI / Swagger WSDL Proto Definition

    Implementation Swagger Codegen / HTTParty Savon / Zeep gRPC Codegen Protocol JSON / XML HTTP XML HTTP Protocol Buffer HTTP/2 Documentation Swagger UI / Slate wsdl2html protoc-doc-gen
  19. gRPC is an opinionated framework for achieving RPC Convention over

    configuration!
  20. Should you be using gRPC?

  21. Probably not.

  22. REST is GREAT!

  23. render json: @stuff

  24. REST is FLEXIBLE

  25. render json: StuffSerializer.new(@stuff)

  26. JSON is READABLE. EVERYWHERE

  27. // Protocol Buffer \n\x06John\x12\x05Smith\ "\x0410/30/1987(\x02 // JSON { "first_name": "John",

    "last_name": "Smith", "dob": "10/30/1987", }
  28. You should probably use REST

  29. But...

  30. JSON APIs start to feel like searching in the dark

    Service B Service A
  31. Service A Service B Hmm.. what are all the things

    you can do?
  32. Things get harder as you scale

  33. Service A Service B Oh boy... Service C Service D

  34. Service A Service B @#$%&! Service C Service D Service

    E Service F Service G Service H Service I Service J
  35. gRPC provides an approach to solve this problem

  36. // Like a schema.rb for your services service Stocks {

    rpc getStockPrice (StockPriceRequest) returns (StockPriceResponse); } message StockPriceRequest { string stockName = 1; } message StockPriceResponse { string stockName = 1; float price = 2; } gRPC
  37. IDL Repo service Stocks { rpc getStockPrice (StockPriceRequest) returns (StockPriceResponse);

    } message StockPriceRequest { string stockName = 1; } message StockPriceResponse { string stockName = 1; float price = 2; }
  38. IDL Repo gRPC Tooling

  39. IDL Repo gRPC Tooling Ruby Python

  40. IDL Repo gRPC Tooling Ruby Python Service A (Client) Service

    B (Server)
  41. Service A Python Client Service B Ruby Server I want

    to call getStockPrice StockPriceRequest StockPriceResponse
  42. IDL Repo gRPC Tooling Ruby Code Server create_table :users do

    ... rails db:migrate class User < ApplicationRecord @user.save! Mapping these concepts to stuff we already do!
  43. Our journey to gRPC

  44. Continue to use JSON Definitely switch to gRPC Time Checkr’s

    journey to gRPC Over time we started to feel pain with all of our endpoints and services. We wanted to move to more microservices so our platform team started to introduce gRPC into critical path services
  45. Continue to use JSON Definitely switch to gRPC Mythical point

    that for sure means you should switch everything to gRPC right now Checkr’s journey to gRPC
  46. Continue to use JSON Definitely switch to gRPC Gray Area

    In reality it looks like this Checkr’s journey to gRPC
  47. Continue to use JSON Definitely switch to gRPC This is

    where Checkr is today Checkr’s journey to gRPC
  48. Walkthrough

  49. Name Matcher Person Matcher I can help you match names!

    QA Testing Name Commonality
  50. Writing the IDL

  51. service NameMatcher { rpc match (MatchRequest) returns (MatchResponse); } message

    PersonName { string first_name = 1; string middle_name = 2; string last_name = 3; } message MatchRequest { PersonName name_a = 1; PersonName name_b = 2; } message MatchResponse { bool match = 1; float confidence = 2; } Service definition
  52. My New Feature GitHub CircleCI Ruby (via Gemfury) Python Javascript

    PR is reviewed on GitHub and merged into IDL repo’s master branch An update is proposed to the IDL to support new functionality Builds and publishes language specific libraries HTML Documentation
  53. None
  54. # Gemfile source 'https://my.gem.server/' do gem 'checkr-idl', '0.1.293' End $

    bundle install Install the IDL library
  55. # my/requirements.txt checkrIdl=0.1.293 $ pip install -r requirements.txt Import in

    another supported language
  56. Serving gRPC

  57. class NameMatcherService < Idl::NameMatcher::Service def match(match_request) # compare names Idl::NameMatcher::MatchResponse.new(

    match: result.match, confidence: result.confidence ) end end A simple grpc Ruby server
  58. s = GRPC::RpcServer.new s.add_http2_port('localhost:9001') s.handle(NameMatcherService.new) s.run_till_terminated Starting the server

  59. # app/rpc/match_controller.rb class MatchController < ::Gruf::Controllers::Base bind Idl::NameMatcher::Service def match

    message = request.message # compare names here... Idl::NameMatcher::MatchResponse.new( match: result.match, confidence: result.confidence ) end end $ bundle exec gruf Serve gPRC in Rails Open Source Gem
  60. Making a gPRC request

  61. > client = Idl::NameMatcher::Stub.new( 'name-matcher.checkr.com') Connect your client

  62. # client.rb > Idl::PersonName.new => <Idl::PersonName: first_name: "", middle_name: "",

    last_name: ""> Message definitions in Ruby
  63. # client.rb > name.first = "Obi" NoMethodError: undefined method `first='

    for #<Idl::PersonName:0x000055cda27e2af0> from (irb):16:in `method_missing' Message definitions in Ruby
  64. # client.rb > Idl::PersonName.new("Obi Wan Kenobi") ArgumentError: Unknown field name

    'name' in initialization map entry. > Idl::PersonName.new(['Obi','Wan', 'Kenobi']) ArgumentError: Expected hash arguments. Message definitions in Ruby
  65. # client.rb > name_a = Idl::PersonName.new(first_name: 'Obi', middle_name: 'Wan', last_name:

    'Kenobi') > name_b = Idl::PersonName.new(first_name: 'Ben', last_name: 'Kenobi') Message definitions in Ruby
  66. # client.rb > client.match(name_a: name_a, name_b: name_b) gRPC request

  67. # client.rb > response = client.match(name_a: name_a, name_b: name_b) >

    response.match # true > response.confidence # 99.1 gRPC request
  68. name = Idl::PersonName.new(first_name: 'Obi-wan', last_name: 'Kenobi') encoded_name = Idl::PersonName.encode(name) AsyncEnqueur.publish(queue:

    'name-analyzer', encoded_name) class AsyncWorker def perform(encoded_name) name = Idl::PersonName.decode(encoded_name) # do work end end Bonus: Use Definitions for all service communication
  69. Throwing and Handling Errors

  70. gRPC Status Codes

  71. 418 I'M A TEAPOT

  72. # server.rb def match(match_request) unless [name_a.middle_name, name_b.middle_name].all?(&:present?) raise GRPC::BadStatus.new( GRPC::Core::StatusCodes::INVALID_ARGUMENT,

    "Middle Name must be present." ) end end Throw an error
  73. # server.rb errors = { middle_names: ['Must be present'] }

    raise GRPC::BadStatus.new(GRPC::Core::StatusCodes::INVALID_ARGUMENT, errors.to_json) Error messages
  74. # server.rb validation_error = IDL::ValidationError.new( name: 'middle_names', message: 'must be

    present.' ) raise GRPC::BadStatus.new( GRPC::Core::StatusCodes::INVALID_ARGUMENT, 'Validation failed', IDL::ValidationError.encode(validation_error) ) Error messages
  75. # client.rb begin ... rescue GRPC::BadStatus => e e.details e.message

    e.code end Client-side Handling
  76. # client.rb begin ... rescue GRPC::InvalidArgument => e e.details e.message

    e.code end Client-side Handling
  77. Takeaways

  78. REST can take you a long way

  79. gPRC is an opinionated framework for service communication

  80. gRPC enforces contract-driven development

  81. Gruf: Streamlined gRPC integration with Rails by BigCommerce https://github.com/bigcommerce/gruf gRPC:

    Open-source universal RPC framework by Google https://grpc.io/ Protoc Gen Doc: Generate documentation from protocol definitions https://github.com/pseudomuto/protoc-gen-doc Shout out to open source!
  82. Thanks! Shameless Plug! Checkr is hiring! Offices in San Francisco

    and Denver