Best practices in web API client development #RubyKaigi

2b680c5f22146e764e2998dd4595ec93?s=47 sue445
April 20, 2019

Best practices in web API client development #RubyKaigi

2b680c5f22146e764e2998dd4595ec93?s=128

sue445

April 20, 2019
Tweet

Transcript

  1. Best practices in web API client development RubyKaigi 2019 /

    #rubykaigi #rubykaigiD pixiv Inc. sue445 2019.4.20
  2. Hello! 2

  3. My name is Go 3

  4. 4 About sue445 • Name: Go Sueyoshi ◦ Go =

    剛 = strong • a.k.a. sue445 ◦ https://twitter.com/sue445 ◦ https://github.com/sue445 sue445
  5. 5 About sue445 • Birthday: 1982/04/07 ◦ Last week was

    my birthday ◦ age 36 -> 37 sue445
  6. 6 Experience • Ruby: 7 years • Golang: 4 years

    • Go: 37 years (age) sue445
  7. 7 About sue445 • Birthplace: Fukuoka ◦ School days ◦

    〜 age 26 • Currently at: Tokyo ◦ pixiv Inc. sue445
  8. 8 About sue445 • Community ◦ Shibuya.rb ◦ Omotesando.rb ◦

    Shinjuku.rb ◦ Meguro.rb ◦ etc sue445
  9. 9 About sue445 • RubyKaja 2014 ◦ http://kaja.rubyist.net/2014/kaja ◦ https://www.slideshare.net/tyabe/ruby-kaja2014

    sue445
  10. • https://rubygems.org/profiles/sue445 ◦ e.g. rubicure (Ruby implementation of Pretty Cure)

    ▪ https://github.com/sue445/rubicure ◦ Maintainer of Itamae and chatwork-ruby 10 Gems I developed
  11. 11 About pixiv Inc.

  12. MAKE CREATIVITIES MORE ENJOYABLE 12 About pixiv Inc.

  13. • https://www.pixiv.co.jp/ • Place: Tokyo and Fukuoka • Silver Sponsor

    13 About pixiv Inc.
  14. 14 FAQ. Is pixiv Inc.’s main lang PHP?

  15. • Ruby is widely used at pixiv Inc, too 15

    FAQ. Is pixiv Inc.’s main lang PHP?
  16. 16 What I've been doing at pixiv Inc.

  17. • SUE (sue445) sounds like SRE (Site Reliability Engineer) ◦

    My job is almost like that of an SRE ▪ e.g. web application, library, infra, CI etc ... ◦ Ansible >>>> Terraform >>> Ruby ▪ BTW. I’m an Itamae maintainer 17 What I've been doing at pixiv Inc.
  18. • https://rubykaigi.org/2019/presentations/ota42y.html#apr18 • @ota42y was my colleague • I was

    his mentor when he was an intern (2012/08) 18 Me and @ota42y
  19. • OpenAPI3 is Bright side of API    (IMO :-P)

    ◦ There is a shining future • My talk is about Dark side of API ◦ Fight against reality 19   Bright side vs Dark side of API
  20. • How to not fail API client development ◦ Good

    architecture and best practice • Architecture and best practices in regular gem development ◦ Architecture techniques in API clients also help the other gem development 20 The Goal/Objective of this talk
  21. • Why do I make API clients? • Responsibilities of

    API clients • My API clients and Tips • 7 Good Patterns of API clients 21 Agenda
  22. 22 Topics covered today App Web APIs API client

  23. • New technologies ◦ e.g. OpenAPI, microservices • API servers

    • non-text-based APIs ◦ APIs of which both requests and responses aren't in text format ◦ e.g. Protocol Buffers • non web APIs 23 Topics NOT covered today
  24. • -> Why do I make API clients? • Responsibilities

    of API clients • My API clients and Tips • 7 Good Patterns of API clients 24 Agenda
  25. • 4 cases 25 Why I make API clients Public

    APIs Private APIs Client developer == API developer 1 2 Client developer != API developer 3 4
  26. • Pros: Convenient for API client users ◦ It’s the

    Official client ▪ Users don't have to worry which client to use ▪ Often kept secure. (since it's maintained officially) ▪ e.g. octokit(GitHub), newrelic-rpm(NewRelic), raven(Sentry.io) 26 1. Public APIs made by ourselves
  27. • Cons: Tough for API client developers ◦ Multi-language support

    is a hard work 27 1. Public APIs made by ourselves
  28. • Context ◦ Internal API ◦ All API users are

    internal (e.g. same company) 28 2. Private APIs made by ourselves
  29. • If API creators make own API client gems, makes

    it easier to introduce services • API creator understands the specification most • Dog fooding 29 2. Private APIs made by ourselves
  30. • There are mostly existing API clients gems for major

    services ◦ Search in https://rubygems.org/ • Recommended to use the official API client 30 3. Public APIs made by others
  31. • Motivation ◦ Reinvention of wheel ◦ New services ◦

    Minor services 31 3. Public APIs made by others
  32. • Context ◦ API for BtoB (Business to Business) service

    ◦ There aren't API clients in OSS, so we have to make it ourselves • BTW. In Japan, the most commonly used API documentation tool is Excel (xls, xlsx) ◦ Naturally there is no swagger 32 4. Private APIs made by others
  33. • Case 1: in app repo ◦ e.g. app/, lib/

    ◦ I have actually seen this many times • Case 2: external gem ◦ e.g. in-house gem ◦ Even if for only 1 app, we should gemify 33 Where to place API client in our app?
  34. • Tight coupling ◦ Implementation ◦ Release cycle 34 Cons

    of Case 1 (in app repo)
  35. • Hard to CI ◦ Need to run all test

    suites ▪ Unit tests other than API client ◦ App testing tends to be time consuming ▪ 1 hour + (include E2E test) 35 Cons of Case 1 (in app repo)
  36. • Loosely coupled with the app • CI finishes fast

    • Easy to maintain 36 Pros of Case 2 (external gem)
  37. • Why do I make API clients? • -> Responsibilities

    of API clients • My API clients and Tips • 7 Good Patterns of API clients 37 Agenda
  38. • tl;dr; ◦ Single Responsibility Principle (SRP) • API client

    responsibilities should be focused on making the target API (e.g Twitter, Facebook etc.) easy to use in the target language (e.g Ruby) • If extra functions are included, it is difficult to follow API specification changes 38 Responsibilities of API clients
  39. • What API client should do • What API client

    should not do 39 Responsibilities of API clients
  40. 1. Convert params from Ruby to HTTP 2. Wrap error

    code with Ruby class 3. Auto update access token 40 What API client should do
  41. • In the Ruby world, we want to handle date

    with `Date`, and want to handle timestamp with `Time` • However, in the HTTP world it is necessary to convert it to `String` ◦ Format depends on API ◦ e.g. YYYY-MM-DD, YYYY/MM/DD • snake_case -> camelCase 41 1. Convert param from Ruby to HTTP
  42. • Context ◦ Some APIs return error code in response

    body • Makes it easier for app to catch errors with rescue 42 2. Wrap error code with Ruby class
  43. def create_message (message) with_error_handling do client.post( "/api/message" , message: message)

    end end def with_error_handling yield rescue Faraday::ClientError => error error_code = error.response.body[ "error_code"] error_class = ERROR_CODES[error_code] || BaseError raise error_class end 43 In API client gem code
  44. begin api_client.create_message( "foo") rescue MyApi::InternalError # do something rescue MyApi::OtherError

    # do something rescue MyApi::BaseError # do something end 44 In app code
  45. • c.f. Refactoring: Ruby Edition ◦ https://www.amazon.com/dp/0321984137 45 Extract Surrounding

    Method
  46. • Hide the following in API client ◦ Access token

    expired error ◦ Update access token using refresh token ◦ Retry original API calling 46 3. Auto update OAuth2 access token
  47. 47 3. Auto update access token class UserToken < AR::Base

    belongs_to :user def create_message with_retryable do api_client.create_message end end def with_retryable yield rescue AcessTokenExpiredError => error new_access_token, new_refresh_token = refresh_access_token user.update!( access_token: new_access_token, refresh_token: new_refresh_token, ) retry end end
  48. 1. API Response caching 2. Request params validation 48 What

    API client should not do
  49. • e.g. want to save API response in memcached for

    a fixed time • Requires dependency and config for memcached ◦ If we support cache backend other than memcached, it requires provider-style config • Requires complicated implementation for generalization ◦ Maintenance becomes difficult • This violates the SRP 49 1. API Response caching
  50. • Only API know if request params is valid or

    not • Acceptable only when validating ◦ required params (use required keyword args) ◦ non-blank params ◦ using official schema 50 2. Request params validation
  51. • Why do I make API clients? • Responsibilities of

    API clients • -> My API clients and Tips • 7 Good Patterns of API clients 51 Agenda
  52. • OSS ◦ pixela ◦ chatwork-ruby • other in-house gems

    52 My API clients and Tips
  53. 53 pixela

  54. • https://github.com/sue445/pixela • pixela is client gem of Pixela 54

    pixela
  55. • 55 Pixela (https://pixe.la/)

  56. https://pixe.la/v1/users/sue445/graphs/tweets.html 56 e.g. My daily tweets

  57. https://github.com/a-know/Pixela/releases/tag/v1.0.0 57 Pixela was launched at 2018/10/14

  58. https://twitter.com/rubygems/status/1051823343517732865 58 client gem was released at 2018/10/15

  59. 59 Tips. Use keyword args (even 1 or 2 args)

  60. https://github.com/a-know/Pixela/releases/tag/v1.3.0 60 Tips. Use keyword args (even 1 or 2

    args)
  61. 61 If I was NOT using keyword args... def graph_url(graph_id,

    date = nil) end def graph_url(graph_id, date = nil, mode = nil) end
  62. https://github.com/sue445/pixela/pull/25/files 62 Since I was using keyword args... def graph_url(graph_id:,

    date: nil) end def graph_url(graph_id:, date: nil, mode: nil) end
  63. 63 chatwork-ruby

  64. • https://github.com/asonas/chatwork-ruby • API client for ChatWork (https://go.chatwork.com/) • I

    became a maintainer when I sent many PRs 64 chatwork-ruby
  65. 65 Tips. Use official schema in test code

  66. • Official schema (RAML) is provided at https://github.com/chatwork/api • Add

    RAML repo to gem repo as submodule 66 Tips. Use official schema in test code
  67. https://github.com/asonas/chatwork-ruby/blob/v0.11.0/spec/lib/chatwork/message_spec.rb#L24-L36 67 Tips. Use official schema in test code describe

    ".create", type: :api do subject { ChatWork::Message.create(room_id: room_id, body: body, self_unread: self_unread, &block) } let(:room_id) { 123 } let(:body) { "Hello Chatwork!" } let(:self_unread ) { false } before do stub_chatwork_request( :post, "/rooms/#{room_id}/messages", "/rooms/{room_id}/messages" ) end it_behaves_like :a_chatwork_api , :post, "/rooms/{room_id}/messages" end
  68. • API clients for internal service • API clients for

    external services used in business • About 10 gems 68 Other in-house gems
  69. • Why do I make API clients? • Responsibilities of

    API clients • My API clients and Tips • -> 7 Good Patterns of API clients 69 Agenda
  70. • What I think of when creating API clients 70

    7 Good Patterns of API clients
  71. • Save labor of gem update • Use open-uri when

    only simple GET method 71 1. Keep dependency as minimal as possible require "open-uri" json = open("http://path/to/api" ).read res = JSON.parse(json)
  72. • However, code could become redundant with net/http • Faraday

    can do anything ◦ https://github.com/lostisland/faraday • Suitable when supporting all CRUD requests • Faraday's request and response is pluggable: easy to handle special cases 72 2. Faraday is a silver bullet
  73. 73 e.g. API that returns “true” and “false”

  74. 74 e.g. API that returns “true” and “false” $ curl

    http://example.com/api/something true $ curl http://example.com/api/something false
  75. 75 e.g. API that returns “true” and “false” connection =

    Faraday.new(url: "http://example.com" ) do |conn| conn.adapter Faraday.default_adapter end res = connection.get( "/api/something" ) res.body #=> "true" res.body.class #=> String
  76. 76 I created https://github.com/sue445/faraday_boolean connection = Faraday.new(url: "http://example.com" ) do

    |conn| conn.response :boolean conn.adapter Faraday.default_adapter end res = connection.get( "/api/something" ) res.body #=> true res.body.class #=> TrueClass
  77. 77 3. Make the subject stand out

  78. 78 3. Make the subject stand out def create_message (consumer_key:,

    consumer_secret: , access_token:, access_token_secret: , message:) # Call API end
  79. • Common variables used by all endpoints ◦ e.g. Auth

    • Variables used in specific endpoint ◦ Subject in endpoint 79 3. Make the subject stand out
  80. 80 3. Make the subject stand out class Client def

    initialize(consumer_key:, consumer_secret:, access_token:, access_token_secret:) @consumer_key = consumer_key, @consumer_secret = consumer_secret @access_token = access_token @access_token_secret = access_token_secret end def create_message (message) # Call API end end
  81. 81 4. Prefer keyword args to hash args

  82. 82 4. Prefer keyword args to hash args # Ruby

    1.9 syntax def create_graph(args = {}) end # Ruby 2.1+ syntax def create_graph( graph_id:, name:, unit:, type:, color:, timezone: nil, self_sufficient: nil) end VS
  83. • Required args and optional args are shown in code

    83 4. Prefer keyword args to hash args # Ruby 2.1+ syntax def create_graph( graph_id:, name:, unit:, type:, color:, timezone: nil, self_sufficient: nil) end
  84. 84 5. Introduce parameter object

  85. • It is difficult to use with 10+ args even

    if you use keyword args ◦ an Array of Hash is difficult to express in comment 85 5. Introduce parameter object # @param request [Array<Hash>] an Array of Hash # (key is name, message, content_type, ....) def create_content (request) # Send a request to API with XML end
  86. 86 5. Introduce parameter object class CreateContentRequest # @!attribute name

    # @return [String] the name of the content attr_accessor :name # and many more args def initialize(name:, message: content_type: "article", ...) end def to_param { name: name, message: message, contentType: content_type, ... } end end
  87. 87 5. Introduce parameter object # @param request [Array<CreateContentRequest>] def

    create_content (request) xml = request.map(&to_param).to_xml # send a request to API with XML end
  88. • Pros ◦ Parameter object itself can have conversion method

    ◦ Easy to write and read comment • If it gets easier to write YARD comment, consider adopting parameter object ◦ https://github.com/lsegal/yard • c.f. Refactoring: Ruby Edition ◦ https://www.amazon.com/dp/0321984137 88 5. Introduce parameter object
  89. 89 6. Method accessible response

  90. • The right is Rubyish 90 6. Method accessible response

    contents = api_client.get_contents first_content_name = contents[0][“name”] names = contents.map do |content| content["name"] end contents = api_client.get_contents first_content_name = contents[0].name names = contents.map( &:name) VS
  91. • Recommend to use mashify ◦ https://github.com/lostisland/faraday_middleware 91 6. Method

    accessible response connection = Faraday.new('http://example.com/api' ) do |conn| conn.response :mashify conn.adapter Faraday.default_adapter end
  92. 92 7. curl is universal language

  93. • When I create API client, I often ask questions

    about unclear things or bugs • Writing sample code in ruby is not appropriate ◦ Reason 1. API providers don't necessarily know Ruby ◦ Reason 2. This may be bugs or specification of Ruby. (instead of API) 93 7. curl is universal language
  94. • curl isn’t dependent on a specific language • curl

    can express all contexts with oneliner 94 Why should we use curl ? $ curl -X GET “https://example.com/api/user” -H “X-ACCESS-TOKEN: $ACCESS_TOKEN” -H “X-ACCESS-SECRET: $ACCESS_SECRET”
  95. • `curl 〜 | jq .` is good for reporting

    ◦ I always use this 95 Why should we use curl ? $ curl -X GET “https://example.com/api/user” -H “X-ACCESS-TOKEN: $ACCESS_TOKEN” -H “X-ACCESS-SECRET: $ACCESS_SECRET” | jq . { “name”: “sue445” }
  96. • `curl 〜 | jq . | pbcopy` ◦ Copy

    response to clickboard (Mac only) 96 Why should we use curl ? $ curl -X GET “https://example.com/api/user” -H “X-ACCESS-TOKEN: $ACCESS_TOKEN” -H “X-ACCESS-SECRET: $ACCESS_SECRET” | jq . | pbcopy
  97. • https://github.com/mauricio/faraday_curl ◦ This middleware logs your HTTP requests as

    CURL compatible commands so you can share the calls you're making with someone else or keep them for debugging purposes. 97 Auto debug logging in curl format
  98. https://github.com/sue445/pixela/blob/v1.1.0/lib/pixela/client.rb#L55-L69 98 Example def connection(request_headers = user_token_headers) Faraday.new(url: API_ENDPOINT, headers:

    request_headers) do |conn| conn.request :json conn.response :mashify, mash_class: Pixela::Response conn.response :json conn.response :raise_error if Pixela.config.debug_logger conn.request :curl, Pixela.config.debug_logger, :debug conn.response :logger, Pixela.config.debug_logger end conn.adapter Faraday.default_adapter end end
  99. 99 Example [1] pry(main)> @client.get_pixel(graph_id: "tweets", date: Date.new(2018, 9, 15))

    D, [2019-04-03T01:36:05.905606 #2167] DEBUG -- : curl -v -X GET -H 'X-USER-TOKEN: XXXXXXXXXXXX -H 'User-Agent: Pixela v1.1.0 (https://github.com/sue445/pixela)' -H 'Content-Type: application/json' "https://pixe.la/v1/users/sue445/graphs/tweets/20180915"
  100. • There are many reasons to create an API client

    • Responsibilities of API clients are SRP • Tips in my API clients • API Client 7 Good Patterns • Read “Refactoring: Ruby Edition” • Happy coding 100 Conclusion