Slide 1
Slide 1 text
Best practices in web API
client development
RubyKaigi 2019 / #rubykaigi #rubykaigiD
pixiv Inc.
sue445
2019.4.20
Slide 2
Slide 2 text
Hello!
2
Slide 3
Slide 3 text
My name is Go
3
Slide 4
Slide 4 text
4
About sue445
● Name: Go Sueyoshi
○ Go = 剛 = strong
● a.k.a. sue445
○ https://twitter.com/sue445
○ https://github.com/sue445
sue445
Slide 5
Slide 5 text
5
About sue445
● Birthday: 1982/04/07
○ Last week was my birthday
○ age 36 -> 37
sue445
Slide 6
Slide 6 text
6
Experience
● Ruby: 7 years
● Golang: 4 years
● Go: 37 years (age)
sue445
Slide 7
Slide 7 text
7
About sue445
● Birthplace: Fukuoka
○ School days
○ 〜 age 26
● Currently at: Tokyo
○ pixiv Inc.
sue445
Slide 8
Slide 8 text
8
About sue445
● Community
○ Shibuya.rb
○ Omotesando.rb
○ Shinjuku.rb
○ Meguro.rb
○ etc
sue445
Slide 9
Slide 9 text
9
About sue445
● RubyKaja 2014
○ http://kaja.rubyist.net/2014/kaja
○ https://www.slideshare.net/tyabe/ruby-kaja2014
sue445
Slide 10
Slide 10 text
● https://rubygems.org/profiles/sue445
○ e.g. rubicure (Ruby implementation of Pretty Cure)
■ https://github.com/sue445/rubicure
○ Maintainer of Itamae and chatwork-ruby
10
Gems I developed
Slide 11
Slide 11 text
11
About pixiv Inc.
Slide 12
Slide 12 text
MAKE CREATIVITIES
MORE ENJOYABLE
12
About pixiv Inc.
Slide 13
Slide 13 text
● https://www.pixiv.co.jp/
● Place: Tokyo and Fukuoka
● Silver Sponsor
13
About pixiv Inc.
Slide 14
Slide 14 text
14
FAQ. Is pixiv Inc.’s main lang PHP?
Slide 15
Slide 15 text
● Ruby is widely used at pixiv Inc, too
15
FAQ. Is pixiv Inc.’s main lang PHP?
Slide 16
Slide 16 text
16
What I've been doing at pixiv Inc.
Slide 17
Slide 17 text
● SUE (sue445) sounds like SRE (Site Reliability Engineer)
○ My job is almost like that of an SRE
■ e.g. web application, library, infra, CI etc ...
○ Ansible >>>> Terraform >>> Ruby
■ BTW. I’m an Itamae maintainer
17
What I've been doing at pixiv Inc.
Slide 18
Slide 18 text
● https://rubykaigi.org/2019/presentations/ota42y.html#apr18
● @ota42y was my colleague
● I was his mentor when he was an intern (2012/08)
18
Me and @ota42y
Slide 19
Slide 19 text
● OpenAPI3 is Bright side of API (IMO :-P)
○ There is a shining future
● My talk is about Dark side of API
○ Fight against reality
19
Bright side vs Dark side of API
Slide 20
Slide 20 text
● How to not fail API client development
○ Good architecture and best practice
● Architecture and best practices in regular gem development
○ Architecture techniques in API clients also help the other gem development
20
The Goal/Objective of this talk
Slide 21
Slide 21 text
● Why do I make API clients?
● Responsibilities of API clients
● My API clients and Tips
● 7 Good Patterns of API clients
21
Agenda
Slide 22
Slide 22 text
22
Topics covered today
App
Web APIs
API client
Slide 23
Slide 23 text
● New technologies
○ e.g. OpenAPI, microservices
● API servers
● non-text-based APIs
○ APIs of which both requests and responses aren't in text format
○ e.g. Protocol Buffers
● non web APIs
23
Topics NOT covered today
Slide 24
Slide 24 text
● -> Why do I make API clients?
● Responsibilities of API clients
● My API clients and Tips
● 7 Good Patterns of API clients
24
Agenda
Slide 25
Slide 25 text
● 4 cases
25
Why I make API clients
Public APIs Private APIs
Client developer == API developer 1 2
Client developer != API developer 3 4
Slide 26
Slide 26 text
● Pros: Convenient for API client users
○ It’s the Official client
■ Users don't have to worry which client to use
■ Often kept secure. (since it's maintained officially)
■ e.g. octokit(GitHub), newrelic-rpm(NewRelic), raven(Sentry.io)
26
1. Public APIs made by ourselves
Slide 27
Slide 27 text
● Cons: Tough for API client developers
○ Multi-language support is a hard work
27
1. Public APIs made by ourselves
Slide 28
Slide 28 text
● Context
○ Internal API
○ All API users are internal (e.g. same company)
28
2. Private APIs made by ourselves
Slide 29
Slide 29 text
● If API creators make own API client gems, makes it easier to introduce services
● API creator understands the specification most
● Dog fooding
29
2. Private APIs made by ourselves
Slide 30
Slide 30 text
● There are mostly existing API clients gems for major services
○ Search in https://rubygems.org/
● Recommended to use the official API client
30
3. Public APIs made by others
Slide 31
Slide 31 text
● Motivation
○ Reinvention of wheel
○ New services
○ Minor services
31
3. Public APIs made by others
Slide 32
Slide 32 text
● Context
○ API for BtoB (Business to Business) service
○ There aren't API clients in OSS, so we have to make it ourselves
● BTW. In Japan, the most commonly used API documentation tool is Excel (xls,
xlsx)
○ Naturally there is no swagger
32
4. Private APIs made by others
Slide 33
Slide 33 text
● Case 1: in app repo
○ e.g. app/, lib/
○ I have actually seen this many times
● Case 2: external gem
○ e.g. in-house gem
○ Even if for only 1 app, we should gemify
33
Where to place API client in our app?
Slide 34
Slide 34 text
● Tight coupling
○ Implementation
○ Release cycle
34
Cons of Case 1 (in app repo)
Slide 35
Slide 35 text
● Hard to CI
○ Need to run all test suites
■ Unit tests other than API client
○ App testing tends to be time consuming
■ 1 hour + (include E2E test)
35
Cons of Case 1 (in app repo)
Slide 36
Slide 36 text
● Loosely coupled with the app
● CI finishes fast
● Easy to maintain
36
Pros of Case 2 (external gem)
Slide 37
Slide 37 text
● Why do I make API clients?
● -> Responsibilities of API clients
● My API clients and Tips
● 7 Good Patterns of API clients
37
Agenda
Slide 38
Slide 38 text
● tl;dr;
○ Single Responsibility Principle (SRP)
● API client responsibilities should be focused on making the target API (e.g Twitter,
Facebook etc.) easy to use in the target language (e.g Ruby)
● If extra functions are included, it is difficult to follow API specification changes
38
Responsibilities of API clients
Slide 39
Slide 39 text
● What API client should do
● What API client should not do
39
Responsibilities of API clients
Slide 40
Slide 40 text
1. Convert params from Ruby to HTTP
2. Wrap error code with Ruby class
3. Auto update access token
40
What API client should do
Slide 41
Slide 41 text
● In the Ruby world, we want to handle date with `Date`, and want to handle
timestamp with `Time`
● However, in the HTTP world it is necessary to convert it to `String`
○ Format depends on API
○ e.g. YYYY-MM-DD, YYYY/MM/DD
● snake_case -> camelCase
41
1. Convert param from Ruby to HTTP
Slide 42
Slide 42 text
● Context
○ Some APIs return error code in response body
● Makes it easier for app to catch errors with rescue
42
2. Wrap error code with Ruby class
Slide 43
Slide 43 text
def create_message (message)
with_error_handling do
client.post( "/api/message" , message: message)
end
end
def with_error_handling
yield
rescue Faraday::ClientError => error
error_code = error.response.body[ "error_code"]
error_class = ERROR_CODES[error_code] || BaseError
raise error_class
end
43
In API client gem code
Slide 44
Slide 44 text
begin
api_client.create_message( "foo")
rescue MyApi::InternalError
# do something
rescue MyApi::OtherError
# do something
rescue MyApi::BaseError
# do something
end
44
In app code
Slide 45
Slide 45 text
● c.f. Refactoring: Ruby Edition
○ https://www.amazon.com/dp/0321984137
45
Extract Surrounding Method
Slide 46
Slide 46 text
● Hide the following in API client
○ Access token expired error
○ Update access token using refresh token
○ Retry original API calling
46
3. Auto update OAuth2 access token
Slide 47
Slide 47 text
47
3. Auto update access token
class UserToken < AR::Base
belongs_to :user
def create_message
with_retryable do
api_client.create_message
end
end
def with_retryable
yield
rescue AcessTokenExpiredError => error
new_access_token, new_refresh_token =
refresh_access_token
user.update!(
access_token: new_access_token,
refresh_token: new_refresh_token,
)
retry
end
end
Slide 48
Slide 48 text
1. API Response caching
2. Request params validation
48
What API client should not do
Slide 49
Slide 49 text
● e.g. want to save API response in memcached for a fixed time
● Requires dependency and config for memcached
○ If we support cache backend other than memcached, it requires provider-style
config
● Requires complicated implementation for generalization
○ Maintenance becomes difficult
● This violates the SRP
49
1. API Response caching
Slide 50
Slide 50 text
● Only API know if request params is valid or not
● Acceptable only when validating
○ required params (use required keyword args)
○ non-blank params
○ using official schema
50
2. Request params validation
Slide 51
Slide 51 text
● Why do I make API clients?
● Responsibilities of API clients
● -> My API clients and Tips
● 7 Good Patterns of API clients
51
Agenda
Slide 52
Slide 52 text
● OSS
○ pixela
○ chatwork-ruby
● other in-house gems
52
My API clients and Tips
Slide 53
Slide 53 text
53
pixela
Slide 54
Slide 54 text
● https://github.com/sue445/pixela
● pixela is client gem of Pixela
54
pixela
Slide 55
Slide 55 text
●
55
Pixela (https://pixe.la/)
Slide 56
Slide 56 text
https://pixe.la/v1/users/sue445/graphs/tweets.html
56
e.g. My daily tweets
Slide 57
Slide 57 text
https://github.com/a-know/Pixela/releases/tag/v1.0.0
57
Pixela was launched at 2018/10/14
Slide 58
Slide 58 text
https://twitter.com/rubygems/status/1051823343517732865
58
client gem was released at 2018/10/15
Slide 59
Slide 59 text
59
Tips. Use keyword args (even 1 or 2 args)
Slide 60
Slide 60 text
https://github.com/a-know/Pixela/releases/tag/v1.3.0
60
Tips. Use keyword args (even 1 or 2 args)
Slide 61
Slide 61 text
61
If I was NOT using keyword args...
def graph_url(graph_id, date = nil)
end
def graph_url(graph_id, date = nil, mode = nil)
end
Slide 62
Slide 62 text
https://github.com/sue445/pixela/pull/25/files
62
Since I was using keyword args...
def graph_url(graph_id:, date: nil)
end
def graph_url(graph_id:, date: nil, mode: nil)
end
Slide 63
Slide 63 text
63
chatwork-ruby
Slide 64
Slide 64 text
● https://github.com/asonas/chatwork-ruby
● API client for ChatWork (https://go.chatwork.com/)
● I became a maintainer when I sent many PRs
64
chatwork-ruby
Slide 65
Slide 65 text
65
Tips. Use official schema in test code
Slide 66
Slide 66 text
● Official schema (RAML) is provided at https://github.com/chatwork/api
● Add RAML repo to gem repo as submodule
66
Tips. Use official schema in test code
Slide 67
Slide 67 text
https://github.com/asonas/chatwork-ruby/blob/v0.11.0/spec/lib/chatwork/message_spec.rb#L24-L36
67
Tips. Use official schema in test code
describe ".create", type: :api do
subject { ChatWork::Message.create(room_id: room_id, body: body, self_unread: self_unread,
&block) }
let(:room_id) { 123 }
let(:body) { "Hello Chatwork!" }
let(:self_unread ) { false }
before do
stub_chatwork_request( :post, "/rooms/#{room_id}/messages", "/rooms/{room_id}/messages" )
end
it_behaves_like :a_chatwork_api , :post, "/rooms/{room_id}/messages"
end
Slide 68
Slide 68 text
● API clients for internal service
● API clients for external services used in business
● About 10 gems
68
Other in-house gems
Slide 69
Slide 69 text
● Why do I make API clients?
● Responsibilities of API clients
● My API clients and Tips
● -> 7 Good Patterns of API clients
69
Agenda
Slide 70
Slide 70 text
● What I think of when creating API clients
70
7 Good Patterns of API clients
Slide 71
Slide 71 text
● Save labor of gem update
● Use open-uri when only simple GET method
71
1. Keep dependency as minimal as possible
require "open-uri"
json = open("http://path/to/api" ).read
res = JSON.parse(json)
Slide 72
Slide 72 text
● However, code could become redundant with net/http
● Faraday can do anything
○ https://github.com/lostisland/faraday
● Suitable when supporting all CRUD requests
● Faraday's request and response is pluggable: easy to handle special cases
72
2. Faraday is a silver bullet
Slide 73
Slide 73 text
73
e.g. API that returns “true” and “false”
Slide 74
Slide 74 text
74
e.g. API that returns “true” and “false”
$ curl http://example.com/api/something
true
$ curl http://example.com/api/something
false
Slide 75
Slide 75 text
75
e.g. API that returns “true” and “false”
connection =
Faraday.new(url: "http://example.com" ) do |conn|
conn.adapter Faraday.default_adapter
end
res = connection.get( "/api/something" )
res.body
#=> "true"
res.body.class
#=> String
Slide 76
Slide 76 text
76
I created https://github.com/sue445/faraday_boolean
connection =
Faraday.new(url: "http://example.com" ) do |conn|
conn.response :boolean
conn.adapter Faraday.default_adapter
end
res = connection.get( "/api/something" )
res.body
#=> true
res.body.class
#=> TrueClass
Slide 77
Slide 77 text
77
3. Make the subject stand out
Slide 78
Slide 78 text
78
3. Make the subject stand out
def create_message (consumer_key:, consumer_secret: ,
access_token:, access_token_secret: ,
message:)
# Call API
end
Slide 79
Slide 79 text
● Common variables used by all endpoints
○ e.g. Auth
● Variables used in specific endpoint
○ Subject in endpoint
79
3. Make the subject stand out
Slide 80
Slide 80 text
80
3. Make the subject stand out
class Client
def initialize(consumer_key:, consumer_secret:,
access_token:, access_token_secret:)
@consumer_key = consumer_key,
@consumer_secret = consumer_secret
@access_token = access_token
@access_token_secret = access_token_secret
end
def
create_message (message)
# Call API
end
end
Slide 81
Slide 81 text
81
4. Prefer keyword args to hash args
Slide 82
Slide 82 text
82
4. Prefer keyword args to hash args
# Ruby 1.9 syntax
def create_graph(args = {})
end
# Ruby 2.1+ syntax
def create_graph(
graph_id:, name:, unit:,
type:, color:,
timezone: nil,
self_sufficient: nil)
end
VS
Slide 83
Slide 83 text
● Required args and optional args are shown in code
83
4. Prefer keyword args to hash args
# Ruby 2.1+ syntax
def create_graph(
graph_id:, name:, unit:, type:, color:,
timezone: nil, self_sufficient: nil)
end
Slide 84
Slide 84 text
84
5. Introduce parameter object
Slide 85
Slide 85 text
● It is difficult to use with 10+ args even if you use keyword args
○ an Array of Hash is difficult to express in comment
85
5. Introduce parameter object
# @param request [Array] an Array of Hash
# (key is name, message, content_type, ....)
def create_content (request)
# Send a request to API with XML
end
Slide 86
Slide 86 text
86
5. Introduce parameter object
class CreateContentRequest
# @!attribute name
# @return [String] the name of the content
attr_accessor :name
# and many more args
def initialize(name:, message: content_type: "article", ...)
end
def to_param
{
name: name, message: message, contentType: content_type, ...
}
end
end
Slide 87
Slide 87 text
87
5. Introduce parameter object
# @param request [Array]
def create_content (request)
xml = request.map(&to_param).to_xml
# send a request to API with XML
end
Slide 88
Slide 88 text
● Pros
○ Parameter object itself can have conversion method
○ Easy to write and read comment
● If it gets easier to write YARD comment, consider adopting parameter object
○ https://github.com/lsegal/yard
● c.f. Refactoring: Ruby Edition
○ https://www.amazon.com/dp/0321984137
88
5. Introduce parameter object
Slide 89
Slide 89 text
89
6. Method accessible response
Slide 90
Slide 90 text
● The right is Rubyish
90
6. Method accessible response
contents = api_client.get_contents
first_content_name =
contents[0][“name”]
names = contents.map do |content|
content["name"]
end
contents = api_client.get_contents
first_content_name =
contents[0].name
names = contents.map( &:name)
VS
Slide 91
Slide 91 text
● Recommend to use mashify
○ https://github.com/lostisland/faraday_middleware
91
6. Method accessible response
connection = Faraday.new('http://example.com/api' ) do |conn|
conn.response :mashify
conn.adapter Faraday.default_adapter
end
Slide 92
Slide 92 text
92
7. curl is universal language
Slide 93
Slide 93 text
● When I create API client, I often ask questions about unclear things or bugs
● Writing sample code in ruby is not appropriate
○ Reason 1. API providers don't necessarily know Ruby
○ Reason 2. This may be bugs or specification of Ruby. (instead of API)
93
7. curl is universal language
Slide 94
Slide 94 text
● curl isn’t dependent on a specific language
● curl can express all contexts with oneliner
94
Why should we use curl ?
$ curl -X GET “https://example.com/api/user” -H
“X-ACCESS-TOKEN: $ACCESS_TOKEN” -H “X-ACCESS-SECRET:
$ACCESS_SECRET”
Slide 95
Slide 95 text
● `curl 〜 | jq .` is good for reporting
○ I always use this
95
Why should we use curl ?
$ curl -X GET “https://example.com/api/user” -H
“X-ACCESS-TOKEN: $ACCESS_TOKEN” -H “X-ACCESS-SECRET:
$ACCESS_SECRET” | jq .
{
“name”: “sue445”
}
Slide 96
Slide 96 text
● `curl 〜 | jq . | pbcopy`
○ Copy response to clickboard (Mac only)
96
Why should we use curl ?
$ curl -X GET “https://example.com/api/user” -H
“X-ACCESS-TOKEN: $ACCESS_TOKEN” -H “X-ACCESS-SECRET:
$ACCESS_SECRET” | jq . | pbcopy
Slide 97
Slide 97 text
● https://github.com/mauricio/faraday_curl
○ This middleware logs your HTTP requests as CURL compatible commands so
you can share the calls you're making with someone else or keep them for
debugging purposes.
97
Auto debug logging in curl format
Slide 98
Slide 98 text
https://github.com/sue445/pixela/blob/v1.1.0/lib/pixela/client.rb#L55-L69
98
Example
def connection(request_headers = user_token_headers)
Faraday.new(url: API_ENDPOINT, headers: request_headers) do |conn|
conn.request :json
conn.response :mashify, mash_class: Pixela::Response
conn.response :json
conn.response :raise_error
if Pixela.config.debug_logger
conn.request :curl, Pixela.config.debug_logger, :debug
conn.response :logger, Pixela.config.debug_logger
end
conn.adapter Faraday.default_adapter
end
end
Slide 99
Slide 99 text
99
Example
[1] pry(main)> @client.get_pixel(graph_id: "tweets", date: Date.new(2018, 9, 15))
D, [2019-04-03T01:36:05.905606 #2167] DEBUG -- : curl -v -X GET -H 'X-USER-TOKEN:
XXXXXXXXXXXX -H 'User-Agent: Pixela v1.1.0 (https://github.com/sue445/pixela)' -H
'Content-Type: application/json'
"https://pixe.la/v1/users/sue445/graphs/tweets/20180915"
Slide 100
Slide 100 text
● There are many reasons to create an API client
● Responsibilities of API clients are SRP
● Tips in my API clients
● API Client 7 Good Patterns
● Read “Refactoring: Ruby Edition”
● Happy coding
100
Conclusion