Best practices in web API client development #RubyKaigi

https://rubykaigi.org/2019/presentations/sue445.html#apr20

Transcript

1. Best practices in web API
   client development
   RubyKaigi 2019 / #rubykaigi #rubykaigiD
   pixiv Inc.
   sue445
   2019.4.20
   

   View Slide

2. Hello!
   2
   

   View Slide

3. My name is Go
   3
   

   View Slide

4. 4
   About sue445
   ● Name: Go Sueyoshi
   ○ Go = 剛 = strong
   ● a.k.a. sue445
   ○ https://twitter.com/sue445
   ○ https://github.com/sue445
   sue445
   

   View Slide

5. 5
   About sue445
   ● Birthday: 1982/04/07
   ○ Last week was my birthday
   ○ age 36 -> 37
   sue445
   

   View Slide

6. 6
   Experience
   ● Ruby: 7 years
   ● Golang: 4 years
   ● Go: 37 years (age)
   sue445
   

   View Slide

7. 7
   About sue445
   ● Birthplace: Fukuoka
   ○ School days
   ○ 〜 age 26
   ● Currently at: Tokyo
   ○ pixiv Inc.
   sue445
   

   View Slide

8. 8
   About sue445
   ● Community
   ○ Shibuya.rb
   ○ Omotesando.rb
   ○ Shinjuku.rb
   ○ Meguro.rb
   ○ etc
   sue445
   

   View Slide

9. 9
   About sue445
   ● RubyKaja 2014
   ○ http://kaja.rubyist.net/2014/kaja
   ○ https://www.slideshare.net/tyabe/ruby-kaja2014
   sue445
   

   View Slide

10. ● https://rubygems.org/profiles/sue445
    ○ e.g. rubicure (Ruby implementation of Pretty Cure)
    ■ https://github.com/sue445/rubicure
    ○ Maintainer of Itamae and chatwork-ruby
    10
    Gems I developed
    

    View Slide

11. 11
    About pixiv Inc.
    

    View Slide

12. MAKE CREATIVITIES
    MORE ENJOYABLE
    12
    About pixiv Inc.
    

    View Slide

13. ● https://www.pixiv.co.jp/
    ● Place: Tokyo and Fukuoka
    ● Silver Sponsor
    13
    About pixiv Inc.
    

    View Slide

14. 14
    FAQ. Is pixiv Inc.’s main lang PHP?
    

    View Slide

15. ● Ruby is widely used at pixiv Inc, too
    15
    FAQ. Is pixiv Inc.’s main lang PHP?
    

    View Slide

16. 16
    What I've been doing at pixiv Inc.
    

    View Slide

17. ● SUE (sue445) sounds like SRE (Site Reliability Engineer)
    ○ My job is almost like that of an SRE
    ■ e.g. web application, library, infra, CI etc ...
    ○ Ansible >>>> Terraform >>> Ruby
    ■ BTW. I’m an Itamae maintainer
    17
    What I've been doing at pixiv Inc.
    

    View Slide

18. ● https://rubykaigi.org/2019/presentations/ota42y.html#apr18
    ● @ota42y was my colleague
    ● I was his mentor when he was an intern (2012/08)
    18
    Me and @ota42y
    

    View Slide

19. ● OpenAPI3 is Bright side of API 　　 (IMO :-P)
    ○ There is a shining future
    ● My talk is about Dark side of API
    ○ Fight against reality
    19
    　 Bright side vs Dark side of API
    

    View Slide

20. ● How to not fail API client development
    ○ Good architecture and best practice
    ● Architecture and best practices in regular gem development
    ○ Architecture techniques in API clients also help the other gem development
    20
    The Goal/Objective of this talk
    

    View Slide

21. ● Why do I make API clients?
    ● Responsibilities of API clients
    ● My API clients and Tips
    ● 7 Good Patterns of API clients
    21
    Agenda
    

    View Slide

22. 22
    Topics covered today
    App
    Web APIs
    API client
    

    View Slide

23. ● New technologies
    ○ e.g. OpenAPI, microservices
    ● API servers
    ● non-text-based APIs
    ○ APIs of which both requests and responses aren't in text format
    ○ e.g. Protocol Buffers
    ● non web APIs
    23
    Topics NOT covered today
    

    View Slide

24. ● -> Why do I make API clients?
    ● Responsibilities of API clients
    ● My API clients and Tips
    ● 7 Good Patterns of API clients
    24
    Agenda
    

    View Slide

25. ● 4 cases
    25
    Why I make API clients
    Public APIs Private APIs
    Client developer == API developer 1 2
    Client developer != API developer 3 4
    

    View Slide

26. ● Pros: Convenient for API client users
    ○ It’s the Official client
    ■ Users don't have to worry which client to use
    ■ Often kept secure. (since it's maintained officially)
    ■ e.g. octokit(GitHub), newrelic-rpm(NewRelic), raven(Sentry.io)
    26
    1. Public APIs made by ourselves
    

    View Slide

27. ● Cons: Tough for API client developers
    ○ Multi-language support is a hard work
    27
    1. Public APIs made by ourselves
    

    View Slide

28. ● Context
    ○ Internal API
    ○ All API users are internal (e.g. same company)
    28
    2. Private APIs made by ourselves
    

    View Slide

29. ● If API creators make own API client gems, makes it easier to introduce services
    ● API creator understands the specification most
    ● Dog fooding
    29
    2. Private APIs made by ourselves
    

    View Slide

30. ● There are mostly existing API clients gems for major services
    ○ Search in https://rubygems.org/
    ● Recommended to use the official API client
    30
    3. Public APIs made by others
    

    View Slide

31. ● Motivation
    ○ Reinvention of wheel
    ○ New services
    ○ Minor services
    31
    3. Public APIs made by others
    

    View Slide

32. ● Context
    ○ API for BtoB (Business to Business) service
    ○ There aren't API clients in OSS, so we have to make it ourselves
    ● BTW. In Japan, the most commonly used API documentation tool is Excel (xls,
    xlsx)
    ○ Naturally there is no swagger
    32
    4. Private APIs made by others
    

    View Slide

33. ● Case 1: in app repo
    ○ e.g. app/, lib/
    ○ I have actually seen this many times
    ● Case 2: external gem
    ○ e.g. in-house gem
    ○ Even if for only 1 app, we should gemify
    33
    Where to place API client in our app?
    

    View Slide

34. ● Tight coupling
    ○ Implementation
    ○ Release cycle
    34
    Cons of Case 1 (in app repo)
    

    View Slide

35. ● Hard to CI
    ○ Need to run all test suites
    ■ Unit tests other than API client
    ○ App testing tends to be time consuming
    ■ 1 hour + (include E2E test)
    35
    Cons of Case 1 (in app repo)
    

    View Slide

36. ● Loosely coupled with the app
    ● CI finishes fast
    ● Easy to maintain
    36
    Pros of Case 2 (external gem)
    

    View Slide

37. ● Why do I make API clients?
    ● -> Responsibilities of API clients
    ● My API clients and Tips
    ● 7 Good Patterns of API clients
    37
    Agenda
    

    View Slide

38. ● tl;dr;
    ○ Single Responsibility Principle (SRP)
    ● API client responsibilities should be focused on making the target API (e.g Twitter,
    Facebook etc.) easy to use in the target language (e.g Ruby)
    ● If extra functions are included, it is difficult to follow API specification changes
    38
    Responsibilities of API clients
    

    View Slide

39. ● What API client should do
    ● What API client should not do
    39
    Responsibilities of API clients
    

    View Slide

40. 1. Convert params from Ruby to HTTP
    2. Wrap error code with Ruby class
    3. Auto update access token
    40
    What API client should do
    

    View Slide

41. ● In the Ruby world, we want to handle date with `Date`, and want to handle
    timestamp with `Time`
    ● However, in the HTTP world it is necessary to convert it to `String`
    ○ Format depends on API
    ○ e.g. YYYY-MM-DD, YYYY/MM/DD
    ● snake_case -> camelCase
    41
    1. Convert param from Ruby to HTTP
    

    View Slide

42. ● Context
    ○ Some APIs return error code in response body
    ● Makes it easier for app to catch errors with rescue
    42
    2. Wrap error code with Ruby class
    

    View Slide

43. def create_message (message)
    with_error_handling do
    client.post( "/api/message" , message: message)
    end
    end
    def with_error_handling
    yield
    rescue Faraday::ClientError => error
    error_code = error.response.body[ "error_code"]
    error_class = ERROR_CODES[error_code] || BaseError
    raise error_class
    end
    43
    In API client gem code
    

    View Slide

44. begin
    api_client.create_message( "foo")
    rescue MyApi::InternalError
    # do something
    rescue MyApi::OtherError
    # do something
    rescue MyApi::BaseError
    # do something
    end
    44
    In app code
    

    View Slide

45. ● c.f. Refactoring: Ruby Edition
    ○ https://www.amazon.com/dp/0321984137
    45
    Extract Surrounding Method
    

    View Slide

46. ● Hide the following in API client
    ○ Access token expired error
    ○ Update access token using refresh token
    ○ Retry original API calling
    46
    3. Auto update OAuth2 access token
    

    View Slide

47. 47
    3. Auto update access token
    class UserToken < AR::Base
    belongs_to :user
    def create_message
    with_retryable do
    api_client.create_message
    end
    end
    def with_retryable
    yield
    rescue AcessTokenExpiredError => error
    new_access_token, new_refresh_token =
    refresh_access_token
    user.update!(
    access_token: new_access_token,
    refresh_token: new_refresh_token,
    )
    retry
    end
    end
    

    View Slide

48. 1. API Response caching
    2. Request params validation
    48
    What API client should not do
    

    View Slide

49. ● e.g. want to save API response in memcached for a fixed time
    ● Requires dependency and config for memcached
    ○ If we support cache backend other than memcached, it requires provider-style
    config
    ● Requires complicated implementation for generalization
    ○ Maintenance becomes difficult
    ● This violates the SRP
    49
    1. API Response caching
    

    View Slide

50. ● Only API know if request params is valid or not
    ● Acceptable only when validating
    ○ required params (use required keyword args)
    ○ non-blank params
    ○ using official schema
    50
    2. Request params validation
    

    View Slide

51. ● Why do I make API clients?
    ● Responsibilities of API clients
    ● -> My API clients and Tips
    ● 7 Good Patterns of API clients
    51
    Agenda
    

    View Slide

52. ● OSS
    ○ pixela
    ○ chatwork-ruby
    ● other in-house gems
    52
    My API clients and Tips
    

    View Slide

53. 53
    pixela
    

    View Slide

54. ● https://github.com/sue445/pixela
    ● pixela is client gem of Pixela
    54
    pixela
    

    View Slide

55. ●
    55
    Pixela (https://pixe.la/)
    

    View Slide

56. https://pixe.la/v1/users/sue445/graphs/tweets.html
    56
    e.g. My daily tweets
    

    View Slide

57. https://github.com/a-know/Pixela/releases/tag/v1.0.0
    57
    Pixela was launched at 2018/10/14
    

    View Slide

58. https://twitter.com/rubygems/status/1051823343517732865
    58
    client gem was released at 2018/10/15
    

    View Slide

59. 59
    Tips. Use keyword args (even 1 or 2 args)
    

    View Slide

60. https://github.com/a-know/Pixela/releases/tag/v1.3.0
    60
    Tips. Use keyword args (even 1 or 2 args)
    

    View Slide

61. 61
    If I was NOT using keyword args...
    def graph_url(graph_id, date = nil)
    end
    def graph_url(graph_id, date = nil, mode = nil)
    end
    

    View Slide

62. https://github.com/sue445/pixela/pull/25/files
    62
    Since I was using keyword args...
    def graph_url(graph_id:, date: nil)
    end
    def graph_url(graph_id:, date: nil, mode: nil)
    end
    

    View Slide

63. 63
    chatwork-ruby
    

    View Slide

64. ● https://github.com/asonas/chatwork-ruby
    ● API client for ChatWork (https://go.chatwork.com/)
    ● I became a maintainer when I sent many PRs
    64
    chatwork-ruby
    

    View Slide

65. 65
    Tips. Use official schema in test code
    

    View Slide

66. ● Official schema (RAML) is provided at https://github.com/chatwork/api
    ● Add RAML repo to gem repo as submodule
    66
    Tips. Use official schema in test code
    

    View Slide

67. https://github.com/asonas/chatwork-ruby/blob/v0.11.0/spec/lib/chatwork/message_spec.rb#L24-L36
    67
    Tips. Use official schema in test code
    describe ".create", type: :api do
    subject { ChatWork::Message.create(room_id: room_id, body: body, self_unread: self_unread,
    &block) }
    let(:room_id) { 123 }
    let(:body) { "Hello Chatwork!" }
    let(:self_unread ) { false }
    before do
    stub_chatwork_request( :post, "/rooms/#{room_id}/messages", "/rooms/{room_id}/messages" )
    end
    it_behaves_like :a_chatwork_api , :post, "/rooms/{room_id}/messages"
    end
    

    View Slide

68. ● API clients for internal service
    ● API clients for external services used in business
    ● About 10 gems
    68
    Other in-house gems
    

    View Slide

69. ● Why do I make API clients?
    ● Responsibilities of API clients
    ● My API clients and Tips
    ● -> 7 Good Patterns of API clients
    69
    Agenda
    

    View Slide

70. ● What I think of when creating API clients
    70
    7 Good Patterns of API clients
    

    View Slide

71. ● Save labor of gem update
    ● Use open-uri when only simple GET method
    71
    1. Keep dependency as minimal as possible
    require "open-uri"
    json = open("http://path/to/api" ).read
    res = JSON.parse(json)
    

    View Slide

72. ● However, code could become redundant with net/http
    ● Faraday can do anything
    ○ https://github.com/lostisland/faraday
    ● Suitable when supporting all CRUD requests
    ● Faraday's request and response is pluggable: easy to handle special cases
    72
    2. Faraday is a silver bullet
    

    View Slide

73. 73
    e.g. API that returns “true” and “false”
    

    View Slide

74. 74
    e.g. API that returns “true” and “false”
    $ curl http://example.com/api/something
    true
    $ curl http://example.com/api/something
    false
    

    View Slide

75. 75
    e.g. API that returns “true” and “false”
    connection =
    Faraday.new(url: "http://example.com" ) do |conn|
    conn.adapter Faraday.default_adapter
    end
    res = connection.get( "/api/something" )
    res.body
    #=> "true"
    res.body.class
    #=> String
    

    View Slide

76. 76
    I created https://github.com/sue445/faraday_boolean
    connection =
    Faraday.new(url: "http://example.com" ) do |conn|
    conn.response :boolean
    conn.adapter Faraday.default_adapter
    end
    res = connection.get( "/api/something" )
    res.body
    #=> true
    res.body.class
    #=> TrueClass
    

    View Slide

77. 77
    3. Make the subject stand out
    

    View Slide

78. 78
    3. Make the subject stand out
    def create_message (consumer_key:, consumer_secret: ,
    access_token:, access_token_secret: ,
    message:)
    # Call API
    end
    

    View Slide

79. ● Common variables used by all endpoints
    ○ e.g. Auth
    ● Variables used in specific endpoint
    ○ Subject in endpoint
    79
    3. Make the subject stand out
    

    View Slide

80. 80
    3. Make the subject stand out
    class Client
    def initialize(consumer_key:, consumer_secret:,
    access_token:, access_token_secret:)
    @consumer_key = consumer_key,
    @consumer_secret = consumer_secret
    @access_token = access_token
    @access_token_secret = access_token_secret
    end
    def
    create_message (message)
    # Call API
    end
    end
    

    View Slide

81. 81
    4. Prefer keyword args to hash args
    

    View Slide

82. 82
    4. Prefer keyword args to hash args
    # Ruby 1.9 syntax
    def create_graph(args = {})
    end
    # Ruby 2.1+ syntax
    def create_graph(
    graph_id:, name:, unit:,
    type:, color:,
    timezone: nil,
    self_sufficient: nil)
    end
    VS
    

    View Slide

83. ● Required args and optional args are shown in code
    83
    4. Prefer keyword args to hash args
    # Ruby 2.1+ syntax
    def create_graph(
    graph_id:, name:, unit:, type:, color:,
    timezone: nil, self_sufficient: nil)
    end
    

    View Slide

84. 84
    5. Introduce parameter object
    

    View Slide

85. ● It is difficult to use with 10+ args even if you use keyword args
    ○ an Array of Hash is difficult to express in comment
    85
    5. Introduce parameter object
    # @param request [Array] an Array of Hash
    # (key is name, message, content_type, ....)
    def create_content (request)
    # Send a request to API with XML
    end
    

    View Slide

86. 86
    5. Introduce parameter object
    class CreateContentRequest
    # @!attribute name
    # @return [String] the name of the content
    attr_accessor :name
    # and many more args
    def initialize(name:, message: content_type: "article", ...)
    end
    def to_param
    {
    name: name, message: message, contentType: content_type, ...
    }
    end
    end
    

    View Slide

87. 87
    5. Introduce parameter object
    # @param request [Array]
    def create_content (request)
    xml = request.map(&to_param).to_xml
    # send a request to API with XML
    end
    

    View Slide

88. ● Pros
    ○ Parameter object itself can have conversion method
    ○ Easy to write and read comment
    ● If it gets easier to write YARD comment, consider adopting parameter object
    ○ https://github.com/lsegal/yard
    ● c.f. Refactoring: Ruby Edition
    ○ https://www.amazon.com/dp/0321984137
    88
    5. Introduce parameter object
    

    View Slide

89. 89
    6. Method accessible response
    

    View Slide

90. ● The right is Rubyish
    90
    6. Method accessible response
    contents = api_client.get_contents
    first_content_name =
    contents[0][“name”]
    names = contents.map do |content|
    content["name"]
    end
    contents = api_client.get_contents
    first_content_name =
    contents[0].name
    names = contents.map( &:name)
    VS
    

    View Slide

91. ● Recommend to use mashify
    ○ https://github.com/lostisland/faraday_middleware
    91
    6. Method accessible response
    connection = Faraday.new('http://example.com/api' ) do |conn|
    conn.response :mashify
    conn.adapter Faraday.default_adapter
    end
    

    View Slide

92. 92
    7. curl is universal language
    

    View Slide

93. ● When I create API client, I often ask questions about unclear things or bugs
    ● Writing sample code in ruby is not appropriate
    ○ Reason 1. API providers don't necessarily know Ruby
    ○ Reason 2. This may be bugs or specification of Ruby. (instead of API)
    93
    7. curl is universal language
    

    View Slide

94. ● curl isn’t dependent on a specific language
    ● curl can express all contexts with oneliner
    94
    Why should we use curl ?
    $ curl -X GET “https://example.com/api/user” -H
    “X-ACCESS-TOKEN: $ACCESS_TOKEN” -H “X-ACCESS-SECRET:
    $ACCESS_SECRET”
    

    View Slide

95. ● `curl 〜 | jq .` is good for reporting
    ○ I always use this
    95
    Why should we use curl ?
    $ curl -X GET “https://example.com/api/user” -H
    “X-ACCESS-TOKEN: $ACCESS_TOKEN” -H “X-ACCESS-SECRET:
    $ACCESS_SECRET” | jq .
    {
    “name”: “sue445”
    }
    

    View Slide

96. ● `curl 〜 | jq . | pbcopy`
    ○ Copy response to clickboard (Mac only)
    96
    Why should we use curl ?
    $ curl -X GET “https://example.com/api/user” -H
    “X-ACCESS-TOKEN: $ACCESS_TOKEN” -H “X-ACCESS-SECRET:
    $ACCESS_SECRET” | jq . | pbcopy
    

    View Slide

97. ● https://github.com/mauricio/faraday_curl
    ○ This middleware logs your HTTP requests as CURL compatible commands so
    you can share the calls you're making with someone else or keep them for
    debugging purposes.
    97
    Auto debug logging in curl format
    

    View Slide

98. https://github.com/sue445/pixela/blob/v1.1.0/lib/pixela/client.rb#L55-L69
    98
    Example
    def connection(request_headers = user_token_headers)
    Faraday.new(url: API_ENDPOINT, headers: request_headers) do |conn|
    conn.request :json
    conn.response :mashify, mash_class: Pixela::Response
    conn.response :json
    conn.response :raise_error
    if Pixela.config.debug_logger
    conn.request :curl, Pixela.config.debug_logger, :debug
    conn.response :logger, Pixela.config.debug_logger
    end
    conn.adapter Faraday.default_adapter
    end
    end
    

    View Slide

99. 99
    Example
    [1] pry(main)> @client.get_pixel(graph_id: "tweets", date: Date.new(2018, 9, 15))
    D, [2019-04-03T01:36:05.905606 #2167] DEBUG -- : curl -v -X GET -H 'X-USER-TOKEN:
    XXXXXXXXXXXX -H 'User-Agent: Pixela v1.1.0 (https://github.com/sue445/pixela)' -H
    'Content-Type: application/json'
    "https://pixe.la/v1/users/sue445/graphs/tweets/20180915"
    

    View Slide

100. ● There are many reasons to create an API client
     ● Responsibilities of API clients are SRP
     ● Tips in my API clients
     ● API Client 7 Good Patterns
     ● Read “Refactoring: Ruby Edition”
     ● Happy coding
     100
     Conclusion
     

     View Slide