Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building a Resilient API with open source

Building a Resilient API with open source

Tools and techniques GitHub employs to handle API changes.

Wynn Netherland

July 23, 2014
Tweet

More Decks by Wynn Netherland

Other Decks in Programming

Transcript

  1. ! I make software for computers who work for people

    who make software and other stuff. — me, at Thanksgiving ” ”
  2. ! re·sil·ient /ri'zilyənt/ able to recoil or spring back into

    shape after stretching, bending, or compressing "
  3. ! measure change & minimize change ' handle change (

    At GitHub, we so we can in order to
  4. ! ”If an API method is removed in the woods,

    and nobody is there to call it, does it still respond with 410?” ! — unknown
  5. !

  6. !

  7. ! /splunk -3m @production "timeline.json" ! pengwynn: ! 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd

    local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" request_id="AD003EA0:5BD8:4B8AE05:53BEC973" remote_address=192.168.1.1 request_method=get path_info="/timeline.json" content_length=375 user_agent="@gitlost account on twitter, Python urllib2" accept=nil language=nil status=410 at=finish elapsed=0.005 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" user=nil repo=nil route=nil zone=UTC referrer=nil requested_at="2014-07-10 17:12:19 UTC" url="https://github.com/timeline.json" controller=EventsController action=index request_id="AD003EA0:5BD8:4B8AE05:53BEC973" dbconn=write serializer=json at=start 2014-07-10T10:12:19-07:00 github-fe117-cp1-prd local7:info unicorn_rails[12813]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00"
  8. ! /splunk -3m @production "timeline.json" ! pengwynn: ! 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd

    local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" request_id="AD003EA0:5BD8:4B8AE05:53BEC973" remote_address=192.168.1.1 request_method=get path_info="/timeline.json" content_length=375 user_agent="@gitlost account on twitter, Python urllib2" accept=nil language=nil status=410 at=finish elapsed=0.005 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" user=nil repo=nil route=nil zone=UTC referrer=nil requested_at="2014-07-10 17:12:19 UTC" url="https://github.com/timeline.json" controller=EventsController action=index request_id="AD003EA0:5BD8:4B8AE05:53BEC973" dbconn=write serializer=json at=start 2014-07-10T10:12:19-07:00 github-fe117-cp1-prd local7:info unicorn_rails[12813]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00"
  9. ! /splunk -3m @production "timeline.json" | stats count by status,

    avg(elapsed) ! pengwynn: ┌──────┬────────────┐ │status│avg(elapsed)│ ├──────┼────────────┤ │200 │0.067745 │ ├──────┼────────────┤ │410 │0.005597 │ └──────┴────────────┘
  10. ! curl https://api.github.com/ -I | grep Request-Id ❯ / !

    X-GitHub-Request-Id: 4317CCE2:6434:21DCB73:53CD6239
  11. ! curl https://api.github.com/ -I | grep Request-Id ❯ / !

    X-GitHub-Request-Id: 4317CCE2:6434:21DCB73:53CD6239 A single identifier to trace requests through logging.
  12. ! tree -d -L 2 github ❯ / github ├──

    app │ ├── api │ ├── assets │ ├── controllers │ ├── helpers │ ├── mailers │ ├── models │ ├── view_models │ └── views ├── bin ├── config │ ├── inititalizers ...
  13. ! tree -d -L 2 github ❯ / github ├──

    app │ ├── api │ ├── assets │ ├── controllers │ ├── helpers │ ├── mailers │ ├── models │ ├── view_models │ └── views ├── bin ├── config │ ├── inititalizers ... Sinatra app inside a Rails app
  14. ! tree -d -L 2 github ❯ / github ├──

    app │ ├── api │ ├── assets │ ├── controllers │ ├── helpers │ ├── mailers │ ├── models │ ├── view_models │ └── views ├── bin ├── config │ ├── inititalizers ... Sinatra app inside a Rails app
  15. ! Testing ❯ testrb test/integration/api/repos_test.rb Loading suite in dotcom mode

    ... Run options: # Running tests: Finished tests in 30.638101s, 4.7979 tests/s, 64.7886 assertions/s. 147 tests, 1985 assertions, 0 failures, 0 errors, 0 skips ruby -v: ruby 2.1.0p0-github (development) [x86_64-darwin13.0]
  16. ! # test/integration/api/root_test.rb ! test "provides hypermedia URLs to top-level

    resources" do get "/" data = JSON.parse(last_response.body) assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -
  17. ! # test/integration/api/root_test.rb ! test "provides hypermedia URLs to top-level

    resources" do get "/" data = JSON.parse(last_response.body) assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -
  18. ! # test/integration/api/root_test.rb ! test "provides hypermedia URLs to top-level

    resources" do data = api :get, "/" ! assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -
  19. ! / # Running tests: ! [1/5] Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 = 2.06

    s 1) Failure: Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 [/Users/ wynn/github/github/test/integration/api/rate_limit_test.rb:67]: {"resources"=>{"core"=>{}, "search"=>{}}, "rate"=>{}} The property '#/rate' did not contain a required property of 'limit' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'remaining' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'reset' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1#
  20. ! / # Running tests: ! [1/5] Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 = 2.06

    s 1) Failure: Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 [/Users/ wynn/github/github/test/integration/api/rate_limit_test.rb:67]: {"resources"=>{"core"=>{}, "search"=>{}}, "rate"=>{}} The property '#/rate' did not contain a required property of 'limit' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'remaining' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'reset' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1#
  21. ! / # Running tests: ! [1/5] Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 = 2.06

    s 1) Failure: Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 [/Users/ wynn/github/github/test/integration/api/rate_limit_test.rb:67]: {"resources"=>{"core"=>{}, "search"=>{}}, "rate"=>{}} The property '#/rate' did not contain a required property of 'limit' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'remaining' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'reset' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1#
  22. ! / # update a label $ curl -ni http://api.github.dev/repos/defunkt/dotjs/labels/bug

    \ -X PATCH -H "Content-Type: application/json" \ -d '{ "name":[], "color":42 } ! HTTP/1.1 422 Unprocessable Entity ! { "message": "Invalid request.\n\nExpected data to be of type \"string \"; value was: [].\nExpected data to be of type \"string\"; value was: 42.", "documentation_url": "https://developer.github.com/v3/issues/labels/ #update-a-label" }
  23. ! / # update a label $ curl -ni http://api.github.dev/repos/defunkt/dotjs/labels/bug

    \ -X PATCH -H "Content-Type: application/json" \ -d '{ "name":[], "color":42 } ! HTTP/1.1 422 Unprocessable Entity ! { "message": "Invalid request.\n\nExpected data to be of type \"string \"; value was: [].\nExpected data to be of type \"string\"; value was: 42.", "documentation_url": "https://developer.github.com/v3/issues/labels/ #update-a-label" }
  24. ! test "can create a pull request" do auth_as @forker

    ! data = api :post, @pub_pulls_url, {}, :input => { :title => "new pr", :body => "this is an amazing change", :base => "master", :head => "#{@forker}:ahead" }.to_json assert_equal 201, last_response.status assert_schema :v3, :pull_request, data end -
  25. ! test "can create a pull request" do auth_as @forker

    ! data = api :post, @pub_pulls_url, {}, :input => { :title => "new pr", :body => "this is an amazing change", :base => "master", :head => "#{@forker}:ahead" }.to_json assert_equal 201, last_response.status assert_schema :v3, :pull_request, data end -
  26. ! test "can retrieve pull requests" do auth_as @owner !

    data = api :get, @pub_pulls_url assert_equal 200, last_response.status ! assert_schema_for_collection \ :v3, :pull_request_simple, data end -
  27. ! test "can retrieve pull requests" do auth_as @owner !

    data = api :get, @pub_pulls_url assert_equal 200, last_response.status ! assert_schema_for_collection \ :v3, :pull_request_simple, data end -
  28. ! test "authz list commits for pull request" do !

    assert_authz :get, @priv_commits_url do |check| check.disallows_with :not_found check.authorized_users @owner, @collaborator check.unauthorized_users :anon, @rando check.minimum_scopes :repo end end -
  29. ! test "authz list commits for pull request" do !

    assert_authz :get, @priv_commits_url do |check| check.disallows_with :not_found check.authorized_users @owner, @collaborator check.unauthorized_users :anon, @rando check.minimum_scopes :repo end end -
  30. ! test "authz list commits for pull request" do !

    assert_authz :get, @priv_commits_url do |check| check.disallows_with :not_found check.authorized_users @owner, @collaborator check.unauthorized_users :anon, @rando check.minimum_scopes :repo end end -
  31. ! test "authz list commits for pull request" do !

    assert_authz :get, @priv_commits_url do |check| check.disallows_with :not_found check.authorized_users @owner, @collaborator check.unauthorized_users :anon, @rando check.minimum_scopes :repo end end -
  32. ! test "authz list commits for pull request" do !

    assert_authz :get, @priv_commits_url do |check| check.disallows_with :not_found check.authorized_users @owner, @collaborator check.unauthorized_users :anon, @rando check.minimum_scopes :repo end end -
  33. ! test "authz list commits for pull request" do !

    assert_authz :get, @priv_commits_url do |check| check.disallows_with :not_found check.authorized_users @owner, @collaborator check.unauthorized_users :anon, @rando check.minimum_scopes :repo end end -
  34. !

  35. !

  36. ! require "dat/science" ! class MyApp::Widget def allows?(user) experiment =

    Dat::Science::Experiment.new \ "widget-permissions" do |e| ! e.control { old_method } # old way e.candidate { new_method } # new way end ! experiment.run end end -
  37. ! require "dat/science" ! class MyApp::Widget def allows?(user) experiment =

    Dat::Science::Experiment.new \ "widget-permissions" do |e| ! e.control { old_method } # old way e.candidate { new_method } # new way end ! experiment.run end end -
  38. ! require "dat/science" ! class MyApp::Widget def allows?(user) experiment =

    Dat::Science::Experiment.new \ "widget-permissions" do |e| ! e.control { old_method } # old way e.candidate { new_method } # new way end ! experiment.run end end -
  39. ! require "dat/science" ! class MyApp::Widget def allows?(user) experiment =

    Dat::Science::Experiment.new \ "widget-permissions" do |e| ! e.control { old_method } # old way e.candidate { new_method } # new way end ! experiment.run end end -
  40. ! require "dat/science" ! class MyApp::Widget def allows?(user) experiment =

    Dat::Science::Experiment.new \ "widget-permissions" do |e| ! e.control { old_method } # old way e.candidate { new_method } # new way end ! experiment.run end end -
  41. ! require "dat/science" ! class MyApp::Widget def allows?(user) experiment =

    Dat::Science::Experiment.new \ "widget-permissions" do |e| ! e.control { old_method } # old way e.candidate { new_method } # new way end ! experiment.run end end -
  42. Runs candidate before control 50% of the time Measures the

    duration of both behaviors Compares the results of both behaviors Swallows any exceptions raised by the candidate behavior Publishes findings for tracking and reporting
  43. ! require "dat/science" ! class MyApp::Widget def allows?(user) science "widget-permissions"

    do |e| e.control { model.check_user(user).valid? } e.candidate { user.can? :read, model } end end end -
  44. ! require "dat/science" ! class MyApp::Widget def allows?(user) science "widget-permissions"

    do |e| e.control { model.check_user(user).valid? } e.candidate { user.can? :read, model } end end end -
  45. ! require "dat/science" ! class Blog::Post def permalink(user) science "permalink"

    do |e| e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -
  46. ! require "dat/science" ! class Blog::Post def permalink(user) science "permalink"

    do |e| e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -
  47. ! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment

    def enabled? rand(100) < 10 # Run 10% of the time end end end -
  48. ! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment

    def enabled? MyApp.flipper[name].enabled? end end end -
  49. ! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment

    def publish MyApp.instrument "science.#{event}", payload end end end -
  50. ! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment

    def publish MyApp.instrument "science.#{event}", payload end end end -
  51. ! require "dat/science" ! class Blog::Post def permalink(user) science "permalink"

    do |e| e.context :post => post e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -
  52. ! require "dat/science" ! class Blog::Post def permalink(user) science "permalink"

    do |e| e.context :post => post e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -
  53. ! module GitHub class Experiment < Dat::Science::Experiment class << self

    # Public: enable mad science mode: returns the candidate values by default # instead of the control. def mad_science=(val) @mad_science = val end ! # Internal: whether or not to always run the candidate instead of the # control. def candidate_instead_of_control? @mad_science end end … end -
  54. ! module GitHub class Experiment < Dat::Science::Experiment class << self

    # Public: enable mad science mode: returns the candidate values by default # instead of the control. def mad_science=(val) @mad_science = val end ! # Internal: whether or not to always run the candidate instead of the # control. def candidate_instead_of_control? @mad_science end end … end -
  55. ! require 'dat/analysis' ! module MyApp class Analysis < Dat::Analysis

    def read Redis.rpop "dat-science.#{experiment_name}.results" end ! def count Redis.llen "dat-science.#{experiment_name}.results" end ! def cook(raw_result) return nil unless raw_result JSON.parse(raw_result) end end end -
  56. ! require 'dat/analysis' ! module MyApp class Analysis < Dat::Analysis

    def read Redis.rpop "dat-science.#{experiment_name}.results" end ! def count Redis.llen "dat-science.#{experiment_name}.results" end ! def cook(raw_result) return nil unless raw_result JSON.parse(raw_result) end end end -
  57. ! require 'dat/analysis' ! module MyApp class Analysis < Dat::Analysis

    def read Redis.rpop "dat-science.#{experiment_name}.results" end ! def count Redis.llen "dat-science.#{experiment_name}.results" end ! def cook(raw_result) return nil unless raw_result JSON.parse(raw_result) end end end -
  58. ! require 'dat/analysis' ! module MyApp class Analysis < Dat::Analysis

    def read Redis.rpop "dat-science.#{experiment_name}.results" end ! def count Redis.llen "dat-science.#{experiment_name}.results" end ! def cook(raw_result) return nil unless raw_result JSON.parse(raw_result) end end end -
  59. ! require 'dat/analysis' ! module MyApp class Analysis < Dat::Analysis

    def read Redis.rpop "dat-science.#{experiment_name}.results" end ! def count Redis.llen "dat-science.#{experiment_name}.results" end ! def cook(raw_result) return nil unless raw_result JSON.parse(raw_result) end end end -
  60. ! require 'dat/analysis' ! module MyApp class Analysis < Dat::Analysis

    def read Redis.rpop "dat-science.#{experiment_name}.results" end ! def count Redis.llen "dat-science.#{experiment_name}.results" end ! def cook(raw_result) return nil unless raw_result JSON.parse(raw_result) end end end -
  61. ! / irb> a = MyApp::Analysis.new('widget-permissions') => #<MyApp::Analysis:0x007fae4a0101f8 …> !

    irb> a.fetch => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } ! !
  62. ! / irb> a = MyApp::Analysis.new('widget-permissions') => #<MyApp::Analysis:0x007fae4a0101f8 …> !

    irb> a.fetch => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } ! irb> a.result => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } !
  63. ! / irb> a.result.control => {"duration"=>12.307, "exception"=>nil, "value"=>false} ! irb>

    a.result.candidate => {"duration"=>12.366999999999, "exception"=>nil, "value"=>true} !
  64. ! / curl -n http://api.github.com/user/emails \ -H "Accept: application/vnd.github.v3" [

    { "email": "[email protected]", "primary": false, "verified": true }, { "email": "[email protected]", "primary": false, "verified": true } ] ❯
  65. ! / curl -n http://api.github.com/user/emails \ -H "Accept: application/vnd.github.v3" [

    { "email": "[email protected]", "primary": false, "verified": true }, { "email": "[email protected]", "primary": false, "verified": true } ] ❯
  66. ! / curl -n http://api.github.com/user/emails \ -H "Accept: application/vnd.github.v3" [

    { "email": "[email protected]", "primary": false, "verified": true }, { "email": "[email protected]", "primary": false, "verified": true } ] ❯
  67. ! curl https://api.github.com/users/technoweenie -I curl https://api.github.com/users/technoweenie -I \ -H "Accept:

    application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json
  68. ! curl https://api.github.com/users/technoweenie -I curl https://api.github.com/users/technoweenie -I \ -H "Accept:

    application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json
  69. ! curl https://api.github.com/users/technoweenie -I curl https://api.github.com/users/technoweenie -I \ -H "Accept:

    application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json
  70. ! curl https://api.github.com/users/technoweenie -I curl https://api.github.com/users/technoweenie -I \ -H "Accept:

    application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json curl https://api.github.com/users/technoweenie -I \ -H "Accept: application/vnd.github.beta.full+json" HTTP/1.1 200 OK X-GitHub-Media-Type: github.beta; param=full; format=json
  71. ! curl https://api.github.com/users/technoweenie -I curl https://api.github.com/users/technoweenie -I \ -H "Accept:

    application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json curl https://api.github.com/users/technoweenie -I \ -H "Accept: application/vnd.github.beta.full+json" HTTP/1.1 200 OK X-GitHub-Media-Type: github.beta; param=full; format=json
  72. ! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !

    { "message": "Unsupported 'Accept' header: github.v3." }
  73. ! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !

    { "message": "Unsupported 'Accept' header: github.v3." }
  74. ! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !

    { "message": "Unsupported 'Accept' header: github.v3." }
  75. ! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !

    { "message": "Unsupported 'Accept' header: github.v3." } curl https://api.github.com/the/new/shiny \ -H "Accept: application/vnd.github.new-shiny-preview" HTTP/1.1 200 OK
  76. ! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !

    { "message": "Unsupported 'Accept' header: github.v3." } curl https://api.github.com/the/new/shiny \ -H "Accept: application/vnd.github.new-shiny-preview" HTTP/1.1 200 OK
  77. ! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !

    { "message": "Unsupported 'Accept' header: github.v3." } curl https://api.github.com/the/new/shiny \ -H "Accept: application/vnd.github.new-shiny-preview" HTTP/1.1 200 OK acknowledge API is a preview release
  78. ! describe Octokit::Client do ! describe ".rate_limit" do it "makes

    a response if there is no last response" do client = Octokit::Client.new VCR.use_cassette "rate_limit" do rate = client.rate_limit ! expect(rate.limit).to be_kind_of Fixnum expect(rate.remaining).to be_kind_of Fixnum end end # .rate_limit end … -
  79. ! describe Octokit::Client do ! describe ".rate_limit" do it "makes

    a response if there is no last response" do client = Octokit::Client.new VCR.use_cassette "rate_limit" do rate = client.rate_limit ! expect(rate.limit).to be_kind_of Fixnum expect(rate.remaining).to be_kind_of Fixnum end end # .rate_limit end … -
  80. ! describe Octokit::Client do ! describe ".rate_limit" do it "makes

    a response if there is no last response" do client = Octokit::Client.new VCR.use_cassette "rate_limit" do rate = client.rate_limit ! expect(rate.limit).to be_kind_of Fixnum expect(rate.remaining).to be_kind_of Fixnum end end # .rate_limit end … -
  81. ! describe Octokit::Client do ! describe ".rate_limit" do it "makes

    a response if there is no last response" do client = Octokit::Client.new VCR.use_cassette "rate_limit" do rate = client.rate_limit ! expect(rate.limit).to be_kind_of Fixnum expect(rate.remaining).to be_kind_of Fixnum end end # .rate_limit end … -
  82. ! / rm -rf spec/cassettes ❯ ~/.coral/repos/octokit.rb@octokit master OCTOKIT_API_ENDPOINT="https://rails5.github.com" rspec

    ❯ ~/.coral/repos/octokit.rb@octokit master* ................................................. ..............F.................................. ........................FFF...................... ...................F............................. ........F........................................ ................................................. .....................F........F..................
  83. ! measure change & minimize change ' handle change (

    At GitHub, we so we can in order to