Building a Resilient API with open source

local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" request_id="AD003EA0:5BD8:4B8AE05:53BEC973" remote_address=192.168.1.1 request_method=get path_info="/timeline.json" content_length=375 user_agent="@gitlost account on twitter, Python urllib2" accept=nil language=nil status=410 at=finish elapsed=0.005 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" user=nil repo=nil route=nil zone=UTC referrer=nil requested_at="2014-07-10 17:12:19 UTC" url="https://github.com/timeline.json" controller=EventsController action=index request_id="AD003EA0:5BD8:4B8AE05:53BEC973" dbconn=write serializer=json at=start 2014-07-10T10:12:19-07:00 github-fe117-cp1-prd local7:info unicorn_rails[12813]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00"

! /splunk -3m @production "timeline.json" ! pengwynn: ! 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd

local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" request_id="AD003EA0:5BD8:4B8AE05:53BEC973" remote_address=192.168.1.1 request_method=get path_info="/timeline.json" content_length=375 user_agent="@gitlost account on twitter, Python urllib2" accept=nil language=nil status=410 at=finish elapsed=0.005 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" user=nil repo=nil route=nil zone=UTC referrer=nil requested_at="2014-07-10 17:12:19 UTC" url="https://github.com/timeline.json" controller=EventsController action=index request_id="AD003EA0:5BD8:4B8AE05:53BEC973" dbconn=write serializer=json at=start 2014-07-10T10:12:19-07:00 github-fe117-cp1-prd local7:info unicorn_rails[12813]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00"

! /splunk -3m @production "timeline.json" | stats count by status,

avg(elapsed) ! pengwynn: ┌──────┬────────────┐ │status│avg(elapsed)│ ├──────┼────────────┤ │200 │0.067745 │ ├──────┼────────────┤ │410 │0.005597 │ └──────┴────────────┘

! key=value logging

! / app=github env=development enterprise=false now="2014-07-21T11:49:45-07:00" request_id=f80765a8-8aa9-4e92-b32b-01300193631a remote_address=127.0.0.1 request_method=get path_info="/"

content_length=1866 user_agent=curl/7.30.0 accept=*/* language=nil status=200 at=finish elapsed=0.110

! / app=github env=development enterprise=false now="2014-07-21T11:49:45-07:00" request_id=f80765a8-8aa9-4e92-b32b-01300193631a remote_address=127.0.0.1 request_method=get path_info="/"

content_length=1866 user_agent=curl/7.30.0 accept=*/* language=nil status=200 at=finish elapsed=0.110

! curl https://api.github.com/ -I | grep Request-Id ❯ / !

X-GitHub-Request-Id: 4317CCE2:6434:21DCB73:53CD6239

! curl https://api.github.com/ -I | grep Request-Id ❯ / !

X-GitHub-Request-Id: 4317CCE2:6434:21DCB73:53CD6239 A single identifier to trace requests through logging.

! /splunk -3m @production request_id=f80765a8-8aa9-4e92- b32b-01300193631a !

! github.com/asenchi/scrolls

! github.com/technoweenie/grohl If Golang is your thang.

! Logstash, Splunk, Loggly *SaaS solutions, not open source

! Minimizing change '

! Isolation

! tree -d -L 2 github ❯ / github ├──

app │ ├── api │ ├── assets │ ├── controllers │ ├── helpers │ ├── mailers │ ├── models │ ├── view_models │ └── views ├── bin ├── config │ ├── inititalizers ...

! tree -d -L 2 github ❯ / github ├──

app │ ├── api │ ├── assets │ ├── controllers │ ├── helpers │ ├── mailers │ ├── models │ ├── view_models │ └── views ├── bin ├── config │ ├── inititalizers ... Sinatra app inside a Rails app

! tree -d -L 2 github ❯ / github ├──

app │ ├── api │ ├── assets │ ├── controllers │ ├── helpers │ ├── mailers │ ├── models │ ├── view_models │ └── views ├── bin ├── config │ ├── inititalizers ... Sinatra app inside a Rails app

! Shared business logic data access logging instrumentation plumbing Independent

workﬂows output

! Testing ❯ testrb test/integration/api/repos_test.rb Loading suite in dotcom mode

... Run options: # Running tests: Finished tests in 30.638101s, 4.7979 tests/s, 64.7886 assertions/s. 147 tests, 1985 assertions, 0 failures, 0 errors, 0 skips ruby -v: ruby 2.1.0p0-github (development) [x86_64-darwin13.0]

! Rack::Test

! brynary/rack-test github.com/

! # test/integration/api/root_test.rb ! test "provides hypermedia URLs to top-level

resources" do get "/" data = JSON.parse(last_response.body) assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -

! # test/integration/api/root_test.rb ! test "provides hypermedia URLs to top-level

resources" do get "/" data = JSON.parse(last_response.body) assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -

! # test/integration/api/root_test.rb ! test "provides hypermedia URLs to top-level

resources" do data = api :get, "/" ! assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -

! JSON Schema

! json-schema.org

! Validating output.

! / # Running tests: ! [1/5] Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 = 2.06

s 1) Failure: Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 [/Users/ wynn/github/github/test/integration/api/rate_limit_test.rb:67]: {"resources"=>{"core"=>{}, "search"=>{}}, "rate"=>{}} The property '#/rate' did not contain a required property of 'limit' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'remaining' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'reset' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1#

! / # Running tests: ! [1/5] Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 = 2.06

! Validating input.

! / # update a label $ curl -ni http://api.github.dev/repos/defunkt/dotjs/labels/bug

\ -X PATCH -H "Content-Type: application/json" \ -d '{ "name":[], "color":42 } ! HTTP/1.1 422 Unprocessable Entity ! { "message": "Invalid request.\n\nExpected data to be of type \"string \"; value was: [].\nExpected data to be of type \"string\"; value was: 42.", "documentation_url": "https://developer.github.com/v3/issues/labels/ #update-a-label" }

! / # update a label $ curl -ni http://api.github.dev/repos/defunkt/dotjs/labels/bug

\ -X PATCH -H "Content-Type: application/json" \ -d '{ "name":[], "color":42 } ! HTTP/1.1 422 Unprocessable Entity ! { "message": "Invalid request.\n\nExpected data to be of type \"string \"; value was: [].\nExpected data to be of type \"string\"; value was: 42.", "documentation_url": "https://developer.github.com/v3/issues/labels/ #update-a-label" }

! MiniTest

! Custom assertions

! test "can create a pull request" do auth_as @forker

! data = api :post, @pub_pulls_url, {}, :input => { :title => "new pr", :body => "this is an amazing change", :base => "master", :head => "#{@forker}:ahead" }.to_json assert_equal 201, last_response.status assert_schema :v3, :pull_request, data end -

! test "can create a pull request" do auth_as @forker

! data = api :post, @pub_pulls_url, {}, :input => { :title => "new pr", :body => "this is an amazing change", :base => "master", :head => "#{@forker}:ahead" }.to_json assert_equal 201, last_response.status assert_schema :v3, :pull_request, data end -

! test "can retrieve pull requests" do auth_as @owner !

data = api :get, @pub_pulls_url assert_equal 200, last_response.status ! assert_schema_for_collection \ :v3, :pull_request_simple, data end -

! test "can retrieve pull requests" do auth_as @owner !

data = api :get, @pub_pulls_url assert_equal 200, last_response.status ! assert_schema_for_collection \ :v3, :pull_request_simple, data end -

! Even more assertions

! test "authz list commits for pull request" do !

assert_authz :get, @priv_commits_url do |check| check.disallows_with :not_found check.authorized_users @owner, @collaborator check.unauthorized_users :anon, @rando check.minimum_scopes :repo end end -

! test "authz list commits for pull request" do !

! 5 lines = dozens of AuthZ assertions.

! But what about those critical code paths?

! What if you don’t trust your tests?

! Science 0

! github/dat-science github.com/

! @jbarnette

! @rick

!

! Safely observe alternate code paths for non-mutating methods that

return a value.

! require "dat/science" ! class MyApp::Widget def allows?(user) experiment =

Dat::Science::Experiment.new \ "widget-permissions" do |e| ! e.control { old_method } # old way e.candidate { new_method } # new way end ! experiment.run end end -

! require "dat/science" ! class MyApp::Widget def allows?(user) experiment =

Runs candidate before control 50% of the time Measures the

duration of both behaviors Compares the results of both behaviors Swallows any exceptions raised by the candidate behavior Publishes ﬁndings for tracking and reporting

! Fancy a DSL instead?

! require "dat/science" ! class MyApp::Widget def allows?(user) science "widget-permissions"

do |e| e.control { model.check_user(user).valid? } e.candidate { user.can? :read, model } end end end -

! require "dat/science" ! class MyApp::Widget def allows?(user) science "widget-permissions"

do |e| e.control { model.check_user(user).valid? } e.candidate { user.can? :read, model } end end end -

! Create and run.

! require "dat/science" ! class Blog::Post def permalink(user) science "permalink"

do |e| e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -

! require "dat/science" ! class Blog::Post def permalink(user) science "permalink"

do |e| e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -

! Ramping up experiments.

! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment

def enabled? rand(100) < 10 # Run 10% of the time end end end -

! github.com/jnunemaker/ﬂipper!

! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment

def enabled? MyApp.flipper[name].enabled? end end end -

! Publishing results.

! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment

def publish MyApp.instrument "science.#{event}", payload end end end -

! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment

def publish MyApp.instrument "science.#{event}", payload end end end -

None

! Adding custom context.

! require "dat/science" ! class Blog::Post def permalink(user) science "permalink"

do |e| e.context :post => post e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -

! require "dat/science" ! class Blog::Post def permalink(user) science "permalink"

do |e| e.context :post => post e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -

Mad science.

1 Don't try this at home

! module GitHub class Experiment < Dat::Science::Experiment class << self

# Public: enable mad science mode: returns the candidate values by default # instead of the control. def mad_science=(val) @mad_science = val end ! # Internal: whether or not to always run the candidate instead of the # control. def candidate_instead_of_control? @mad_science end end … end -

! module GitHub class Experiment < Dat::Science::Experiment class << self

# Public: enable mad science mode: returns the candidate values by default # instead of the control. def mad_science=(val) @mad_science = val end ! # Internal: whether or not to always run the candidate instead of the # control. def candidate_instead_of_control? @mad_science end end … end -

! Analyzing results.

! github.com/github/dat-analysis

! Build an analyzer to poke on results.

! require 'dat/analysis' ! module MyApp class Analysis < Dat::Analysis

def read Redis.rpop "dat-science.#{experiment_name}.results" end ! def count Redis.llen "dat-science.#{experiment_name}.results" end ! def cook(raw_result) return nil unless raw_result JSON.parse(raw_result) end end end -

! require 'dat/analysis' ! module MyApp class Analysis < Dat::Analysis

! / irb> a = MyApp::Analysis.new('widget-permissions') => #<MyApp::Analysis:0x007fae4a0101f8 …> !

! !

! / irb> a = MyApp::Analysis.new('widget-permissions') => #<MyApp::Analysis:0x007fae4a0101f8 …> !

irb> a.fetch => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } ! !

! / irb> a = MyApp::Analysis.new('widget-permissions') => #<MyApp::Analysis:0x007fae4a0101f8 …> !

irb> a.fetch => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } ! irb> a.result => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } !

! / irb> a.result.control => {"duration"=>12.307, "exception"=>nil, "value"=>false} ! !

!

! / irb> a.result.control => {"duration"=>12.307, "exception"=>nil, "value"=>false} ! irb>

a.result.candidate => {"duration"=>12.366999999999, "exception"=>nil, "value"=>true} !

! Handling change (

! New formats

! Features evolve. URLs don’t have to.

! / [ "wynn@example.com", "pengwynn@example.com" ] curl -n http://api.github.com/user/emails ❯

! / curl -n http://api.github.com/user/emails \ -H "Accept: application/vnd.github.v3" [

{ "email": "wynn@example.com", "primary": false, "verified": true }, { "email": "pengwynn@example.com", "primary": false, "verified": true } ] ❯

! / curl -n http://api.github.com/user/emails \ -H "Accept: application/vnd.github.v3" [

! Media types

! curl https://api.github.com/users/technoweenie -I / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3

! curl https://api.github.com/users/technoweenie -I curl https://api.github.com/users/technoweenie -I \ -H "Accept:

application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json

! curl https://api.github.com/users/technoweenie -I curl https://api.github.com/users/technoweenie -I \ -H "Accept:

application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json curl https://api.github.com/users/technoweenie -I \ -H "Accept: application/vnd.github.beta.full+json" HTTP/1.1 200 OK X-GitHub-Media-Type: github.beta; param=full; format=json

! curl https://api.github.com/users/technoweenie -I curl https://api.github.com/users/technoweenie -I \ -H "Accept:

application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json curl https://api.github.com/users/technoweenie -I \ -H "Accept: application/vnd.github.beta.full+json" HTTP/1.1 200 OK X-GitHub-Media-Type: github.beta; param=full; format=json

! @jasonrudolph “Beta like you mean it.”

! Media types as beta code words.

! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !

{ "message": "Unsupported 'Accept' header: github.v3." }

! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !

{ "message": "Unsupported 'Accept' header: github.v3." } curl https://api.github.com/the/new/shiny \ -H "Accept: application/vnd.github.new-shiny-preview" HTTP/1.1 200 OK

! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !

{ "message": "Unsupported 'Accept' header: github.v3." } curl https://api.github.com/the/new/shiny \ -H "Accept: application/vnd.github.new-shiny-preview" HTTP/1.1 200 OK

! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !

{ "message": "Unsupported 'Accept' header: github.v3." } curl https://api.github.com/the/new/shiny \ -H "Accept: application/vnd.github.new-shiny-preview" HTTP/1.1 200 OK acknowledge API is a preview release

! Client testing

None

! octokit.github.com

! github.com/octokit/octokit.rb

! github.com/rspec/rspec

! describe Octokit::Client do ! describe ".rate_limit" do it "makes

a response if there is no last response" do client = Octokit::Client.new VCR.use_cassette "rate_limit" do rate = client.rate_limit ! expect(rate.limit).to be_kind_of Fixnum expect(rate.remaining).to be_kind_of Fixnum end end # .rate_limit end … -

! describe Octokit::Client do ! describe ".rate_limit" do it "makes

! Record live API requests to play back later. Is

it live, or is it Memorex®

! github.com/vcr/vcr

! Environment smoke tests

! / rm -rf spec/cassettes ❯ ~/.coral/repos/octokit.rb@octokit master ❯ ~/.coral/repos/octokit.rb@octokit

master*

! / rm -rf spec/cassettes ❯ ~/.coral/repos/octokit.rb@octokit master OCTOKIT_API_ENDPOINT="https://rails5.github.com" rspec

❯ ~/.coral/repos/octokit.rb@octokit master* ................................................. ..............F.................................. ........................FFF...................... ...................F............................. ........F........................................ ................................................. .....................F........F..................

Building a Resilient API with open source

Building a Resilient API with open source

More Decks by Wynn Netherland

Other Decks in Programming

Featured

Transcript