Building a Resilient API with open source

Slide 1

Slide 1 text

! Building a Resilient API with open source Wynn Netherland GitHub

Slide 2

Slide 2 text

! Hello, my name is Wynn.

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

 Folks call me @pengwynn.

Slide 5

Slide 5 text

! I work @ GitHub.

Slide 6

Slide 6 text

! GitHub makes software for people who make software.

Slide 7

Slide 7 text

! I work mostly on the GitHub API.

Slide 8

Slide 8 text

! I make software for computers who work for people who make software and other stuﬀ. — me, at Thanksgiving ” ”

Slide 9

Slide 9 text

! Building a Resilient API with open source I want to talk about

Slide 10

Slide 10 text

! re·sil·ient /ri'zilyənt/ able to recoil or spring back into shape after stretching, bending, or compressing "

Slide 11

Slide 11 text

! Stretching

Slide 12

Slide 12 text

! # LOAD #

Slide 13

Slide 13 text

! Bending

Slide 14

Slide 14 text

! Feature changes

Slide 15

Slide 15 text

! % Bug ﬁxes

Slide 16

Slide 16 text

! Compressing

Slide 17

Slide 17 text

! Feature removal

Slide 18

Slide 18 text

! Internal refactoring

Slide 19

Slide 19 text

! Infrastructure changes

Slide 20

Slide 20 text

! Change happens.

Slide 21

Slide 21 text

! Who 301'd my cheese?

Slide 22

Slide 22 text

! measure change & minimize change ' handle change ( At GitHub, we so we can in order to

Slide 23

Slide 23 text

! Measuring change &

Slide 24

Slide 24 text

! Use data to inform decisions.

Slide 25

Slide 25 text

! How often is this feature used?

Slide 26

Slide 26 text

! Who is using this feature?

Slide 27

Slide 27 text

! ”If an API method is removed in the woods, and nobody is there to call it, does it still respond with 410?” ! — unknown

Slide 28

Slide 28 text

! /graph me -1hour @api.status.codes !

Slide 29

Slide 29 text

! ) Chatops

Slide 30

Slide 30 text

! Hubot *

Slide 31

Slide 31 text

! hubot.github.com

Slide 32

Slide 32 text

! /graph me -1hr @api.queries.serialized !

Slide 33

Slide 33 text

! /graph me -3hr @api.events.delivered !

Slide 34

Slide 34 text

! The Graph Store +

Slide 35

Slide 35 text

! The Graph Store + There's a graph for that™

Slide 36

Slide 36 text

! The Graph Store + There's a graph for that™ …if not, make one.

Slide 37

Slide 37 text

! Built on Graphite

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

! , Instrumentation

Slide 41

Slide 41 text

! # counter GitHub.stats.increment("the-thing.the-action") ! # gauge GitHub.stats.gauge("the-thing.the-gauge", thing.current_reading) ! # timing GitHub.stats.time("the-thing.the-operation") { the_operation } -

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

! github.com/graphite-project

Slide 45

Slide 45 text

! . Logging

Slide 46

Slide 46 text

! /splunk -3m @production "timeline.json" ! pengwynn: ! 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" request_id="AD003EA0:5BD8:4B8AE05:53BEC973" remote_address=192.168.1.1 request_method=get path_info="/timeline.json" content_length=375 user_agent="@gitlost account on twitter, Python urllib2" accept=nil language=nil status=410 at=finish elapsed=0.005 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" user=nil repo=nil route=nil zone=UTC referrer=nil requested_at="2014-07-10 17:12:19 UTC" url="https://github.com/timeline.json" controller=EventsController action=index request_id="AD003EA0:5BD8:4B8AE05:53BEC973" dbconn=write serializer=json at=start 2014-07-10T10:12:19-07:00 github-fe117-cp1-prd local7:info unicorn_rails[12813]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00"

Slide 47

Slide 47 text

Slide 48

Slide 48 text

! /splunk -3m @production "timeline.json" | stats count by status, avg(elapsed) ! pengwynn: ┌──────┬────────────┐ │status│avg(elapsed)│ ├──────┼────────────┤ │200 │0.067745 │ ├──────┼────────────┤ │410 │0.005597 │ └──────┴────────────┘

Slide 49

Slide 49 text

! key=value logging

Slide 50

Slide 50 text

! / app=github env=development enterprise=false now="2014-07-21T11:49:45-07:00" request_id=f80765a8-8aa9-4e92-b32b-01300193631a remote_address=127.0.0.1 request_method=get path_info="/" content_length=1866 user_agent=curl/7.30.0 accept=*/* language=nil status=200 at=finish elapsed=0.110

Slide 51

Slide 51 text

Slide 52

Slide 52 text

! curl https://api.github.com/ -I | grep Request-Id ❯ / ! X-GitHub-Request-Id: 4317CCE2:6434:21DCB73:53CD6239

Slide 53

Slide 53 text

! curl https://api.github.com/ -I | grep Request-Id ❯ / ! X-GitHub-Request-Id: 4317CCE2:6434:21DCB73:53CD6239 A single identifier to trace requests through logging.

Slide 54

Slide 54 text

! /splunk -3m @production request_id=f80765a8-8aa9-4e92- b32b-01300193631a !

Slide 55

Slide 55 text

! github.com/asenchi/scrolls

Slide 56

Slide 56 text

! github.com/technoweenie/grohl If Golang is your thang.

Slide 57

Slide 57 text

! Logstash, Splunk*, Loggly* *SaaS solutions, not open source

Slide 58

Slide 58 text

! Minimizing change '

Slide 59

Slide 59 text

! Isolation

Slide 60

Slide 60 text

! tree -d -L 2 github ❯ / github ├── app │ ├── api │ ├── assets │ ├── controllers │ ├── helpers │ ├── mailers │ ├── models │ ├── view_models │ └── views ├── bin ├── config │ ├── inititalizers ...

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

! Shared business logic data access logging instrumentation plumbing Independent workﬂows output

Slide 64

Slide 64 text

! Testing ❯ testrb test/integration/api/repos_test.rb Loading suite in dotcom mode ... Run options: # Running tests: Finished tests in 30.638101s, 4.7979 tests/s, 64.7886 assertions/s. 147 tests, 1985 assertions, 0 failures, 0 errors, 0 skips ruby -v: ruby 2.1.0p0-github (development) [x86_64-darwin13.0]

Slide 65

Slide 65 text

! Rack::Test

Slide 66

Slide 66 text

! brynary/rack-test github.com/

Slide 67

Slide 67 text

! # test/integration/api/root_test.rb ! test "provides hypermedia URLs to top-level resources" do get "/" data = JSON.parse(last_response.body) assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -

Slide 68

Slide 68 text

Slide 69

Slide 69 text

! # test/integration/api/root_test.rb ! test "provides hypermedia URLs to top-level resources" do data = api :get, "/" ! assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -

Slide 70

Slide 70 text

! JSON Schema

Slide 71

Slide 71 text

! json-schema.org

Slide 72

Slide 72 text

! Validating output.

Slide 73

Slide 73 text

! / # Running tests: ! [1/5] Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 = 2.06 s 1) Failure: Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 [/Users/ wynn/github/github/test/integration/api/rate_limit_test.rb:67]: {"resources"=>{"core"=>{}, "search"=>{}}, "rate"=>{}} The property '#/rate' did not contain a required property of 'limit' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'remaining' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'reset' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1#

Slide 74

Slide 74 text

Slide 75

Slide 75 text

Slide 76

Slide 76 text

! Validating input.

Slide 77

Slide 77 text

! / # update a label $ curl -ni http://api.github.dev/repos/defunkt/dotjs/labels/bug \ -X PATCH -H "Content-Type: application/json" \ -d '{ "name":[], "color":42 } ! HTTP/1.1 422 Unprocessable Entity ! { "message": "Invalid request.\n\nExpected data to be of type \"string \"; value was: [].\nExpected data to be of type \"string\"; value was: 42.", "documentation_url": "https://developer.github.com/v3/issues/labels/ #update-a-label" }

Slide 78

Slide 78 text

Slide 79

Slide 79 text

! MiniTest

Slide 80

Slide 80 text

! Custom assertions

Slide 81

Slide 81 text

! test "can create a pull request" do auth_as @forker ! data = api :post, @pub_pulls_url, {}, :input => { :title => "new pr", :body => "this is an amazing change", :base => "master", :head => "#{@forker}:ahead" }.to_json assert_equal 201, last_response.status assert_schema :v3, :pull_request, data end -

Slide 82

Slide 82 text

Slide 83

Slide 83 text

! test "can retrieve pull requests" do auth_as @owner ! data = api :get, @pub_pulls_url assert_equal 200, last_response.status ! assert_schema_for_collection \ :v3, :pull_request_simple, data end -

Slide 84

Slide 84 text

! test "can retrieve pull requests" do auth_as @owner ! data = api :get, @pub_pulls_url assert_equal 200, last_response.status ! assert_schema_for_collection \ :v3, :pull_request_simple, data end -

Slide 85

Slide 85 text

! Even more assertions

Slide 86

Slide 86 text

! test "authz list commits for pull request" do ! assert_authz :get, @priv_commits_url do |check| check.disallows_with :not_found check.authorized_users @owner, @collaborator check.unauthorized_users :anon, @rando check.minimum_scopes :repo end end -

Slide 87

Slide 87 text

Slide 88

Slide 88 text

Slide 89

Slide 89 text

Slide 90

Slide 90 text

Slide 91

Slide 91 text

Slide 92

Slide 92 text

! 5 lines = dozens of AuthZ assertions.

Slide 93

Slide 93 text

! But what about those critical code paths?

Slide 94

Slide 94 text

! What if you don’t trust your tests?

Slide 95

Slide 95 text

! Science 0

Slide 96

Slide 96 text

! github/dat-science github.com/

Slide 97

Slide 97 text

! @jbarnette

Slide 98

Slide 98 text

! @rick

Slide 99

Slide 99 text

Slide 100

Slide 100 text

Slide 101

Slide 101 text

! Safely observe alternate code paths for non-mutating methods that return a value.

Slide 102

Slide 102 text

! require "dat/science" ! class MyApp::Widget def allows?(user) experiment = Dat::Science::Experiment.new \ "widget-permissions" do |e| ! e.control { old_method } # old way e.candidate { new_method } # new way end ! experiment.run end end -

Slide 103

Slide 103 text

Slide 104

Slide 104 text

Slide 105

Slide 105 text

Slide 106

Slide 106 text

Slide 107

Slide 107 text

Slide 108

Slide 108 text

Runs candidate before control 50% of the time Measures the duration of both behaviors Compares the results of both behaviors Swallows any exceptions raised by the candidate behavior Publishes ﬁndings for tracking and reporting

Slide 109

Slide 109 text

! Fancy a DSL instead?

Slide 110

Slide 110 text

! require "dat/science" ! class MyApp::Widget def allows?(user) science "widget-permissions" do |e| e.control { model.check_user(user).valid? } e.candidate { user.can? :read, model } end end end -

Slide 111

Slide 111 text

! require "dat/science" ! class MyApp::Widget def allows?(user) science "widget-permissions" do |e| e.control { model.check_user(user).valid? } e.candidate { user.can? :read, model } end end end -

Slide 112

Slide 112 text

! Create and run.

Slide 113

Slide 113 text

! require "dat/science" ! class Blog::Post def permalink(user) science "permalink" do |e| e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -

Slide 114

Slide 114 text

Slide 115

Slide 115 text

! Ramping up experiments.

Slide 116

Slide 116 text

! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment def enabled? rand(100) < 10 # Run 10% of the time end end end -

Slide 117

Slide 117 text

! github.com/jnunemaker/ﬂipper!

Slide 118

Slide 118 text

! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment def enabled? MyApp.flipper[name].enabled? end end end -

Slide 119

Slide 119 text

! Publishing results.

Slide 120

Slide 120 text

! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment def publish MyApp.instrument "science.#{event}", payload end end end -

Slide 121

Slide 121 text

! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment def publish MyApp.instrument "science.#{event}", payload end end end -

Slide 122

Slide 122 text

No content

Slide 123

Slide 123 text

No content

Slide 124

Slide 124 text

No content

Slide 125

Slide 125 text

! Adding custom context.

Slide 126

Slide 126 text

! require "dat/science" ! class Blog::Post def permalink(user) science "permalink" do |e| e.context :post => post e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -

Slide 127

Slide 127 text

Slide 128

Slide 128 text

Mad science.

Slide 129

Slide 129 text

1 Don't try this at home

Slide 130

Slide 130 text

! module GitHub class Experiment < Dat::Science::Experiment class << self # Public: enable mad science mode: returns the candidate values by default # instead of the control. def mad_science=(val) @mad_science = val end ! # Internal: whether or not to always run the candidate instead of the # control. def candidate_instead_of_control? @mad_science end end … end -

Slide 131

Slide 131 text

Slide 132

Slide 132 text

! Analyzing results.

Slide 133

Slide 133 text

! github.com/github/dat-analysis

Slide 134

Slide 134 text

! Build an analyzer to poke on results.

Slide 135

Slide 135 text

! require 'dat/analysis' ! module MyApp class Analysis < Dat::Analysis def read Redis.rpop "dat-science.#{experiment_name}.results" end ! def count Redis.llen "dat-science.#{experiment_name}.results" end ! def cook(raw_result) return nil unless raw_result JSON.parse(raw_result) end end end -

Slide 136

Slide 136 text

Slide 137

Slide 137 text

Slide 138

Slide 138 text

Slide 139

Slide 139 text

Slide 140

Slide 140 text

Slide 141

Slide 141 text

! / irb> a = MyApp::Analysis.new('widget-permissions') => # ! ! !

Slide 142

Slide 142 text

! / irb> a = MyApp::Analysis.new('widget-permissions') => # ! irb> a.fetch => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } ! !

Slide 143

Slide 143 text

! / irb> a = MyApp::Analysis.new('widget-permissions') => # ! irb> a.fetch => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } ! irb> a.result => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } !

Slide 144

Slide 144 text

! / irb> a.result.control => {"duration"=>12.307, "exception"=>nil, "value"=>false} ! ! !

Slide 145

Slide 145 text

! / irb> a.result.control => {"duration"=>12.307, "exception"=>nil, "value"=>false} ! irb> a.result.candidate => {"duration"=>12.366999999999, "exception"=>nil, "value"=>true} !

Slide 146

Slide 146 text

! Handling change (

Slide 147

Slide 147 text

! New formats

Slide 148

Slide 148 text

! Features evolve. URLs don’t have to.

Slide 149

Slide 149 text

! / [ "[email protected]", "[email protected]" ] curl -n http://api.github.com/user/emails ❯

Slide 150

Slide 150 text

! / [ "[email protected]", "[email protected]" ] curl -n http://api.github.com/user/emails ❯

Slide 151

Slide 151 text

! / curl -n http://api.github.com/user/emails \ -H "Accept: application/vnd.github.v3" [ { "email": "[email protected]", "primary": false, "verified": true }, { "email": "[email protected]", "primary": false, "verified": true } ] ❯

Slide 152

Slide 152 text

Slide 153

Slide 153 text

Slide 154

Slide 154 text

! Media types

Slide 155

Slide 155 text

! curl https://api.github.com/users/technoweenie -I / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3

Slide 156

Slide 156 text

! curl https://api.github.com/users/technoweenie -I / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3

Slide 157

Slide 157 text

! curl https://api.github.com/users/technoweenie -I / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3

Slide 158

Slide 158 text

Slide 159

Slide 159 text

Slide 160

Slide 160 text

Slide 161

Slide 161 text

! curl https://api.github.com/users/technoweenie -I curl https://api.github.com/users/technoweenie -I \ -H "Accept: application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json curl https://api.github.com/users/technoweenie -I \ -H "Accept: application/vnd.github.beta.full+json" HTTP/1.1 200 OK X-GitHub-Media-Type: github.beta; param=full; format=json

Slide 162

Slide 162 text

Slide 163

Slide 163 text

! @jasonrudolph “Beta like you mean it.”

Slide 164

Slide 164 text

! Media types as beta code words.

Slide 165

Slide 165 text

! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type ! { "message": "Unsupported 'Accept' header: github.v3." }

Slide 166

Slide 166 text

! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type ! { "message": "Unsupported 'Accept' header: github.v3." }

Slide 167

Slide 167 text

! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type ! { "message": "Unsupported 'Accept' header: github.v3." }

Slide 168

Slide 168 text

! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type ! { "message": "Unsupported 'Accept' header: github.v3." } curl https://api.github.com/the/new/shiny \ -H "Accept: application/vnd.github.new-shiny-preview" HTTP/1.1 200 OK

Slide 169

Slide 169 text

Slide 170

Slide 170 text

Slide 171

Slide 171 text

! Client testing

Slide 172

Slide 172 text

No content

Slide 173

Slide 173 text

! octokit.github.com

Slide 174

Slide 174 text

! github.com/octokit/octokit.rb

Slide 175

Slide 175 text

! github.com/rspec/rspec

Slide 176

Slide 176 text

! describe Octokit::Client do ! describe ".rate_limit" do it "makes a response if there is no last response" do client = Octokit::Client.new VCR.use_cassette "rate_limit" do rate = client.rate_limit ! expect(rate.limit).to be_kind_of Fixnum expect(rate.remaining).to be_kind_of Fixnum end end # .rate_limit end … -

Slide 177

Slide 177 text

Slide 178

Slide 178 text

Slide 179

Slide 179 text

Slide 180

Slide 180 text

! Record live API requests to play back later. Is it live, or is it Memorex®

Slide 181

Slide 181 text

! github.com/vcr/vcr

Slide 182

Slide 182 text

! Environment smoke tests

Slide 183

Slide 183 text

! / rm -rf spec/cassettes ❯ ~/.coral/repos/octokit.rb@octokit master ❯ ~/.coral/repos/octokit.rb@octokit master*

Slide 184

Slide 184 text

! / rm -rf spec/cassettes ❯ ~/.coral/repos/octokit.rb@octokit master OCTOKIT_API_ENDPOINT="https://rails5.github.com" rspec ❯ ~/.coral/repos/octokit.rb@octokit master* ................................................. ..............F.................................. ........................FFF...................... ...................F............................. ........F........................................ ................................................. .....................F........F..................

Slide 185

Slide 185 text

! measure change & minimize change ' handle change ( At GitHub, we so we can in order to