API Driven Development (Data Science Edition)

Slide 1

Slide 1 text

API Driven Development Kenneth Reitz How I Build Things and Why.

Slide 2

Slide 2 text

Hi.

Slide 3

Slide 3 text

@kennethreitz

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Open Source

Slide 6

Slide 6 text

Requests HTTP for Humans

Slide 7

Slide 7 text

Httpbin.org $ curl http://httpbin.org/get?test=1 { "url": "http://httpbin.org/get", "headers": { "Content-Length": "", "Connection": "keep-alive", "Accept": "*/*", "User-Agent": "curl/7.21.4 ...", "Host": "httpbin.org", "Content-Type": "" }, "args": { “test”: “1” }, "origin": "67.163.102.42" }

Slide 8

Slide 8 text

Et Cetera • Legit: Git Workflow for Humans • Envoy: Subprocess for Humans • Tablib: Tabular Data for Humans • Clint: CLI App Toolkit • Autoenv: Magic Shell Environments • OSX-GCC-Installer: Provokes Lawyers 275+ More

Slide 9

Slide 9 text

I am not a data scientist. I’d like to be, but the APIs are painful.

Slide 10

Slide 10 text

What do we have in common?

Slide 11

Slide 11 text

We’re Makers. We craft experiences & interfaces.

Slide 12

Slide 12 text

Developers! Developers, Developers, Developers.

Slide 13

Slide 13 text

— Steve Jobs, 1983 People are going to be spending two or three hours a day with these machines — more than they spend with a car.

Slide 14

Slide 14 text

— Steve Jobs, 1983 Software design must be given at least as much consideration as we give automobiles today — if not a lot more.

Slide 15

Slide 15 text

That worked.

Slide 16

Slide 16 text

Beautiful Interfaces. • Industrial Design • Web Interfaces • iOS, Android, Mobile Apps • Desktop Clients & Applications Today, beautiful applications abound.

Slide 17

Slide 17 text

Hackers are the real Makers.

Slide 18

Slide 18 text

Developers spend 8+ hours a day with APIs. Why are they treated differently?

Slide 19

Slide 19 text

Web Application Management Tools Supporting Services Tools & Utilities Web Process Worker Process Scheduled Tasks Deferred Tasks API Service CRUD Admin Data Persistence User Interface Authentication

Slide 20

Slide 20 text

API Service End Users API Service Internal API Service Developers ( ) Data Persistence Message Queue Workers

Slide 21

Slide 21 text

Everything is a remix*. * APIs Rule Everything Around Us.

Slide 22

Slide 22 text

How?

Slide 23

Slide 23 text

Step I: Have an Issue.

Slide 24

Slide 24 text

A Real, Tangible Problem. You can't solve a problem properly if you don't experience it firsthand.

Slide 25

Slide 25 text

Example: GitHub • GitHub wasn't built for the developer community at large. • Resonated with millions of developers. • They themselves happen to be developers. Over two million people collaborating.

Slide 26

Slide 26 text

Other’s Success • Gumroad, built for the founder. • 37 Signals product, build for the team. • Ruby on Rails, by Rubyists for Rubyists.

Slide 27

Slide 27 text

Optimization • Feature driven development? • Profit driven development? • Growth driven development? • Problem driven development. What drives your decisions?

Slide 28

Slide 28 text

pra•gmat•ic |pragˈmatik|, adj: Dealing with things sensibly and realistically in a way that is based on practical rather than theoretical considerations

Slide 29

Slide 29 text

We know Ruby... require 'net/http' require 'uri' uri = URI.parse('https://api.github.com/user') http = Net::HTTP.new(uri.host, uri.port) http.use_ssl = true req = Net::HTTP::Get.new(uri.request_uri) req.basic_auth('username', 'password') r = http.request(req) puts r

Slide 30

Slide 30 text

Python’s net/http? http/url/lib/2

Slide 31

Slide 31 text

Several hours later...

Slide 32

Slide 32 text

import urllib2 gh_url = 'https://api.github.com/user' req = urllib2.Request(gh_url) password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm() password_manager.add_password(None, gh_url, 'user', 'pass') auth_manager = urllib2.HTTPBasicAuthHandler(password_manager) opener = urllib2.build_opener(auth_manager) urllib2.install_opener(opener) handler = urllib2.urlopen(req) print handler.read()

Slide 33

Slide 33 text

import re class HTTPForcedBasicAuthHandler(HTTPBasicAuthHandler): auth_header = 'Authorization' rx = re.compile('(?:.*,)*[ \t]*([^ \t]+)[ \t]+' 'realm=(["\'])(.*?)\\2', re.I) def __init__(self, *args, **kwargs): HTTPBasicAuthHandler.__init__(self, *args, **kwargs) def http_error_401(self, req, fp, code, msg, headers): url = req.get_full_url() response = self._http_error_auth_reqed( 'www-authenticate', url, req, headers) self.reset_retry_count() return response http_error_404 = http_error_401

Slide 34

Slide 34 text

Admit it. You’d leave and never come back.

Slide 35

Slide 35 text

This is a serious problem. HTTP should be as simple as a print statement.

Slide 36

Slide 36 text

APIs For Humans

Slide 37

Slide 37 text

Let’s Break it Down. • A small set of methods with consistent parameters. • HEAD, GET, POST, PUSH, PUT, PATCH, DELETE, &c. • They all accept Headers, URL Parameters, Body/Form Data. What is HTTP at its core?

Slide 38

Slide 38 text

Enter Requests.

Slide 39

Slide 39 text

HTTP for Humans.

Slide 40

Slide 40 text

import requests url = 'https://api.github.com/user' auth = ('username', 'password') r = requests.get(url, auth=auth) print r.content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

Achievement Unlocked! • A small set of methods with consistent parameters. • HEAD, GET, POST, PUSH, PUT, PATCH, DELETE, &c. • They all accept Headers, URL Parameters, Body/Form Data.

Slide 43

Slide 43 text

Requests Success • Python is a language built for Humans. • Why should HTTP be non-trivial? • I explored and discovered what I really needed, and built it. • I had a real problem that I solved for myself.

Slide 44

Slide 44 text

Requests Success • At first, Requests was far from powerful. • But, it deeply resonated with people. • Features grew over time, but the API was never compromised. • Quite popular (48,000,000 downloads).

Slide 45

Slide 45 text

Developers spend 8+ hours a day with APIs. Build for yourself—a developer.

Slide 46

Slide 46 text

Step II: Respond.

Slide 47

Slide 47 text

Write the README.

Slide 48

Slide 48 text

• Before any code is written, write the README — show some examples. • Write some code with the theoretical code that you’ve documented.

Slide 49

Slide 49 text

Achievement Unlocked! • Instead of engineering something to get the job done, you interact with the problem itself and build an interface that reacts to it. • You discover it. You respond to it.

Slide 50

Slide 50 text

• Great sculptures aren’t engineered or manufactured—they’re discovered. • The sculptor studies and listens to the marble. He identifies with it. • Then, he responds. • Setting free something hidden that inside all along. Sculptures, Etc.

Slide 51

Slide 51 text

• It’s not about a design that will “work” on a phone, tablet, and desktop. • It’s about making something that identifies itself enough to respond to the environment it’s placed in. • Free of arbitrary constraints. Responsive Design

Slide 52

Slide 52 text

Readme-Driven Development? Responsive API Design.

Slide 53

Slide 53 text

Step III: Build.

Slide 54

Slide 54 text

• Once you discover the API: build it. • Write all the code necessary to make exactly what you documented happen. • Complex code? Layer your API. • “Porcelain” layer is documented. Responsive Design

Slide 55

Slide 55 text

The API is all that matters. Everything else is secondary.

Slide 56

Slide 56 text

Do unto others as you would have them do to you? Build tools for others that you want to be built for you.

Slide 57

Slide 57 text

Pro Tips™

Slide 58

Slide 58 text

CONSTRAINTS FOSTER CREATIVITY

Slide 59

Slide 59 text

Open Source All The Things!

Slide 60

Slide 60 text

Build for Open Source • Components become concise & decoupled. • Concerns separate themselves. • Best practices emerge (e.g. no creds in code). • Documentation and tests become crucial. • Code can be released at any time.

Slide 61

Slide 61 text

• Anaconda: Python distribution for humans and scientists (“it just works”) • Heroku: turnkey machine resources. • Conda Buildpack for Heroku. Recommended Tools