Rethinking how we build HTTP APIs

Slide 1

Slide 1 text

github.com/fergalwalsh/pico Rethinking how we build HTTP APIs Fergal Walsh @hipolabs

Slide 2

Slide 2 text

github.com/fergalwalsh/pico Rethinking how we build HTTP APIs

Slide 3

Slide 3 text

github.com/fergalwalsh/pico Let's write an API: Pretty simple: import api result = api.hello('world') assert(result == {'message': 'hello world!'}) # api.py def hello(who): s = "hello %s!" % who return {'message': s}

Slide 4

Slide 4 text

github.com/fergalwalsh/pico Let's write a HTTP API: GET http://example.com/api/hello?who="world" {"message": "hello world!"}

Slide 5

Slide 5 text

github.com/fergalwalsh/pico Django from django.http import JsonResponse def hello(request): who = request.GET['who'] s = "hello %s!" % who return JsonResponse({'message': s}) from . import views urlpatterns = [ url(r'^api/hello$', views.hello), ]

Slide 6

Slide 6 text

github.com/fergalwalsh/pico Django Rest Framework from rest_framework.views import APIView from rest_framework.response import Response class HelloView(APIView): def get(self, request): who = request.query_params.get("who") s = "hello %s!" % who return Response({"message": s}) from . import views urlpatterns = [ url(r'^api/hello$', HelloView.as_view()), ]

Slide 7

Slide 7 text

github.com/fergalwalsh/pico Flask from flask import Flask from flask import jsonify from flask import request app = Flask(__name__) @app.route('/hello') def hello(): who = request.args['who'] s = "hello %s!" % who return jsonify(message=s)

Slide 8

Slide 8 text

github.com/fergalwalsh/pico FlaskRESTful from flask import Flask from flask import request from flask.ext.restful import Api, Resource app = Flask(__name__) api = Api(app) class HelloAPI(Resource): def get(self): who = request.args['who'] s = "hello %s!" % who return {'message': s} api.add_resource(HelloAPI, '/hello', endpoint = 'hello')

Slide 9

Slide 9 text

github.com/fergalwalsh/pico Pyramid from pyramid.config import Configurator from pyramid.view import view_config @view_config(route_name='hello', renderer='json') def hello(request): who = request.params.get('who') s = "hello %s!" % who return {'message': s} if __name__ == '__main__': config = Configurator() config.add_route('hello', '/hello') app = config.make_wsgi_app()

Slide 10

Slide 10 text

github.com/fergalwalsh/pico Tornado import tornado.ioloop import tornado.web class HelloHandler(tornado.web.RequestHandler): def get(self): who = self.get_argument('who') s = "hello %s!" % who self.write(json.dumps({'message': s})) def make_app(): return tornado.web.Application([ (r"/api/hello", HelloHandler), ])

Slide 11

Slide 11 text

github.com/fergalwalsh/pico A Common Pattern # Import boilerplate... # Import boilerplate... # App boilerplate... # URL/Routing boilerplate... def hello(request): # get args from some request object who = request.args[...] # our logic s = "hello %s!" % who return Response(...)

Slide 12

Slide 12 text

github.com/fergalwalsh/pico A Common Pattern # Import boilerplate... # Import boilerplate... # App boilerplate... # URL/Routing boilerplate... class HelloAPI(Handler) def get(self): # get args from some request object who = request.args[...] # our logic s = "hello %s!" % who return Response(...)

Slide 13

Slide 13 text

github.com/fergalwalsh/pico def hello(who): s = "hello %s!" % who return {'message': s}

Slide 14

Slide 14 text

github.com/fergalwalsh/pico Enter Pico

Slide 15

Slide 15 text

github.com/fergalwalsh/pico Pico import pico from pico import PicoApp @pico.expose() def hello(who): s = "hello %s!" % who return {'message': s} app = PicoApp() app.register_module(__name__)

Slide 16

Slide 16 text

github.com/fergalwalsh/pico Pico passes the data we need to our functions as arguments Pico serialises the data we return (as JSON) Pico determines the URL from module and function name We write framework ignorant function bodies

Slide 17

Slide 17 text

github.com/fergalwalsh/pico No framework code inside our functions

Slide 18

Slide 18 text

github.com/fergalwalsh/pico So what?

Slide 19

Slide 19 text

github.com/fergalwalsh/pico Easy interactive development: In [1]: import api In [2]: api.hello('world') Out[2]: {'message': 'Hello world!'} In [3]:

Slide 20

Slide 20 text

github.com/fergalwalsh/pico Easy testing: class TestAPI(unittest.TestCase): def test_hello(self): result = api.hello('world') self.assertEqual(result, {'message': 'hello world!'})

Slide 21

Slide 21 text

github.com/fergalwalsh/pico Easy to compose: @pico.expose() def hello(who): s = "hello %s!" % who return {'message': s} @pico.expose() def get_profile(user_id): user = get_user(user_id) name = user['name'] greeting = hello(name) return {'user': user, 'greeting': greeting}

Slide 22

Slide 22 text

github.com/fergalwalsh/pico Nice, but just another Hello World framework! Real code needs to access the request object!

Slide 23

Slide 23 text

github.com/fergalwalsh/pico List movies filtered by the user's IP address @pico.expose() def list_movies(): request = pico.get_request() ip = request['REMOTE_ADDR'] client_country = lookup_ip(ip) movies = fetch_movies(client_country) ...

Slide 24

Slide 24 text

github.com/fergalwalsh/pico Easy interactive development: In [1]: import webflicks In [2]: webflicks.list_movies() KeyError: 'REMOTE_ADDR' In [3]: import pico In [4]: pico.set_dummy_request({'REMOTE_ADDR': '86.45.123.136'}) In [5]: webflicks.list_movies() Out[5]: [{'title': 'The Life of Brian', ...}, ...]

Slide 25

Slide 25 text

github.com/fergalwalsh/pico Easy Painful interactive development: In [1]: import webflicks In [2]: webflicks.list_movies() KeyError: 'REMOTE_ADDR' In [3]: import pico In [4]: pico.set_dummy_request({'REMOTE_ADDR': '86.45.123.136'}) In [5]: webflicks.list_movies() Out[5]: [{'title': 'The Life of Brian', ...}, ...]

Slide 26

Slide 26 text

github.com/fergalwalsh/pico

Slide 27

Slide 27 text

github.com/fergalwalsh/pico Remember: No framework code inside our functions!

Slide 28

Slide 28 text

github.com/fergalwalsh/pico Hmmmmm... Can we inject data into a function? Yes! Can we have a decorator that is active only when called by Pico? Yes! Decorators are outside the function!

Slide 29

Slide 29 text

github.com/fergalwalsh/pico List movies filtered by the user's IP address from pico.decorators import request_args @pico.expose() @request_args(ip='remote_addr') def list_movies(ip): client_country = lookup_ip(ip) movies = fetch_movies(client_country) ...

Slide 30

Slide 30 text

github.com/fergalwalsh/pico Easy interactive development: In [1]: import webflicks In [2]: webflicks.list_movies('86.45.123.136') Out[2]: [{'title': 'The Life of Brian', ...}, ...] In [3]:

Slide 31

Slide 31 text

github.com/fergalwalsh/pico Easy testing: class TestMoviesList(unittest.TestCase): def test_movies_ireland(self): movies = webflicks.list_movies('86.45.123.136') self.assertEqual(movies, movies_list['ie'])

Slide 32

Slide 32 text

github.com/fergalwalsh/pico List movies starred by the current user def current_user(request): # check basic_auth if request.authorization: auth = request.authorization if not check_password(auth.username, auth.password): raise Unauthorized("Incorrect username or password!") return auth.username else: ... @pico.expose() @request_args(username=current_user) def list_starred_movies(username): movies = fetch_movies_by_user(username) ...

Slide 33

Slide 33 text

github.com/fergalwalsh/pico Allow admin users to delete a movie from the database This works but our delete function is unnecessarily coupled to a user object. @pico.expose() @request_args(user=current_user) def delete_movie(id, user): if user in admin_users: # delete the movie else: raise Unauthorized()

Slide 34

Slide 34 text

github.com/fergalwalsh/pico Allow admin users to delete a movie from the database If the protector does not raise an exception the function executes normally. If the user is not an admin a 401 Unauthorized response is returned to the client. from pico.decorators import protected def is_admin(request, wrapped, args, kwargs): user = current_user(request) if user not in admin_users: raise Unauthorized @pico.expose() @protected(is_admin) def delete_movie(id): # delete the movie

Slide 35

Slide 35 text

github.com/fergalwalsh/pico Protectors have no effect outside of the request context. We can still call our functions from anywhere else without passing around User objects. In [1]: import webflicks In [2]: webflicks.delete_movie(42) Out[2]:

Slide 36

Slide 36 text

github.com/fergalwalsh/pico How do these decorators work? A simple UI at the cost of some internal complexity

Slide 37

Slide 37 text

github.com/fergalwalsh/pico

Slide 38

Slide 38 text

github.com/fergalwalsh/pico Routing import api api.hello(who='world') http://example.com/{module}/{function}?{kwargs} GET http://example.com/api/hello?who="world" POST http://example.com/api/hello {"who": "world"}

Slide 39

Slide 39 text

github.com/fergalwalsh/pico Routing import api.users api.users.profile(id=1) http://example.com/{package}/{module}/{function}?{kwargs} GET http://example.com/api/users/profile?id=1 POST http://example.com/api/users/profile {"id": 1}

Slide 40

Slide 40 text

github.com/fergalwalsh/pico 1 URL == 1 Function Quickly find the relevant code No mental overhead No decisions about URL naming

Slide 41

Slide 41 text

github.com/fergalwalsh/pico What about URLs with multiple HTTP methods? GET http://example.com/api/users POST http://example.com/api/users GET http://example.com/api/users/1 PUT http://example.com/api/users/1 DELETE http://example.com/api/users/1

Slide 42

Slide 42 text

github.com/fergalwalsh/pico Pico purposely does not support multiple functions per URL 1 URL == 1 Function This keeps life simple GET http://example.com/api/list_users POST http://example.com/api/add_user GET http://example.com/api/get_user?id=1 POST http://example.com/api/update_user?id=1 POST http://example.com/api/delete_user?id=1

Slide 43

Slide 43 text

github.com/fergalwalsh/pico But that's not RESTful!

Slide 44

Slide 44 text

github.com/fergalwalsh/pico Pico APIs are not RESTful!

Slide 45

Slide 45 text

github.com/fergalwalsh/pico HTTP APIs do not have to be RESTful! By not trying to be RESTful we relieve ourselves of a huge burden (and endless debate about correctness). Pico APIs are pragmatic, optimising for ease of use and ease of development over theoretical correctness.

Slide 46

Slide 46 text

github.com/fergalwalsh/pico Pico's Pragmatism Simple handler functions Single widely used response format (JSON) Automatic JSON serialisation beyond normal json with pico.pragmaticjson Sensible simple URLs Self describing Supports passing arguments as query parameters, JSON and FormEncoded. Works with any HTTP client Includes a Python and Javascript client Easy to override various parts where necessary Deployed like any other WSGI application

Slide 47

Slide 47 text

github.com/fergalwalsh/pico Pico APIs are self describing """ Provides functions for basic greetings. """ import pico from pico.decorators import require_method @pico.expose() def hello(name='World'): """ Returns a hello greeting message """ return {'message': 'Hello %s!' % name} @pico.expose() @require_method('post') def goodbye(): """ Returns a goodbye message """ return {'message': 'Goodbye!'}

Slide 48

Slide 48 text

github.com/fergalwalsh/pico Pico APIs are self describing GET https://example.com/api { "url": "https://example.com/api", "doc": "Provides functions for basic greetings.", "functions": [ { "name": "hello", "url": "https://example.com/api/hello", "doc": "Returns a hello greeting message", "args": [{"name": "name", "default": "World"}] }, { "name": "goodbye", "url": "https://example.com/api/goodbye", "doc": "Returns a goodbye message", "args": [], "method": "post"

Slide 49

Slide 49 text

github.com/fergalwalsh/pico Python Client import pico.client api = pico.client.load('https://example.com/api') result = api.hello('everyone')

Slide 50

Slide 50 text

github.com/fergalwalsh/pico Python Client In [1]: import pico.client In [2]: api = pico.client.load('https://example.com/api') In [3]: api. api.goodbye api.hello In [3]: api.hello? Signature: api.hello(name='World', _timeout=None, _headers={}) Docstring: Returns a hello greeting message File: Dynamically generated function. No source code available. Type: function In [4]: api.hello('everyone') Out[4]: {'message': 'Hello everyone!'}

Slide 51

Slide 51 text

github.com/fergalwalsh/pico Javascript Client Pico Example

var api = pico.importModule('api') api.hello('everyone') .then(function(response){ document.getElementById('message').innerHTML = response.message; });

Slide 52

Slide 52 text

github.com/fergalwalsh/pico Pico's Origin Story Originally developed in an academic research environment. Was originally an app in a Django project developed to ease interacting with Python based analysis tools from Javascript based visualisation tools. Optimised for quick iterative prototyping. Main users were researchers putting a simple api in front of Numpy/ML/GIS tools, NOT web developers. Later extracted into standalone framework. Intended to be a 'smaller than micro' framework, hence Pico. Now used for a variety of production services, from singlemodule 'microservices' to multimodule projects with 100+ endpoints.

Slide 53

Slide 53 text

github.com/fergalwalsh/pico Mistakes & Lessons Learned

Slide 54

Slide 54 text

github.com/fergalwalsh/pico 1) Too much magic import pico def hello(who): s = "hello %s!" % who return {'message': s} Literally add one line of code (import pico) to your Python module to turn it into a web service — README Why do I import pico but not use it? How do I know which functions are exposed? — set(pico_users) - [me] F401'pico' imported but unused — flake8

Slide 55

Slide 55 text

github.com/fergalwalsh/pico 2) Trying too hard to be a single file framework Everything, including the Javascript client, was in one .py file On PyPi, Nobody knows you're a Single File Framework — Nobody

Slide 56

Slide 56 text

github.com/fergalwalsh/pico 3) Trying too hard to have no dependencies Implemented from scratch Request, Response, HTTP Errors, Static file handling, etc Pico 2.0 was rebuilt on top of Werkzeug from werkzeug.wrappers import Request, Response from werkzeug.exceptions import * from werkzeug.serving import run_simple from werkzeug.wsgi import SharedDataMiddleware

Slide 57

Slide 57 text

github.com/fergalwalsh/pico 4) Tight coupling with the app instance Deceptively simple... app = PicoApp() @app.expose() def foo(arg): pass

Slide 58

Slide 58 text

github.com/fergalwalsh/pico 4) Tight coupling with the app instance Became messy and brittle with multi module apps... # users.py from app import app @app.expose() def foo(arg): pass # app.py app = PicoApp() import users import admin

Slide 59

Slide 59 text

github.com/fergalwalsh/pico Using @pico.expose and explicitly registering modules is much cleaner: # users.py @pico.expose() def foo(arg): pass # app.py app = PicoApp() app.register_module('users') app.register_module('admin') app.register_module('posts')

Slide 60

Slide 60 text

github.com/fergalwalsh/pico Conclusion Writing HTTP APIs can be as simple as writing basic Python modules Frameworks should help us write clean testable & maintainable code Frameworks should aim to minimise mental overhead, not increase it Minimalistic frameworks can be powerful pip install pico

Slide 61

Slide 61 text

github.com/fergalwalsh/pico Learn More github.com/fergalwalsh/pico pico.readthedocs.org pip install pico

Slide 62

Slide 62 text

github.com/fergalwalsh/pico Thanks: The giants whose shoulders' Pico stands upon: Werkzeug Wrapt Requests Python Pico development is kindly supported by