Pico: Rethinking how we build HTTP APIs

Talk delivered at PyConIE 2017

Transcript

1. github.com/fergalwalsh/pico Rethinking how we build HTTP APIs Fergal Walsh @hipolabs
2. github.com/fergalwalsh/pico Rethinking how we build HTTP APIs
3. github.com/fergalwalsh/pico Let's write an API: Pretty simple: import api result = api.hello('world') assert(result == {'message': 'hello world!'}) # api.py def hello(who): s = "hello %s!" % who return {'message': s}
4. github.com/fergalwalsh/pico Let's write a HTTP API: GET http://example.com/api/hello?who="world" {"message": "hello world!"}
5. github.com/fergalwalsh/pico Django from django.http import JsonResponse def hello(request): who = request.GET['who'] s = "hello %s!" % who return JsonResponse({'message': s}) from . import views urlpatterns = [ url(r'^api/hello$', views.hello), ]
6. github.com/fergalwalsh/pico Django Rest Framework from rest_framework.views import APIView from rest_framework.response import Response class HelloView(APIView): def get(self, request): who = request.query_params.get("who") s = "hello %s!" % who return Response({"message": s}) from . import views urlpatterns = [ url(r'^api/hello$', HelloView.as_view()), ]
7. github.com/fergalwalsh/pico Flask from flask import Flask from flask import jsonify from flask import request app = Flask(__name__) @app.route('/hello') def hello(): who = request.args['who'] s = "hello %s!" % who return jsonify(message=s)
8. github.com/fergalwalsh/pico Flask­RESTful from flask import Flask from flask import request from flask.ext.restful import Api, Resource app = Flask(__name__) api = Api(app) class HelloAPI(Resource): def get(self): who = request.args['who'] s = "hello %s!" % who return {'message': s} api.add_resource(HelloAPI, '/hello', endpoint = 'hello')
9. github.com/fergalwalsh/pico Pyramid from pyramid.config import Configurator from pyramid.view import view_config @view_config(route_name='hello', renderer='json') def hello(request): who = request.params.get('who') s = "hello %s!" % who return {'message': s} if __name__ == '__main__': config = Configurator() config.add_route('hello', '/hello') app = config.make_wsgi_app()
10. github.com/fergalwalsh/pico Tornado import tornado.ioloop import tornado.web class HelloHandler(tornado.web.RequestHandler): def get(self): who = self.get_argument('who') s = "hello %s!" % who self.write(json.dumps({'message': s})) def make_app(): return tornado.web.Application([ (r"/api/hello", HelloHandler), ])
11. github.com/fergalwalsh/pico A Common Pattern # Import boilerplate... # Import boilerplate... # App boilerplate... # URL/Routing boilerplate... def hello(request): # get args from some request object who = request.args[...] # our logic s = "hello %s!" % who return Response(...)
12. github.com/fergalwalsh/pico A Common Pattern # Import boilerplate... # Import boilerplate... # App boilerplate... # URL/Routing boilerplate... class HelloAPI(Handler) def get(self): # get args from some request object who = request.args[...] # our logic s = "hello %s!" % who return Response(...)
13. github.com/fergalwalsh/pico def hello(who): s = "hello %s!" % who return {'message': s}
14. github.com/fergalwalsh/pico import pico
15. github.com/fergalwalsh/pico Pico import pico from pico import PicoApp @pico.expose() def hello(who): s = "hello %s!" % who return {'message': s} app = PicoApp() app.register_module(__name__)
16. github.com/fergalwalsh/pico Pico passes the data we need to our functions as arguments Pico serialises the data we return (as JSON) Pico determines the URL from module and function name We write framework ignorant function bodies
17. github.com/fergalwalsh/pico No framework code inside our functions
18. github.com/fergalwalsh/pico So what?
19. github.com/fergalwalsh/pico Easy interactive development: In [1]: import api In [2]: api.hello('world') Out[2]: {'message': 'Hello world!'} In [3]:
20. github.com/fergalwalsh/pico Easy testing: class TestAPI(unittest.TestCase): def test_hello(self): result = api.hello('world') self.assertEqual(result, {'message': 'hello world!'})
21. github.com/fergalwalsh/pico Easy to compose: @pico.expose() def hello(who): s = "hello %s!" % who return {'message': s} @pico.expose() def get_profile(user_id): user = get_user(user_id) name = user['name'] greeting = hello(name) return {'user': user, 'greeting': greeting}
22. github.com/fergalwalsh/pico Nice, but just another Hello World framework! Real code needs to access the request object!
23. github.com/fergalwalsh/pico List movies filtered by the user's IP address @pico.expose() def list_movies(): request = pico.get_request() ip = request['REMOTE_ADDR'] client_country = lookup_ip(ip) movies = fetch_movies(client_country) ...
24. github.com/fergalwalsh/pico Easy interactive development: In [1]: import webflicks In [2]: webflicks.list_movies() KeyError: 'REMOTE_ADDR' In [3]: import pico In [4]: pico.set_dummy_request({'REMOTE_ADDR': '86.45.123.136'}) In [5]: webflicks.list_movies() Out[5]: [{'title': 'The Life of Brian', ...}, ...]
25. github.com/fergalwalsh/pico Easy Painful interactive development: In [1]: import webflicks In [2]: webflicks.list_movies() KeyError: 'REMOTE_ADDR' In [3]: import pico In [4]: pico.set_dummy_request({'REMOTE_ADDR': '86.45.123.136'}) In [5]: webflicks.list_movies() Out[5]: [{'title': 'The Life of Brian', ...}, ...]
26. github.com/fergalwalsh/pico
27. github.com/fergalwalsh/pico Remember: No framework code inside our functions!
28. github.com/fergalwalsh/pico Hmmmmm... Can we inject data into a function? Yes! Can we have a decorator that is active only when called by Pico? Yes! Decorators are outside the function!
29. github.com/fergalwalsh/pico List movies filtered by the user's IP address from pico.decorators import request_args @pico.expose() @request_args(ip='remote_addr') def list_movies(ip): client_country = lookup_ip(ip) movies = fetch_movies(client_country) ...
30. github.com/fergalwalsh/pico Easy interactive development: In [1]: import webflicks In [2]: webflicks.list_movies('86.45.123.136') Out[2]: [{'title': 'The Life of Brian', ...}, ...] In [3]:
31. github.com/fergalwalsh/pico Easy testing: class TestMoviesList(unittest.TestCase): def test_movies_ireland(self): movies = webflicks.list_movies('86.45.123.136') self.assertEqual(movies, movies_list['ie'])
32. github.com/fergalwalsh/pico List movies starred by the current user def current_user(request): # check basic_auth if request.authorization: auth = request.authorization if not check_password(auth.username, auth.password): raise Unauthorized("Incorrect username or password!") return auth.username else: ... @pico.expose() @request_args(username=current_user) def list_starred_movies(username): movies = fetch_movies_by_user(username) ...
33. github.com/fergalwalsh/pico Allow admin users to delete a movie from the database This works but our delete function is unnecessarily coupled to a user object. @pico.expose() @request_args(user=current_user) def delete_movie(id, user): if user in admin_users: # delete the movie else: raise Forbidden()
34. github.com/fergalwalsh/pico Allow admin users to delete a movie from the database If the protector does not raise an exception the function executes normally. If the user is not an admin a 403 Forbidden response is returned to the client. from pico.decorators import protected def is_admin(request, wrapped, args, kwargs): user = current_user(request) if user not in admin_users: raise Forbidden() @pico.expose() @protected(is_admin) def delete_movie(id): # delete the movie
35. github.com/fergalwalsh/pico Protectors have no effect outside of the request context. We can still call our functions from anywhere else without passing around User objects. In [1]: import webflicks In [2]: webflicks.delete_movie(42) Out[2]:
36. github.com/fergalwalsh/pico How do these decorators work? A simple UI at the cost of some internal complexity
37. github.com/fergalwalsh/pico
38. github.com/fergalwalsh/pico Routing import api api.hello(who='world') http://example.com/{module}/{function}?{kwargs} GET http://example.com/api/hello?who="world" POST http://example.com/api/hello {"who": "world"}
39. github.com/fergalwalsh/pico Routing import api.users api.users.profile(id=1) http://example.com/{package}/{module}/{function}?{kwargs} GET http://example.com/api/users/profile?id=1 POST http://example.com/api/users/profile {"id": 1}
40. github.com/fergalwalsh/pico 1 URL == 1 Function Quickly find the relevant code No mental overhead No decisions about URL naming
41. github.com/fergalwalsh/pico What about URLs with multiple HTTP methods? GET http://example.com/api/users POST http://example.com/api/users GET http://example.com/api/users/1 PUT http://example.com/api/users/1 DELETE http://example.com/api/users/1
42. github.com/fergalwalsh/pico Pico purposely does not support multiple functions per URL 1 URL == 1 Function This keeps life simple GET http://example.com/api/list_users POST http://example.com/api/add_user GET http://example.com/api/get_user?id=1 POST http://example.com/api/update_user?id=1 POST http://example.com/api/delete_user?id=1
43. github.com/fergalwalsh/pico But that's not RESTful!
44. github.com/fergalwalsh/pico Pico APIs are not RESTful!
45. github.com/fergalwalsh/pico HTTP APIs do not have to be RESTful! By not trying to be RESTful we relieve ourselves of a huge burden (and endless debate about correctness). Pico APIs are pragmatic, optimising for ease of use and ease of development over theoretical correctness.
46. github.com/fergalwalsh/pico Pico's Pragmatism Simple handler functions Single widely used response format (JSON) Automatic JSON serialisation beyond normal json with pico.pragmaticjson Sensible simple URLs Self describing Supports passing arguments as query parameters, JSON and FormEncoded. Works with any HTTP client Includes a Python and Javascript client Easy to override various parts where necessary Deployed like any other WSGI application
47. github.com/fergalwalsh/pico Pico APIs are self describing """ Provides functions for basic greetings. """ import pico from pico.decorators import require_method @pico.expose() def hello(name='World'): """ Returns a hello greeting message """ return {'message': 'Hello %s!' % name} @pico.expose() @require_method('post') def goodbye(): """ Returns a goodbye message """ return {'message': 'Goodbye!'}
48. github.com/fergalwalsh/pico Pico APIs are self describing GET https://example.com/api { "url": "https://example.com/api", "doc": "Provides functions for basic greetings.", "functions": [ { "name": "hello", "url": "https://example.com/api/hello", "doc": "Returns a hello greeting message", "args": [{"name": "name", "default": "World"}] }, { "name": "goodbye", "url": "https://example.com/api/goodbye", "doc": "Returns a goodbye message", "args": [], "method": "post"
49. github.com/fergalwalsh/pico Python Client import pico.client api = pico.client.load('https://example.com/api') result = api.hello('everyone')
50. github.com/fergalwalsh/pico Python Client In [1]: import pico.client In [2]: api = pico.client.load('https://example.com/api') In [3]: api. api.goodbye api.hello In [3]: api.hello? Signature: api.hello(name='World', _timeout=None, _headers={}) Docstring: Returns a hello greeting message File: Dynamically generated function. No source code available. Type: function In [4]: api.hello('everyone') Out[4]: {'message': 'Hello everyone!'}
51. github.com/fergalwalsh/pico Python Client In [1]: import pico.client In [2]: api = pico.client.load('https://example.com/api') In [3]: api. api.goodbye api.hello In [3]: api.hello? Signature: api.hello(name='World', _timeout=None, _headers={}) Docstring: Returns a hello greeting message File: Dynamically generated function. No source code available. Type: function In [4]: api.hello('everyone') Out[4]: {'message': 'Hello everyone!'}
52. github.com/fergalwalsh/pico Javascript Client <html> <head> <title>Pico Example</title> <!-- The pico client is automatically available at /pico.js --> <script src="https://example.com/pico.js"></script> <!-- Load our api module --> <script src="https://example.com/api.js"></script> </head> <body> <p id="message"></p> <script> var api = pico.importModule('api') api.hello('everyone') .then(function(response){ document.getElementById('message').innerHTML = response.message; });
53. github.com/fergalwalsh/pico Pico's Origin Story Originally developed in an academic research environment. Was originally an app in a Django project developed to ease interacting with Python based analysis tools from Javascript based visualisation tools. Optimised for quick iterative prototyping. Main users were researchers putting a simple api in front of Numpy/ML/GIS tools, NOT web developers. Later extracted into standalone framework. Intended to be a 'smaller than micro' framework, hence Pico. Now used for a variety of production services, from single­module 'microservices' to multi­ module projects with 100+ endpoints.
54. github.com/fergalwalsh/pico Mistakes & Lessons Learned
55. github.com/fergalwalsh/pico 1) Too much magic import pico def hello(who): s = "hello %s!" % who return {'message': s} Literally add one line of code (import pico) to your Python module to turn it into a web service — README Why do I import pico but not use it? How do I know which functions are exposed? — set(pico_users) - {me} F401'pico' imported but unused — flake8
56. github.com/fergalwalsh/pico 2) Trying too hard to be a single file framework Everything, including the Javascript client, was in one .py file On PyPi, Nobody knows you're a Single File Framework — Nobody
57. github.com/fergalwalsh/pico 3) Trying too hard to have no dependencies Implemented from scratch Request, Response, HTTP Errors, Static file handling, etc Pico 2.0 was rebuilt on top of Werkzeug from werkzeug.wrappers import Request, Response from werkzeug.exceptions import * from werkzeug.serving import run_simple from werkzeug.wsgi import SharedDataMiddleware
58. github.com/fergalwalsh/pico 4) Tight coupling with the app instance Deceptively simple... app = PicoApp() @app.expose() def foo(arg): pass
59. github.com/fergalwalsh/pico 4) Tight coupling with the app instance Became messy and brittle with multi module apps... # users.py from app import app @app.expose() def foo(arg): pass # app.py app = PicoApp() import users import admin
60. github.com/fergalwalsh/pico Using @pico.expose and explicitly registering modules is much cleaner: # users.py @pico.expose() def foo(arg): pass # app.py app = PicoApp() app.register_module('users') app.register_module('admin') app.register_module('posts')
61. github.com/fergalwalsh/pico Conclusion Writing HTTP APIs can be as simple as writing basic Python modules Frameworks should help us write clean testable & maintainable code Frameworks should aim to minimise mental overhead, not increase it Minimalistic frameworks can be powerful pip install pico
62. github.com/fergalwalsh/pico Learn More github.com/fergalwalsh/pico pico.readthedocs.org pip install pico
63. github.com/fergalwalsh/pico Thanks: The giants whose shoulders' Pico stands upon: Werkzeug Wrapt Requests Python Pico development is kindly supported by