RESTful APIs with Tastypie

Slide 1

Slide 1 text

Slide 2

Slide 2 text

...or...

Slide 3

Slide 3 text

How I learned to stop worrying and love the JSON

Slide 4

Slide 4 text

Who am I? • Daniel Lindsley

Slide 5

Slide 5 text

Who am I? • Daniel Lindsley • Consulting & OSS as Toast Driven

Slide 6

Slide 6 text

Who am I? • Daniel Lindsley • Consulting & OSS as Toast Driven • Primary author of Tastypie

Slide 7

Slide 7 text

What is Tastypie? • A REST framework for Django

Slide 8

Slide 8 text

What is Tastypie? • A REST framework for Django • Designed for extension

Slide 9

Slide 9 text

What is Tastypie? • A REST framework for Django • Designed for extension • Supports both Model & non-Model data

Slide 10

Slide 10 text

What is Tastypie? • A REST framework for Django • Designed for extension • Supports both Model & non-Model data • http://tastypieapi.org/

Slide 11

Slide 11 text

Philosophy

Slide 12

Slide 12 text

Make good use of HTTP

Slide 13

Slide 13 text

Make good use of HTTP Try to be “of the internet” & use the REST methods/status codes properly

Slide 14

Slide 14 text

Graceful Degradation

Slide 15

Slide 15 text

Graceful Degradation Try to keep backward compatibility & give users a gradual upgrade path

Slide 16

Slide 16 text

Flexible serialization

Slide 17

Slide 17 text

Flexible serialization Not everyone wants JSON

Slide 18

Slide 18 text

Wait. Scratch that.

Slide 19

Slide 19 text

Flexible serialization Not everyone wants JSON

Slide 20

Slide 20 text

Flexible EVERYTHING

Slide 21

Slide 21 text

Flexible EVERYTHING Customizability is a core feature

Slide 22

Slide 22 text

Data can round-trip

Slide 23

Slide 23 text

Data can round-trip Anything you can GET, you should be able to POST/PUT

Slide 24

Slide 24 text

Reasonable defaults

Slide 25

Slide 25 text

Reasonable defaults But easy to extend

Slide 26

Slide 26 text

URIs everywhere!

Slide 27

Slide 27 text

URIs everywhere! Make HATEOAS a reality

Slide 28

Slide 28 text

HATEOAS?

Slide 29

Slide 29 text

Gesundheit!

Slide 30

Slide 30 text

HATEOAS • “Hypermedia As The Engine Of Application State” • Basically, the user shouldn’t have to know anything in advance • All about explore-ability • Deep linking • http://en.wikipedia.org/wiki/HATEOAS

Slide 31

Slide 31 text

So what about Tastypie?

Slide 32

Slide 32 text

Pie?

Slide 33

Slide 33 text

Pie? Pie? Pie? Pie? Pie? Pie? Pie? Pie?

Slide 34

Slide 34 text

He can’t haz.

Slide 35

Slide 35 text

He can’t haz. But you can!

Slide 36

Slide 36 text

Tastypie • Builds on top of Django & plays nicely with other apps • Full GET/POST/PUT/DELETE/PATCH • Any data source (not just Models) • Designed to be extended

Slide 37

Slide 37 text

Tastypie (cont.) • Supports a variety of serialization formats • JSON • XML • YAML • bplist • Easy to add more

Slide 38

Slide 38 text

Tastypie (cont.) • HATEOAS by default (you’ll see soon) • Lots of hooks for customization • Well-tested • Decently documented

Slide 39

Slide 39 text

The Setup

Slide 40

Slide 40 text

Installation pip install django-tastypie

Slide 41

Slide 41 text

Installation (cont.) INSTALLED_APPS += [‘tastypie’]

Slide 42

Slide 42 text

Installation (cont.) $ ./manage.py syncdb

Slide 43

Slide 43 text

Done.

Slide 44

Slide 44 text

Let’s add an API for django.contrib.auth

Slide 45

Slide 45 text

High-Level 1. Code goes in our apps, not Django

Slide 46

Slide 46 text

High-Level 1. Code goes in our apps, not Django 2. Deﬁne the resource for User

Slide 47

Slide 47 text

High-Level 1. Code goes in our apps, not Django 2. Deﬁne the resource for User 3. Hook up that resource in your URLconf

Slide 48

Slide 48 text

1. Don’t fork Django! No, seriously. You don’t need to. Done.

Slide 49

Slide 49 text

2a. Setup # Assuming we’re in your project directory... $ cd # Substitute your app_name here. $ mkdir api $ touch api/__init__.py $ touch api/resources.py # Done!

Slide 50

Slide 50 text

2b. User Resource from django.contrib.auth.models import User from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: queryset = User.objects.all()

Slide 51

Slide 51 text

3. URLconf # In your ``ROOT_URLCONF``... # Stuff then... from tastypie.api import Api from .api.resources import UserResource v1_api = Api() v1_api.register(UserResource()) urlpatterns = patterns(‘’, (r’^api/’, include(v1_api.urls), # Then the usual... )

Slide 52

Slide 52 text

HTTP 201

Slide 53

Slide 53 text

Hit your new API. Curl: http://localhost:8000/api/v1/ Browser: http://localhost:8000/api/v1/?format=json

Slide 54

Slide 54 text

Demo-time.

Slide 55

Slide 55 text

What’s there? • /api/v1/ - A list of all available resources • /api/v1/user/ - A list of all users • /api/v1/user/2/ - A speciﬁc user • /api/v1/user/schema/ - A deﬁnition of what an individual user consists of • /api/v1/user/multiple/1;4;5/ - Get those three users as one request

Slide 56

Slide 56 text

What’s there? (cont.) • All serialization formats available* • curl -H “Accept: application/xml” http://localhost:8000/api/v1/ user/ • http://localhost:8000/api/v1/user/2/? format=yaml * Provided lxml, PyYAML, biplist are installed.

Slide 57

Slide 57 text

What’s there? (cont.) • Serialization format negotiated by either Accepts header or the ”? format=json” GET param • Pagination by default • Everyone has full read-only GET access

Slide 58

Slide 58 text

What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser

Slide 59

Slide 59 text

What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser • Ability to ﬁlter

Slide 60

Slide 60 text

What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser • Ability to ﬁlter • Authentication / Authorization

Slide 61

Slide 61 text

What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser • Ability to ﬁlter • Authentication / Authorization • Caching (disabled by default)

Slide 62

Slide 62 text

What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser • Ability to ﬁlter • Authentication / Authorization • Caching (disabled by default) • Throttling (disabled by default)

Slide 63

Slide 63 text

Fix data leaks from django.contrib.auth.models import User from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’]

Slide 64

Slide 64 text

Authentication “Are you a recognized user?”

Slide 65

Slide 65 text

Add BASIC Auth from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: # What was there before... authentication = BasicAuthentication()

Slide 66

Slide 66 text

Add filtering from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication from tastypie.resources import ModelResource, ALL class UserResource(ModelResource): class Meta: # What was there before... filtering = { ‘username’: ALL, ‘date_joined’: [‘range’, ‘gt’, ‘gte’, ‘lt’, ‘lte’], }

Slide 67

Slide 67 text

Filtering • Using GET params, we can now ﬁlter out what we want. • Examples: • curl http://localhost:8000/api/v1/user/? username__startswith=a • curl http://localhost:8000/api/v1/user/? date_joined__gte=2011-12-01

Slide 68

Slide 68 text

Authorization “Are you allowed to perform that action?”

Slide 69

Slide 69 text

Add authorization from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication from tastypie.authorization import DjangoAuthorization from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: # What was there before... authorization = DjangoAuthorization()

Slide 70

Slide 70 text

Add caching* from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication from tastypie.authorization import DjangoAuthorization from tastypie.cache import SimpleCache from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: # What was there before... cache = SimpleCache() * We’ll talk more about caching later.

Slide 71

Slide 71 text

Add throttling from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication from tastypie.authorization import DjangoAuthorization from tastypie.cache import SimpleCache from tastypie.resources import ModelResource from tastypie.throttle import CacheDBThrottle class UserResource(ModelResource): class Meta: # What was there before... throttle = CacheDBThrottle()

Slide 72

Slide 72 text

Whew.

Slide 73

Slide 73 text

What’s there now? • Everything we had before

Slide 74

Slide 74 text

What’s there now? • Everything we had before • Full GET/POST/PUT/DELETE/PATCH access

Slide 75

Slide 75 text

What’s there now? • Everything we had before • Full GET/POST/PUT/DELETE/PATCH access • Only registered users can use the API & only perform actions on objects they’re allowed to

Slide 76

Slide 76 text

What’s there now? (cont.) • Object-level caching (GET detail)

Slide 77

Slide 77 text

What’s there now? (cont.) • Object-level caching (GET detail) • Logged throttling that limits users to 150 reqs per hour

Slide 78

Slide 78 text

What’s there now? (cont.) • Object-level caching (GET detail) • Logged throttling that limits users to 150 reqs per hour • The ability to ﬁlter the content

Slide 79

Slide 79 text

HTTP 302

Slide 80

Slide 80 text

Extensibility

Slide 81

Slide 81 text

Why classes?

Slide 82

Slide 82 text

Why classes? Not because I’m OO-crazy. It makes extending behavior trivial.

Slide 83

Slide 83 text

Why so many classes?

Slide 84

Slide 84 text

Why so many classes? Composition > Inheritance

Slide 85

Slide 85 text

Why so many methods?

Slide 86

Slide 86 text

Why so many methods? Hooks, hooks, hooks. Also makes delegating to composition behaviors easy.

Slide 87

Slide 87 text

Extensibility • Tastypie tries to use reasonable defaults: • You probably want JSON • You probably want full POST/PUT/ DELETE by default • You probably want to use the Model’s default manager unﬁltered

Slide 88

Slide 88 text

But.

Slide 89

Slide 89 text

YMMV, so let’s make that possible.

Slide 90

Slide 90 text

Extensibility • Plug in custom classes/instances for things like: • Serialization

Slide 91

Slide 91 text

Extensibility • Plug in custom classes/instances for things like: • Serialization • Authentication

Slide 92

Slide 92 text

Extensibility • Plug in custom classes/instances for things like: • Serialization • Authentication • Authorization

Slide 93

Slide 93 text

Extensibility • Plug in custom classes/instances for things like: • Serialization • Authentication • Authorization • Pagination

Slide 94

Slide 94 text

Extensibility • Plug in custom classes/instances for things like: • Serialization • Authentication • Authorization • Pagination • Caching

Slide 95

Slide 95 text

Extensibility • Plug in custom classes/instances for things like: • Serialization • Authentication • Authorization • Pagination • Caching • Throttling

Slide 96

Slide 96 text

Extensibility • Resource has lots of methods, many of which are pretty granular • Override or extend as meets your needs

Slide 97

Slide 97 text

Customize serialization • As an example, let’s customize serialization

Slide 98

Slide 98 text

Customize serialization • As an example, let’s customize serialization • Supports JSON, XML, YAML, bplist by default

Slide 99

Slide 99 text

Customize serialization • As an example, let’s customize serialization • Supports JSON, XML, YAML, bplist by default • Let’s disable everything but JSON & XML, then add a custom type

Slide 100

Slide 100 text

Just JSON & XML, please from django.contrib.auth.models import User from tastypie.resources import ModelResource from tastypie.serialization import Serializer class UserResource(ModelResource): class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’] serializer = Serializer(formats=[‘json’, ‘xml’])

Slide 101

Slide 101 text

Hm, now what format is missing?

Slide 102

Slide 102 text

HTML

Slide 103

Slide 103 text

HTML serialization from django.shortcuts import render_to_response from tastypie.serialization import Serializer class TemplateSerializer(Serializer): formats = Serializer.formats + [‘html’] def to_html(self, data): template_name = ‘api/api_detail.html’ if ‘objects’ in data: template_name = ‘api/api_list.html’ return render_to_response(template_name, data)

Slide 104

Slide 104 text

HTML serialization (cont.) # ಠ_ಠ import cgi from stringio import StringIO class TemplateSerializer(Serializer): # ... def from_html(self, content): form = cgi.FieldStorage(fp=StringIO(content)) data = {} for key in form: data[key] = form[key].value return data

Slide 105

Slide 105 text

Now use it from django.contrib.auth.models import User from tastypie.resources import ModelResource from myapp.api.serializers import TemplateSerializer class UserResource(ModelResource): class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’] serializer = TemplateSerializer(formats=[‘json’, ‘xml’, ‘html’])

Slide 106

Slide 106 text

Fields

Slide 107

Slide 107 text

Fields • You haven’t seen it yet, but just like a ModelForm, you can control all the exposed ﬁelds on a Resource/ ModelResource

Slide 108

Slide 108 text

Fields • You haven’t seen it yet, but just like a ModelForm, you can control all the exposed ﬁelds on a Resource/ ModelResource • Just like Django, you use a declarative syntax

Slide 109

Slide 109 text

Fields from django.contrib.auth.models import User from tastypie import fields from tastypie.resources import ModelResource class UserResource(ModelResource): # Provided they take no args, even callables work! full_name = fields.CharField(‘get_full_name’, blank=True) class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’]

Slide 110

Slide 110 text

Fields (cont.) • Also like ModelForm, you can control how data gets prepared for presentation (dehydrate) or accepted from the user (hydrate)

Slide 111

Slide 111 text

Fields (cont.) • Also like ModelForm, you can control how data gets prepared for presentation (dehydrate) or accepted from the user (hydrate) • Happens automatically on ﬁelds with attribute=... set

Slide 112

Slide 112 text

Slide 113

Slide 113 text

Fields (cont.) class UserResource(ModelResource): # Assuming all the same bits as before. full_name = fields.CharField(blank=True) def dehydrate_full_name(self, bundle): return bundle.obj.get_full_name() def hydrate_full_name(self, bundle): name_bits = bundle.data.get(‘full_name’,’’).split() bundle.obj.first_name = name_bits[0] bundle.obj.last_name = ‘ ‘.join(name_bits[1:]) return bundle

Slide 114

Slide 114 text

Fields (cont.) • This just scratches the surface of what dehydrate/hydrate can do • ModelResource uses introspection & just creates the ﬁelds for you

Slide 115

Slide 115 text

Caching

Slide 116

Slide 116 text

Caching • The SimpleCache combined with Resource.cached_obj_get caches SINGLE objects only!

Slide 117

Slide 117 text

Caching • The SimpleCache combined with Resource.cached_obj_get caches SINGLE objects only! • Doesn’t cache the serialized output

Slide 118

Slide 118 text

Caching • The SimpleCache combined with Resource.cached_obj_get caches SINGLE objects only! • Doesn’t cache the serialized output • Doesn’t cache the list view

Slide 119

Slide 119 text

Why? • More complex behaviors get opinionated fast

Slide 120

Slide 120 text

Why? • More complex behaviors get opinionated fast • Tastypie would rather be general & give you the tools to build what you need

Slide 121

Slide 121 text

Why? • More complex behaviors get opinionated fast • Tastypie would rather be general & give you the tools to build what you need • Filters & serialization formats make it complex

Slide 122

Slide 122 text

Why? • More complex behaviors get opinionated fast • Tastypie would rather be general & give you the tools to build what you need • Filters & serialization formats make it complex • Besides...

Slide 123

Slide 123 text

What you actually want is Varnish.

Slide 124

Slide 124 text

Varnish! • https://www.varnish-cache.org/

Slide 125

Slide 125 text

Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C

Slide 126

Slide 126 text

Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C • Already caches by URI/headers

Slide 127

Slide 127 text

Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C • Already caches by URI/headers • Way faster than the Django request/ response cycle

Slide 128

Slide 128 text

Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C • Already caches by URI/headers • Way faster than the Django request/ response cycle • POST/PUT/DELETE just pass through

Slide 129

Slide 129 text

Varnish! (cont.) • So put Varnish in front of your API (& perhaps the rest of your site) & win in the general case

Slide 130

Slide 130 text

Slide 131

Slide 131 text

Varnish! (cont.) • So put Varnish in front of your API (& perhaps the rest of your site) & win in the general case • Additionally, use Tastypie’s internal caching to further speed up Varnish cache-misses • Easy to extend Resource to add in more caching

Slide 132

Slide 132 text

Slide 133

Slide 133 text

Long story short: You should be using Varnish.

Slide 134

Slide 134 text

Data source Not Just Models!

Slide 135

Slide 135 text

Source dive? • If you hit up tastypie/resources.py, you’ll ﬁnd something interesting

Slide 136

Slide 136 text

Source dive? • If you hit up tastypie/resources.py, you’ll ﬁnd something interesting • ModelResource is just a relatively thin (~300 lines) wrapper on top of Resource (~1200 lines)

Slide 137

Slide 137 text

Source dive? • If you hit up tastypie/resources.py, you’ll ﬁnd something interesting • ModelResource is just a relatively thin (~300 lines) wrapper on top of Resource (~1200 lines) • Just the ORM/Model bits

Slide 138

Slide 138 text

That’s right, virtually everything in Tastypie is available to non-ORM setups.

Slide 139

Slide 139 text

The Mighty Resource • By subclassing from Resource & overriding at least 3 (up to 9 for CRUD) methods, you can hook up any data source • For giggles, let’s hook up Solr! • For brevity, we’ll do GET-only

Slide 140

Slide 140 text

SolrResource import pysolr from tastypie import fields from tastypie.resources import Resource # Wrap the response dict to be object-like. class SolrObject(object): def __init__(self, initial=None): self.__dict__[‘_data’] = initial or {} def __getattr__(self, key): return self._data.get(key, None)

Slide 141

Slide 141 text

SolrResource (cont.) class SolrResource(Resource): id = fields.CharField(attribute=’id’) title = fields.CharField(attribute=‘text’) class Meta: resource_name = ‘solr’ object_class = SolrObject

Slide 142

Slide 142 text

SolrResource (cont.) class SolrResource(Resource): # ... def get_resource_uri(self, bundle_or_obj): # Super-naïve. return ‘/’.join([self._meta.api_name, self._meta.resource_name, bundle_or_obj.id]) def get_object_list(self, request, **kwargs): query = kwargs.get(‘query’, None) or request.GET.get(‘q’, ‘*:*’) solr = pysolr.Solr(‘http://localhost:8963/solr’) return [SolrObject(initial=res) for res in solr.search(query)])

Slide 143

Slide 143 text

SolrResource (cont.) class SolrResource(Resource): # ... def obj_get_list(self, request=None, **kwargs): return self.get_object_list(request) def obj_get(self, request=None, **kwargs): return self.get_object_list(request, {‘query’: ‘id:%s’ % kwargs[‘pk’])

Slide 144

Slide 144 text

The Mighty Resource • Takes some work but still does a lot for you • Docs have a more complete example based on Riak • See also django-tastypie-nonrel

Slide 145

Slide 145 text

HTTP 418

Slide 146

Slide 146 text

Now, let’s talk about a personal failure.

Slide 147

Slide 147 text

Man, this seems super-useful. Wish I could use it with...

Slide 148

Slide 148 text

Man, this seems super-useful. Wish I could use it with... Flask

Slide 149

Slide 149 text

Man, this seems super-useful. Wish I could use it with... Flask Pyramid

Slide 150

Slide 150 text

Man, this seems super-useful. Wish I could use it with... Flask Pyramid Itty

Slide 151

Slide 151 text

Man, this seems super-useful. Wish I could use it with... Flask Pyramid Itty Straight WSGI

Slide 152

Slide 152 text

Piecrust The extraction that failed

Slide 153

Slide 153 text

Piecrust • Seems like an awesome, helpful idea. Let’s do it! • Late 2011, I tried extracting Tastypie to work everywhere • Tastypie would just become a light shim on top with Django conveniences • https://github.com/toastdriven/piecrust

Slide 154

Slide 154 text

Piecrust • Finished the extraction, not the tests/ docs

Slide 155

Slide 155 text

Piecrust • Finished the extraction, not the tests/ docs • Close to functional, but...

Slide 156

Slide 156 text

Piecrust • Finished the extraction, not the tests/ docs • Close to functional, but... • Failed in terms of complexity & lack of standardization

Slide 157

Slide 157 text

Piecrust • Complex areas: • Non-standard request/response objects • Non-standard URL setups • Data storage layer • Relying on Django makes these easy(- ier)

Slide 158

Slide 158 text

Piecrust • Just implementing for Flask took a couple hundred lines alone & it was incomplete • Paradox of choice • Code is there if you want to play