RESTful APIs with Tastypie

Transcript

1. RESTful APIs with Tastypie

2. ...or...

3. How I learned to stop worrying and love the JSON

4. Who am I? • Daniel Lindsley

5. Who am I? • Daniel Lindsley • Consulting & OSS

   as Toast Driven

6. Who am I? • Daniel Lindsley • Consulting & OSS

   as Toast Driven • Primary author of Tastypie

7. What is Tastypie? • A REST framework for Django

8. What is Tastypie? • A REST framework for Django •

   Designed for extension

9. What is Tastypie? • A REST framework for Django •

   Designed for extension • Supports both Model & non-Model data

10. What is Tastypie? • A REST framework for Django •

    Designed for extension • Supports both Model & non-Model data • http://tastypieapi.org/

11. Philosophy

12. Make good use of HTTP

13. Make good use of HTTP Try to be “of the

    internet” & use the REST methods/status codes properly

14. Graceful Degradation

15. Graceful Degradation Try to keep backward compatibility & give users

    a gradual upgrade path

16. Flexible serialization

17. Flexible serialization Not everyone wants JSON

18. Wait. Scratch that.

19. Flexible serialization Not everyone wants JSON

20. Flexible EVERYTHING

21. Flexible EVERYTHING Customizability is a core feature

22. Data can round-trip

23. Data can round-trip Anything you can GET, you should be

    able to POST/PUT

24. Reasonable defaults

25. Reasonable defaults But easy to extend

26. URIs everywhere!

27. URIs everywhere! Make HATEOAS a reality

28. HATEOAS?

29. Gesundheit!

30. HATEOAS • “Hypermedia As The Engine Of Application State” •

    Basically, the user shouldn’t have to know anything in advance • All about explore-ability • Deep linking • http://en.wikipedia.org/wiki/HATEOAS

31. So what about Tastypie?

32. Pie?

33. Pie? Pie? Pie? Pie? Pie? Pie? Pie? Pie?

34. He can’t haz.

35. He can’t haz. But you can!

36. Tastypie • Builds on top of Django & plays nicely

    with other apps • Full GET/POST/PUT/DELETE/PATCH • Any data source (not just Models) • Designed to be extended

37. Tastypie (cont.) • Supports a variety of serialization formats •

    JSON • XML • YAML • bplist • Easy to add more

38. Tastypie (cont.) • HATEOAS by default (you’ll see soon) •

    Lots of hooks for customization • Well-tested • Decently documented

39. The Setup

40. Installation pip install django-tastypie

41. Installation (cont.) INSTALLED_APPS += [‘tastypie’]

42. Installation (cont.) $ ./manage.py syncdb

43. Done.

44. Let’s add an API for django.contrib.auth

45. High-Level 1. Code goes in our apps, not Django

46. High-Level 1. Code goes in our apps, not Django 2.

    Define the resource for User

47. High-Level 1. Code goes in our apps, not Django 2.

    Define the resource for User 3. Hook up that resource in your URLconf

48. 1. Don’t fork Django! No, seriously. You don’t need to.

    Done.

49. 2a. Setup # Assuming we’re in your project directory... $

    cd <myapp> # Substitute your app_name here. $ mkdir api $ touch api/__init__.py $ touch api/resources.py # Done!

50. 2b. User Resource from django.contrib.auth.models import User from tastypie.resources import

    ModelResource class UserResource(ModelResource): class Meta: queryset = User.objects.all()

51. 3. URLconf # In your ``ROOT_URLCONF``... # Stuff then... from

    tastypie.api import Api from <myapp>.api.resources import UserResource v1_api = Api() v1_api.register(UserResource()) urlpatterns = patterns(‘’, (r’^api/’, include(v1_api.urls), # Then the usual... )

52. HTTP 201

53. Hit your new API. Curl: http://localhost:8000/api/v1/ Browser: http://localhost:8000/api/v1/?format=json

54. Demo-time.

55. What’s there? • /api/v1/ - A list of all available

    resources • /api/v1/user/ - A list of all users • /api/v1/user/2/ - A specific user • /api/v1/user/schema/ - A definition of what an individual user consists of • /api/v1/user/multiple/1;4;5/ - Get those three users as one request

56. What’s there? (cont.) • All serialization formats available* • curl

    -H “Accept: application/xml” http://localhost:8000/api/v1/ user/ • http://localhost:8000/api/v1/user/2/? format=yaml * Provided lxml, PyYAML, biplist are installed.

57. What’s there? (cont.) • Serialization format negotiated by either Accepts

    header or the ”? format=json” GET param • Pagination by default • Everyone has full read-only GET access

58. What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser

59. What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser

    • Ability to filter

60. What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser

    • Ability to filter • Authentication / Authorization

61. What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser

    • Ability to filter • Authentication / Authorization • Caching (disabled by default)

62. What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser

    • Ability to filter • Authentication / Authorization • Caching (disabled by default) • Throttling (disabled by default)

63. Fix data leaks from django.contrib.auth.models import User from tastypie.resources import

    ModelResource class UserResource(ModelResource): class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’]

64. Authentication “Are you a recognized user?”

65. Add BASIC Auth from django.contrib.auth.models import User from tastypie.authentication import

    BasicAuthentication from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: # What was there before... authentication = BasicAuthentication()

66. Add filtering from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication

    from tastypie.resources import ModelResource, ALL class UserResource(ModelResource): class Meta: # What was there before... filtering = { ‘username’: ALL, ‘date_joined’: [‘range’, ‘gt’, ‘gte’, ‘lt’, ‘lte’], }

67. Filtering • Using GET params, we can now filter out

    what we want. • Examples: • curl http://localhost:8000/api/v1/user/? username__startswith=a • curl http://localhost:8000/api/v1/user/? date_joined__gte=2011-12-01

68. Authorization “Are you allowed to perform that action?”

69. Add authorization from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication

    from tastypie.authorization import DjangoAuthorization from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: # What was there before... authorization = DjangoAuthorization()

70. Add caching* from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication

    from tastypie.authorization import DjangoAuthorization from tastypie.cache import SimpleCache from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: # What was there before... cache = SimpleCache() * We’ll talk more about caching later.

71. Add throttling from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication

    from tastypie.authorization import DjangoAuthorization from tastypie.cache import SimpleCache from tastypie.resources import ModelResource from tastypie.throttle import CacheDBThrottle class UserResource(ModelResource): class Meta: # What was there before... throttle = CacheDBThrottle()

72. Whew.

73. What’s there now? • Everything we had before

74. What’s there now? • Everything we had before • Full

    GET/POST/PUT/DELETE/PATCH access

75. What’s there now? • Everything we had before • Full

    GET/POST/PUT/DELETE/PATCH access • Only registered users can use the API & only perform actions on objects they’re allowed to

76. What’s there now? (cont.) • Object-level caching (GET detail)

77. What’s there now? (cont.) • Object-level caching (GET detail) •

    Logged throttling that limits users to 150 reqs per hour

78. What’s there now? (cont.) • Object-level caching (GET detail) •

    Logged throttling that limits users to 150 reqs per hour • The ability to filter the content

79. HTTP 302

80. Extensibility

81. Why classes?

82. Why classes? Not because I’m OO-crazy. It makes extending behavior

    trivial.

83. Why so many classes?

84. Why so many classes? Composition > Inheritance

85. Why so many methods?

86. Why so many methods? Hooks, hooks, hooks. Also makes delegating

    to composition behaviors easy.

87. Extensibility • Tastypie tries to use reasonable defaults: • You

    probably want JSON • You probably want full POST/PUT/ DELETE by default • You probably want to use the Model’s default manager unfiltered

88. But.

89. YMMV, so let’s make that possible.

90. Extensibility • Plug in custom classes/instances for things like: •

    Serialization

91. Extensibility • Plug in custom classes/instances for things like: •

    Serialization • Authentication

92. Extensibility • Plug in custom classes/instances for things like: •

    Serialization • Authentication • Authorization

93. Extensibility • Plug in custom classes/instances for things like: •

    Serialization • Authentication • Authorization • Pagination

94. Extensibility • Plug in custom classes/instances for things like: •

    Serialization • Authentication • Authorization • Pagination • Caching

95. Extensibility • Plug in custom classes/instances for things like: •

    Serialization • Authentication • Authorization • Pagination • Caching • Throttling

96. Extensibility • Resource has lots of methods, many of which

    are pretty granular • Override or extend as meets your needs

97. Customize serialization • As an example, let’s customize serialization

98. Customize serialization • As an example, let’s customize serialization •

    Supports JSON, XML, YAML, bplist by default

99. Customize serialization • As an example, let’s customize serialization •

    Supports JSON, XML, YAML, bplist by default • Let’s disable everything but JSON & XML, then add a custom type

100. Just JSON & XML, please from django.contrib.auth.models import User from

     tastypie.resources import ModelResource from tastypie.serialization import Serializer class UserResource(ModelResource): class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’] serializer = Serializer(formats=[‘json’, ‘xml’])

101. Hm, now what format is missing?

102. HTML

103. HTML serialization from django.shortcuts import render_to_response from tastypie.serialization import Serializer

     class TemplateSerializer(Serializer): formats = Serializer.formats + [‘html’] def to_html(self, data): template_name = ‘api/api_detail.html’ if ‘objects’ in data: template_name = ‘api/api_list.html’ return render_to_response(template_name, data)

104. HTML serialization (cont.) # ಠ_ಠ import cgi from stringio import

     StringIO class TemplateSerializer(Serializer): # ... def from_html(self, content): form = cgi.FieldStorage(fp=StringIO(content)) data = {} for key in form: data[key] = form[key].value return data

105. Now use it from django.contrib.auth.models import User from tastypie.resources import

     ModelResource from myapp.api.serializers import TemplateSerializer class UserResource(ModelResource): class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’] serializer = TemplateSerializer(formats=[‘json’, ‘xml’, ‘html’])

106. Fields

107. Fields • You haven’t seen it yet, but just like

     a ModelForm, you can control all the exposed fields on a Resource/ ModelResource

108. Fields • You haven’t seen it yet, but just like

     a ModelForm, you can control all the exposed fields on a Resource/ ModelResource • Just like Django, you use a declarative syntax

109. Fields from django.contrib.auth.models import User from tastypie import fields from

     tastypie.resources import ModelResource class UserResource(ModelResource): # Provided they take no args, even callables work! full_name = fields.CharField(‘get_full_name’, blank=True) class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’]

110. Fields (cont.) • Also like ModelForm, you can control how

     data gets prepared for presentation (dehydrate) or accepted from the user (hydrate)

111. Fields (cont.) • Also like ModelForm, you can control how

     data gets prepared for presentation (dehydrate) or accepted from the user (hydrate) • Happens automatically on fields with attribute=... set

112. Fields (cont.) • Also like ModelForm, you can control how

     data gets prepared for presentation (dehydrate) or accepted from the user (hydrate) • Happens automatically on fields with attribute=... set • Can provide methods for non-simple access

113. Fields (cont.) class UserResource(ModelResource): # Assuming all the same bits

     as before. full_name = fields.CharField(blank=True) def dehydrate_full_name(self, bundle): return bundle.obj.get_full_name() def hydrate_full_name(self, bundle): name_bits = bundle.data.get(‘full_name’,’’).split() bundle.obj.first_name = name_bits[0] bundle.obj.last_name = ‘ ‘.join(name_bits[1:]) return bundle

114. Fields (cont.) • This just scratches the surface of what

     dehydrate/hydrate can do • ModelResource uses introspection & just creates the fields for you

115. Caching

116. Caching • The SimpleCache combined with Resource.cached_obj_get caches SINGLE objects

     only!

117. Caching • The SimpleCache combined with Resource.cached_obj_get caches SINGLE objects

     only! • Doesn’t cache the serialized output

118. Caching • The SimpleCache combined with Resource.cached_obj_get caches SINGLE objects

     only! • Doesn’t cache the serialized output • Doesn’t cache the list view

119. Why? • More complex behaviors get opinionated fast

120. Why? • More complex behaviors get opinionated fast • Tastypie

     would rather be general & give you the tools to build what you need

121. Why? • More complex behaviors get opinionated fast • Tastypie

     would rather be general & give you the tools to build what you need • Filters & serialization formats make it complex

122. Why? • More complex behaviors get opinionated fast • Tastypie

     would rather be general & give you the tools to build what you need • Filters & serialization formats make it complex • Besides...

123. What you actually want is Varnish.

124. Varnish! • https://www.varnish-cache.org/

125. Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C

126. Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C

     • Already caches by URI/headers

127. Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C

     • Already caches by URI/headers • Way faster than the Django request/ response cycle

128. Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C

     • Already caches by URI/headers • Way faster than the Django request/ response cycle • POST/PUT/DELETE just pass through

129. Varnish! (cont.) • So put Varnish in front of your

     API (& perhaps the rest of your site) & win in the general case

130. Varnish! (cont.) • So put Varnish in front of your

     API (& perhaps the rest of your site) & win in the general case • Additionally, use Tastypie’s internal caching to further speed up Varnish cache-misses

131. Varnish! (cont.) • So put Varnish in front of your

     API (& perhaps the rest of your site) & win in the general case • Additionally, use Tastypie’s internal caching to further speed up Varnish cache-misses • Easy to extend Resource to add in more caching

132. Varnish! (cont.) • So put Varnish in front of your

     API (& perhaps the rest of your site) & win in the general case • Additionally, use Tastypie’s internal caching to further speed up Varnish cache-misses • Easy to extend Resource to add in more caching • If you get to that point, you’re already serving way more load than I ever have

133. Long story short: You should be using Varnish.

134. Data source Not Just Models!

135. Source dive? • If you hit up tastypie/resources.py, you’ll find

     something interesting

136. Source dive? • If you hit up tastypie/resources.py, you’ll find

     something interesting • ModelResource is just a relatively thin (~300 lines) wrapper on top of Resource (~1200 lines)

137. Source dive? • If you hit up tastypie/resources.py, you’ll find

     something interesting • ModelResource is just a relatively thin (~300 lines) wrapper on top of Resource (~1200 lines) • Just the ORM/Model bits

138. That’s right, virtually everything in Tastypie is available to non-ORM

     setups.

139. The Mighty Resource • By subclassing from Resource & overriding

     at least 3 (up to 9 for CRUD) methods, you can hook up any data source • For giggles, let’s hook up Solr! • For brevity, we’ll do GET-only

140. SolrResource import pysolr from tastypie import fields from tastypie.resources import

     Resource # Wrap the response dict to be object-like. class SolrObject(object): def __init__(self, initial=None): self.__dict__[‘_data’] = initial or {} def __getattr__(self, key): return self._data.get(key, None)

141. SolrResource (cont.) class SolrResource(Resource): id = fields.CharField(attribute=’id’) title = fields.CharField(attribute=‘text’)

     class Meta: resource_name = ‘solr’ object_class = SolrObject

142. SolrResource (cont.) class SolrResource(Resource): # ... def get_resource_uri(self, bundle_or_obj): #

     Super-naïve. return ‘/’.join([self._meta.api_name, self._meta.resource_name, bundle_or_obj.id]) def get_object_list(self, request, **kwargs): query = kwargs.get(‘query’, None) or request.GET.get(‘q’, ‘*:*’) solr = pysolr.Solr(‘http://localhost:8963/solr’) return [SolrObject(initial=res) for res in solr.search(query)])

143. SolrResource (cont.) class SolrResource(Resource): # ... def obj_get_list(self, request=None, **kwargs):

     return self.get_object_list(request) def obj_get(self, request=None, **kwargs): return self.get_object_list(request, {‘query’: ‘id:%s’ % kwargs[‘pk’])

144. The Mighty Resource • Takes some work but still does

     a lot for you • Docs have a more complete example based on Riak • See also django-tastypie-nonrel

145. HTTP 418

146. Now, let’s talk about a personal failure.

147. Man, this seems super-useful. Wish I could use it with...

148. Man, this seems super-useful. Wish I could use it with...

     Flask

149. Man, this seems super-useful. Wish I could use it with...

     Flask Pyramid

150. Man, this seems super-useful. Wish I could use it with...

     Flask Pyramid Itty

151. Man, this seems super-useful. Wish I could use it with...

     Flask Pyramid Itty Straight WSGI

152. Piecrust The extraction that failed

153. Piecrust • Seems like an awesome, helpful idea. Let’s do

     it! • Late 2011, I tried extracting Tastypie to work everywhere • Tastypie would just become a light shim on top with Django conveniences • https://github.com/toastdriven/piecrust

154. Piecrust • Finished the extraction, not the tests/ docs

155. Piecrust • Finished the extraction, not the tests/ docs •

     Close to functional, but...

156. Piecrust • Finished the extraction, not the tests/ docs •

     Close to functional, but... • Failed in terms of complexity & lack of standardization

157. Piecrust • Complex areas: • Non-standard request/response objects • Non-standard

     URL setups • Data storage layer • Relying on Django makes these easy(- ier)

158. Piecrust • Just implementing for Flask took a couple hundred

     lines alone & it was incomplete • Paradox of choice • Code is there if you want to play

159. Thanks!

160. Images in main theme by Julia Elman @juliaelman http://juliaelman.com/

161. I’m Daniel Lindsley of Toast Driven @toastdriven http://toastdriven.com/