Taking Django's ORM Async

A talk I gave at DjangoCon AU 2020 about the plans and sequencing for making Django's ORM support asynchronous call styles.

Transcript

1. TAKING DJANGO'S ORM ANDREW GODWIN // @andrewgodwin ASYNC
2. Andrew Godwin / @andrewgodwin Hi, I’m Andrew Godwin • Django core developer • Worked on Migrations, Channels & Async • Not currently in Australia
3. Andrew Godwin / @andrewgodwin
4. Andrew Godwin / @andrewgodwin Async views are released! Find them in a Django 3.1 near you.
5. Andrew Godwin / @andrewgodwin WSGIHandler __call__ WSGI Server WSGIRequest URLs Middleware View __call__ ASGIHandler __call__ ASGI Server ASGIRequest Asynchronous request path BaseHandler get_response_async BaseHandler get_response URLs Middleware Async View __call__ Implemented async request flow
6. Andrew Godwin / @andrewgodwin Phase One: ASGI Support Allowing Django to be async at all Phase Two: Async Views Unlocking async use in normal apps Phase Three: The ORM High-level async use for the most common case
7. Andrew Godwin / @andrewgodwin The ORM is the majority of Django And, of course, the most complex part.
8. Andrew Godwin / @andrewgodwin API Design is crucial It must be familiar, yet safe.
9. Andrew Godwin / @andrewgodwin Async is… a bit different You can't quite do everything you're used to
10. Andrew Godwin / @andrewgodwin Can't tell if a function returns a coroutine! There are standard hints, but no actual guaranteed way
11. Andrew Godwin / @andrewgodwin async def calculate(x): result = await coroutine(x) return result # These both return a coroutine def calculate(x): result = coroutine(x) return result
12. Andrew Godwin / @andrewgodwin You have to namespace async functions I really, really wish we didn't have to
13. Andrew Godwin / @andrewgodwin instance = MyModel.objects.get(id=3) instance = await MyModel.objects.async_get(id=3)
14. Andrew Godwin / @andrewgodwin Can't do asynchronous attribute access! Foreign keys will be a bit less magic.
15. Andrew Godwin / @andrewgodwin book = Book.objects.async_get(id=3) # Will error - needs to run a query print(book.author.name)
16. Andrew Godwin / @andrewgodwin book = Book.objects.get(id=3) .select_related("author") # Works fine - prefetched print(book.author.name)
17. Andrew Godwin / @andrewgodwin Some things do have nice analogues! They have async versions of the operations that call a different special method.
18. Andrew Godwin / @andrewgodwin async for book in Book.objects.filter(name="Django"): print(book)
19. Andrew Godwin / @andrewgodwin The Plan
20. Andrew Godwin / @andrewgodwin 1. Async Model API Querysets & model instances mostly 2. Async Query Internals Django is internally async through its stack 3. Async Database Adapters Async all the way down, no threads at all
21. Andrew Godwin / @andrewgodwin Query Threaded if in async mode After Phase One QuerySet Managers Model Connection Compiler Database Library
22. Andrew Godwin / @andrewgodwin Query Threaded if in async mode After Phase Two QuerySet Managers Model Connection Compiler Database Library
23. Andrew Godwin / @andrewgodwin Query After Phase Three QuerySet Managers Model Connection Compiler Database Library
24. Andrew Godwin / @andrewgodwin We may never get to Phase Three ...and that's alright.
25. Andrew Godwin / @andrewgodwin No sign of an async DBAPI... yet It might emerge once there's a need for it
26. Andrew Godwin / @andrewgodwin Databases via threads? Not the worst! Native async only matters for pure performance
27. Andrew Godwin / @andrewgodwin Transactions are tricky. They are very threadlocal, and that's not great with async
28. Andrew Godwin / @andrewgodwin Some things don't need async Migrations, introspection, most fields
29. Andrew Godwin / @andrewgodwin So what's first? (Apart from having to curse various non-PostgreSQL databases)
30. Andrew Godwin / @andrewgodwin Deciding on async API design Namespacing? async_ prefixes? Something else?
31. Andrew Godwin / @andrewgodwin Grappling with transactions Do we make them work across async/sync boundaries?
32. Andrew Godwin / @andrewgodwin Async connection management Namespacing? async_ prefixes? Something else?
33. Andrew Godwin / @andrewgodwin Asyncio only benefits IO-bound code Code that thrashes the CPU doesn't benefit at all
34. Andrew Godwin / @andrewgodwin There's lots to come! Hopefully starting in 3.2, but no guarantees...
35. Thanks. Andrew Godwin @andrewgodwin // aeracode.org