Taking Django's ORM Async

Taking Django's ORM Async

A talk I gave at DjangoCon AU 2020 about the plans and sequencing for making Django's ORM support asynchronous call styles.

077e9a0cb34fa3eba2699240c9509717?s=128

Andrew Godwin

September 04, 2020
Tweet

Transcript

  1. TAKING DJANGO'S ORM ANDREW GODWIN // @andrewgodwin ASYNC

  2. Andrew Godwin / @andrewgodwin Hi, I’m Andrew Godwin • Django

    core developer • Worked on Migrations, Channels & Async • Not currently in Australia
  3. Andrew Godwin / @andrewgodwin

  4. Andrew Godwin / @andrewgodwin Async views are released! Find them

    in a Django 3.1 near you.
  5. Andrew Godwin / @andrewgodwin WSGIHandler __call__ WSGI Server WSGIRequest URLs

    Middleware View __call__ ASGIHandler __call__ ASGI Server ASGIRequest Asynchronous request path BaseHandler get_response_async BaseHandler get_response URLs Middleware Async View __call__ Implemented async request flow
  6. Andrew Godwin / @andrewgodwin Phase One: ASGI Support Allowing Django

    to be async at all Phase Two: Async Views Unlocking async use in normal apps Phase Three: The ORM High-level async use for the most common case
  7. Andrew Godwin / @andrewgodwin The ORM is the majority of

    Django And, of course, the most complex part.
  8. Andrew Godwin / @andrewgodwin API Design is crucial It must

    be familiar, yet safe.
  9. Andrew Godwin / @andrewgodwin Async is… a bit different You

    can't quite do everything you're used to
  10. Andrew Godwin / @andrewgodwin Can't tell if a function returns

    a coroutine! There are standard hints, but no actual guaranteed way
  11. Andrew Godwin / @andrewgodwin async def calculate(x): result = await

    coroutine(x) return result # These both return a coroutine def calculate(x): result = coroutine(x) return result
  12. Andrew Godwin / @andrewgodwin You have to namespace async functions

    I really, really wish we didn't have to
  13. Andrew Godwin / @andrewgodwin instance = MyModel.objects.get(id=3) instance = await

    MyModel.objects.async_get(id=3)
  14. Andrew Godwin / @andrewgodwin Can't do asynchronous attribute access! Foreign

    keys will be a bit less magic.
  15. Andrew Godwin / @andrewgodwin book = Book.objects.async_get(id=3) # Will error

    - needs to run a query print(book.author.name)
  16. Andrew Godwin / @andrewgodwin book = Book.objects.get(id=3) .select_related("author") # Works

    fine - prefetched print(book.author.name)
  17. Andrew Godwin / @andrewgodwin Some things do have nice analogues!

    They have async versions of the operations that call a different special method.
  18. Andrew Godwin / @andrewgodwin async for book in Book.objects.filter(name="Django"): print(book)

  19. Andrew Godwin / @andrewgodwin The Plan

  20. Andrew Godwin / @andrewgodwin 1. Async Model API Querysets &

    model instances mostly 2. Async Query Internals Django is internally async through its stack 3. Async Database Adapters Async all the way down, no threads at all
  21. Andrew Godwin / @andrewgodwin Query Threaded if in async mode

    After Phase One QuerySet Managers Model Connection Compiler Database Library
  22. Andrew Godwin / @andrewgodwin Query Threaded if in async mode

    After Phase Two QuerySet Managers Model Connection Compiler Database Library
  23. Andrew Godwin / @andrewgodwin Query After Phase Three QuerySet Managers

    Model Connection Compiler Database Library
  24. Andrew Godwin / @andrewgodwin We may never get to Phase

    Three ...and that's alright.
  25. Andrew Godwin / @andrewgodwin No sign of an async DBAPI...

    yet It might emerge once there's a need for it
  26. Andrew Godwin / @andrewgodwin Databases via threads? Not the worst!

    Native async only matters for pure performance
  27. Andrew Godwin / @andrewgodwin Transactions are tricky. They are very

    threadlocal, and that's not great with async
  28. Andrew Godwin / @andrewgodwin Some things don't need async Migrations,

    introspection, most fields
  29. Andrew Godwin / @andrewgodwin So what's first? (Apart from having

    to curse various non-PostgreSQL databases)
  30. Andrew Godwin / @andrewgodwin Deciding on async API design Namespacing?

    async_ prefixes? Something else?
  31. Andrew Godwin / @andrewgodwin Grappling with transactions Do we make

    them work across async/sync boundaries?
  32. Andrew Godwin / @andrewgodwin Async connection management Namespacing? async_ prefixes?

    Something else?
  33. Andrew Godwin / @andrewgodwin Asyncio only benefits IO-bound code Code

    that thrashes the CPU doesn't benefit at all
  34. Andrew Godwin / @andrewgodwin There's lots to come! Hopefully starting

    in 3.2, but no guarantees...
  35. Thanks. Andrew Godwin @andrewgodwin // aeracode.org