$30 off During Our Annual Pro Sale. View Details »

Taking Django's ORM Async

Andrew Godwin
September 04, 2020

Taking Django's ORM Async

A talk I gave at DjangoCon AU 2020 about the plans and sequencing for making Django's ORM support asynchronous call styles.

Andrew Godwin

September 04, 2020

More Decks by Andrew Godwin

Other Decks in Programming



  2. Andrew Godwin / @andrewgodwin Hi, I’m Andrew Godwin • Django

    core developer • Worked on Migrations, Channels & Async • Not currently in Australia
  3. Andrew Godwin / @andrewgodwin

  4. Andrew Godwin / @andrewgodwin Async views are released! Find them

    in a Django 3.1 near you.
  5. Andrew Godwin / @andrewgodwin WSGIHandler __call__ WSGI Server WSGIRequest URLs

    Middleware View __call__ ASGIHandler __call__ ASGI Server ASGIRequest Asynchronous request path BaseHandler get_response_async BaseHandler get_response URLs Middleware Async View __call__ Implemented async request flow
  6. Andrew Godwin / @andrewgodwin Phase One: ASGI Support Allowing Django

    to be async at all Phase Two: Async Views Unlocking async use in normal apps Phase Three: The ORM High-level async use for the most common case
  7. Andrew Godwin / @andrewgodwin The ORM is the majority of

    Django And, of course, the most complex part.
  8. Andrew Godwin / @andrewgodwin API Design is crucial It must

    be familiar, yet safe.
  9. Andrew Godwin / @andrewgodwin Async is… a bit different You

    can't quite do everything you're used to
  10. Andrew Godwin / @andrewgodwin Can't tell if a function returns

    a coroutine! There are standard hints, but no actual guaranteed way
  11. Andrew Godwin / @andrewgodwin async def calculate(x): result = await

    coroutine(x) return result # These both return a coroutine def calculate(x): result = coroutine(x) return result
  12. Andrew Godwin / @andrewgodwin You have to namespace async functions

    I really, really wish we didn't have to
  13. Andrew Godwin / @andrewgodwin instance = MyModel.objects.get(id=3) instance = await

  14. Andrew Godwin / @andrewgodwin Can't do asynchronous attribute access! Foreign

    keys will be a bit less magic.
  15. Andrew Godwin / @andrewgodwin book = Book.objects.async_get(id=3) # Will error

    - needs to run a query print(book.author.name)
  16. Andrew Godwin / @andrewgodwin book = Book.objects.get(id=3) .select_related("author") # Works

    fine - prefetched print(book.author.name)
  17. Andrew Godwin / @andrewgodwin Some things do have nice analogues!

    They have async versions of the operations that call a different special method.
  18. Andrew Godwin / @andrewgodwin async for book in Book.objects.filter(name="Django"): print(book)

  19. Andrew Godwin / @andrewgodwin The Plan

  20. Andrew Godwin / @andrewgodwin 1. Async Model API Querysets &

    model instances mostly 2. Async Query Internals Django is internally async through its stack 3. Async Database Adapters Async all the way down, no threads at all
  21. Andrew Godwin / @andrewgodwin Query Threaded if in async mode

    After Phase One QuerySet Managers Model Connection Compiler Database Library
  22. Andrew Godwin / @andrewgodwin Query Threaded if in async mode

    After Phase Two QuerySet Managers Model Connection Compiler Database Library
  23. Andrew Godwin / @andrewgodwin Query After Phase Three QuerySet Managers

    Model Connection Compiler Database Library
  24. Andrew Godwin / @andrewgodwin We may never get to Phase

    Three ...and that's alright.
  25. Andrew Godwin / @andrewgodwin No sign of an async DBAPI...

    yet It might emerge once there's a need for it
  26. Andrew Godwin / @andrewgodwin Databases via threads? Not the worst!

    Native async only matters for pure performance
  27. Andrew Godwin / @andrewgodwin Transactions are tricky. They are very

    threadlocal, and that's not great with async
  28. Andrew Godwin / @andrewgodwin Some things don't need async Migrations,

    introspection, most fields
  29. Andrew Godwin / @andrewgodwin So what's first? (Apart from having

    to curse various non-PostgreSQL databases)
  30. Andrew Godwin / @andrewgodwin Deciding on async API design Namespacing?

    async_ prefixes? Something else?
  31. Andrew Godwin / @andrewgodwin Grappling with transactions Do we make

    them work across async/sync boundaries?
  32. Andrew Godwin / @andrewgodwin Async connection management Namespacing? async_ prefixes?

    Something else?
  33. Andrew Godwin / @andrewgodwin Asyncio only benefits IO-bound code Code

    that thrashes the CPU doesn't benefit at all
  34. Andrew Godwin / @andrewgodwin There's lots to come! Hopefully starting

    in 3.2, but no guarantees...
  35. Thanks. Andrew Godwin @andrewgodwin // aeracode.org