Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Taking Django's ORM Async

Andrew Godwin
September 04, 2020

Taking Django's ORM Async

A talk I gave at DjangoCon AU 2020 about the plans and sequencing for making Django's ORM support asynchronous call styles.

Andrew Godwin

September 04, 2020
Tweet

More Decks by Andrew Godwin

Other Decks in Programming

Transcript

  1. TAKING DJANGO'S ORM
    ANDREW GODWIN // @andrewgodwin
    ASYNC

    View full-size slide

  2. Andrew Godwin / @andrewgodwin
    Hi, I’m
    Andrew Godwin
    • Django core developer
    • Worked on Migrations, Channels & Async
    • Not currently in Australia

    View full-size slide

  3. Andrew Godwin / @andrewgodwin

    View full-size slide

  4. Andrew Godwin / @andrewgodwin
    Async views are released!
    Find them in a Django 3.1 near you.

    View full-size slide

  5. Andrew Godwin / @andrewgodwin
    WSGIHandler
    __call__
    WSGI Server
    WSGIRequest URLs Middleware
    View
    __call__
    ASGIHandler
    __call__
    ASGI Server
    ASGIRequest
    Asynchronous
    request path
    BaseHandler
    get_response_async
    BaseHandler
    get_response
    URLs Middleware
    Async View
    __call__
    Implemented async request flow

    View full-size slide

  6. Andrew Godwin / @andrewgodwin
    Phase One: ASGI Support
    Allowing Django to be async at all
    Phase Two: Async Views
    Unlocking async use in normal apps
    Phase Three: The ORM
    High-level async use for the most common case

    View full-size slide

  7. Andrew Godwin / @andrewgodwin
    The ORM is the majority of Django
    And, of course, the most complex part.

    View full-size slide

  8. Andrew Godwin / @andrewgodwin
    API Design is crucial
    It must be familiar, yet safe.

    View full-size slide

  9. Andrew Godwin / @andrewgodwin
    Async is… a bit different
    You can't quite do everything you're used to

    View full-size slide

  10. Andrew Godwin / @andrewgodwin
    Can't tell if a function returns a coroutine!
    There are standard hints, but no actual guaranteed way

    View full-size slide

  11. Andrew Godwin / @andrewgodwin
    async def calculate(x):
    result = await coroutine(x)
    return result
    # These both return a coroutine
    def calculate(x):
    result = coroutine(x)
    return result

    View full-size slide

  12. Andrew Godwin / @andrewgodwin
    You have to namespace async functions
    I really, really wish we didn't have to

    View full-size slide

  13. Andrew Godwin / @andrewgodwin
    instance = MyModel.objects.get(id=3)
    instance = await MyModel.objects.async_get(id=3)

    View full-size slide

  14. Andrew Godwin / @andrewgodwin
    Can't do asynchronous attribute access!
    Foreign keys will be a bit less magic.

    View full-size slide

  15. Andrew Godwin / @andrewgodwin
    book = Book.objects.async_get(id=3)
    # Will error - needs to run a query
    print(book.author.name)

    View full-size slide

  16. Andrew Godwin / @andrewgodwin
    book = Book.objects.get(id=3)
    .select_related("author")
    # Works fine - prefetched
    print(book.author.name)

    View full-size slide

  17. Andrew Godwin / @andrewgodwin
    Some things do have nice analogues!
    They have async versions of the operations that call a different special method.

    View full-size slide

  18. Andrew Godwin / @andrewgodwin
    async for book in Book.objects.filter(name="Django"):
    print(book)

    View full-size slide

  19. Andrew Godwin / @andrewgodwin
    The Plan

    View full-size slide

  20. Andrew Godwin / @andrewgodwin
    1. Async Model API
    Querysets & model instances mostly
    2. Async Query Internals
    Django is internally async through its stack
    3. Async Database Adapters
    Async all the way down, no threads at all

    View full-size slide

  21. Andrew Godwin / @andrewgodwin
    Query
    Threaded if in
    async mode
    After Phase One
    QuerySet Managers
    Model
    Connection Compiler
    Database
    Library

    View full-size slide

  22. Andrew Godwin / @andrewgodwin
    Query
    Threaded if in
    async mode
    After Phase Two
    QuerySet Managers
    Model
    Connection Compiler
    Database
    Library

    View full-size slide

  23. Andrew Godwin / @andrewgodwin
    Query
    After Phase Three
    QuerySet Managers
    Model
    Connection Compiler
    Database
    Library

    View full-size slide

  24. Andrew Godwin / @andrewgodwin
    We may never get to Phase Three
    ...and that's alright.

    View full-size slide

  25. Andrew Godwin / @andrewgodwin
    No sign of an async DBAPI... yet
    It might emerge once there's a need for it

    View full-size slide

  26. Andrew Godwin / @andrewgodwin
    Databases via threads? Not the worst!
    Native async only matters for pure performance

    View full-size slide

  27. Andrew Godwin / @andrewgodwin
    Transactions are tricky.
    They are very threadlocal, and that's not great with async

    View full-size slide

  28. Andrew Godwin / @andrewgodwin
    Some things don't need async
    Migrations, introspection, most fields

    View full-size slide

  29. Andrew Godwin / @andrewgodwin
    So what's first?
    (Apart from having to curse various non-PostgreSQL databases)

    View full-size slide

  30. Andrew Godwin / @andrewgodwin
    Deciding on async API design
    Namespacing? async_ prefixes? Something else?

    View full-size slide

  31. Andrew Godwin / @andrewgodwin
    Grappling with transactions
    Do we make them work across async/sync boundaries?

    View full-size slide

  32. Andrew Godwin / @andrewgodwin
    Async connection management
    Namespacing? async_ prefixes? Something else?

    View full-size slide

  33. Andrew Godwin / @andrewgodwin
    Asyncio only benefits IO-bound code
    Code that thrashes the CPU doesn't benefit at all

    View full-size slide

  34. Andrew Godwin / @andrewgodwin
    There's lots to come!
    Hopefully starting in 3.2, but no guarantees...

    View full-size slide

  35. Thanks.
    Andrew Godwin
    @andrewgodwin // aeracode.org

    View full-size slide