How To Break Django: With Async

How To Break Django: With Async

A talk I gave at DjangoCon Europe 2020.

077e9a0cb34fa3eba2699240c9509717?s=128

Andrew Godwin

September 19, 2020
Tweet

Transcript

  1. HOW TO BREAK DJANGO: ANDREW GODWIN // @andrewgodwin WITH ASYNC

  2. Andrew Godwin / @andrewgodwin Hi, I’m Andrew Godwin • Django

    core developer • Worked on Migrations, Channels & Async • Not currently in Europe
  3. Andrew Godwin / @andrewgodwin

  4. Andrew Godwin / @andrewgodwin Async views are released! Find them

    in a Django 3.1 near you.
  5. Andrew Godwin / @andrewgodwin WSGIHandler __call__ WSGI Server WSGIRequest URLs

    Middleware View __call__ ASGIHandler __call__ ASGI Server ASGIRequest Asynchronous request path BaseHandler get_response_async BaseHandler get_response URLs Middleware Async View __call__ Implemented async request flow
  6. Andrew Godwin / @andrewgodwin Synchronous programming is quite safe Sometimes

    that's worth the performance cost
  7. Andrew Godwin / @andrewgodwin Async programming is hard But it

    can definitely be worth it!
  8. Andrew Godwin / @andrewgodwin I love learning by example …

    so please don't do any of these things.
  9. Andrew Godwin / @andrewgodwin async def my_view(request, book_id): book =

    Book.objects.get(pk=book_id) return render( request, "book.html", {"book": book} )
  10. Andrew Godwin / @andrewgodwin async def my_view(request, book_id): book =

    await Book.objects.get_async( pk=book_id ) return render( request, "book.html", {"book": book} )
  11. Andrew Godwin / @andrewgodwin SynchronousOnlyOperation: You cannot call this from

    an async context - use a thread or sync_to_async
  12. Andrew Godwin / @andrewgodwin async def my_view(request, book_id): book =

    Book.objects.get_async( pk=book_id ) return render( request, "book.html", {"book": book} )
  13. Andrew Godwin / @andrewgodwin Django's got your back ...this time,

    anyway.
  14. Andrew Godwin / @andrewgodwin async def my_view(request): ... asyncio.wait({ create_user_account(),

    send_created_email(), }) ...
  15. Andrew Godwin / @andrewgodwin There's no ordering guarantee! Or even

    a guarantee it will execute.
  16. Andrew Godwin / @andrewgodwin async def my_view(request): ... await create_user_account()

    await send_created_email() ...
  17. Andrew Godwin / @andrewgodwin Race conditions! Names are power -

    now you can research it!
  18. Andrew Godwin / @andrewgodwin # This is how you need

    to do DB access in 3.1 @sync_to_async def create_user_account(): User.objects.create(...)
  19. Andrew Godwin / @andrewgodwin async def my_view(request): with transaction.atomic(): await

    verify_valid_user(...) await create_user_account(...)
  20. Andrew Godwin / @andrewgodwin async def my_view(request): with transaction.atomic(): await

    verify_valid_user(...) await create_user_account(...) Runs in async thread In different, sync thread
  21. Andrew Godwin / @andrewgodwin You don't actually have a transaction.

    We need to address this when we make the ORM async-aware!
  22. Andrew Godwin / @andrewgodwin stats = {"alive": 0, "dead": 0}

    async def fetch_site(site): alive = stats["alive"] response = await client.get(site) if response.status_code < 400: stats["alive"] = alive + 1 ... asyncio.wait([ fetch_site(site) for site in sites ])
  23. Andrew Godwin / @andrewgodwin stats = {"alive": 0, "dead": 0}

    async def fetch_site(site): response = await client.get(site) if response.status_code < 400: alive = stats["alive"] stats["alive"] = alive + 1 ... asyncio.wait([ fetch_site(site) for site in sites ])
  24. Andrew Godwin / @andrewgodwin Bad with threads. Good with async!

    The await model brings us atomic sections for free
  25. Andrew Godwin / @andrewgodwin ready = False async def run():

    await client.get("some-url") ready = True async def notify(): while not ready: time.sleep(0.01) send_email() asyncio.wait({run, notify})
  26. Andrew Godwin / @andrewgodwin ready = False async def run():

    await client.get("some-url") ready = True async def notify(): while not ready: await time.sleep(0.01) send_email() asyncio.wait({run, notify})
  27. Andrew Godwin / @andrewgodwin ready = False async def run():

    await client.get("some-url") ready = True async def notify(): while not ready: await asyncio.sleep(0.01) send_email() asyncio.wait({run, notify})
  28. Andrew Godwin / @andrewgodwin This maybe is a Python thing

    We can't have functions be both sync and async.
  29. Andrew Godwin / @andrewgodwin @sync_to_async def setup_conn(): connection.execute("SET SESSION CHARACTERISTICS…")

    @sync_to_async def deactivate_users(): with transaction.atomic(): for user in User.objects.filter(old=True): ... async def my_view(request): await setup_conn() await deactivate_users(users)
  30. Andrew Godwin / @andrewgodwin @sync_to_async def setup_conn(): connection.execute("SET SESSION CHARACTERISTICS…")

    @sync_to_async def deactivate_users(): with transaction.atomic(): for user in User.objects.filter(old=True): ... async def my_view(request): await setup_conn() await deactivate_users(users)
  31. Andrew Godwin / @andrewgodwin This one's on me. Fix is

    already in the works!
  32. Andrew Godwin / @andrewgodwin How do we defend against these?

    Seems awfully easy to slip up, doesn't it.
  33. Andrew Godwin / @andrewgodwin It's an env variable. Set it

    to 1! PYTHONASYNCIODEBUG
  34. Andrew Godwin / @andrewgodwin Detects slow coroutines Like ones that

    are calling synchronous code! Detects unawaited coroutines Because we all forget await sometimes Slow I/O & thread-safety too Though these are not usually your fault
  35. Andrew Godwin / @andrewgodwin SynchronousOnlyOperation Django's here to save you

    from the worst errors, where we can
  36. Andrew Godwin / @andrewgodwin Again, async programming is hard This

    is not just Python!
  37. Andrew Godwin / @andrewgodwin Write things sync first, with tests

    As they say, "make it work, then make it fast"
  38. Andrew Godwin / @andrewgodwin Django will do our part -

    where we can We want to provide a place where you can quickly & safely add async.
  39. Andrew Godwin / @andrewgodwin We'll get better patterns! Until then,

    be safe out there
  40. Thanks. Andrew Godwin @andrewgodwin // aeracode.org