How To Break Django: With Async

A talk I gave at DjangoCon Europe 2020.

Transcript

1. HOW TO
   BREAK DJANGO:
   ANDREW GODWIN // @andrewgodwin
   WITH
   ASYNC
   

   View Slide

2. Andrew Godwin / @andrewgodwin
   Hi, I’m
   Andrew Godwin
   • Django core developer
   • Worked on Migrations, Channels & Async
   • Not currently in Europe
   

   View Slide

3. Andrew Godwin / @andrewgodwin
   

   View Slide

4. Andrew Godwin / @andrewgodwin
   Async views are released!
   Find them in a Django 3.1 near you.
   

   View Slide

5. Andrew Godwin / @andrewgodwin
   WSGIHandler
   __call__
   WSGI Server
   WSGIRequest URLs Middleware
   View
   __call__
   ASGIHandler
   __call__
   ASGI Server
   ASGIRequest
   Asynchronous
   request path
   BaseHandler
   get_response_async
   BaseHandler
   get_response
   URLs Middleware
   Async View
   __call__
   Implemented async request flow
   

   View Slide

6. Andrew Godwin / @andrewgodwin
   Synchronous programming is quite safe
   Sometimes that's worth the performance cost
   

   View Slide

7. Andrew Godwin / @andrewgodwin
   Async programming is hard
   But it can definitely be worth it!
   

   View Slide

8. Andrew Godwin / @andrewgodwin
   I love learning by example
   … so please don't do any of these things.
   

   View Slide

9. Andrew Godwin / @andrewgodwin
   async def my_view(request, book_id):
   book = Book.objects.get(pk=book_id)
   return render(
   request,
   "book.html",
   {"book": book}
   )
   

   View Slide

10. Andrew Godwin / @andrewgodwin
    async def my_view(request, book_id):
    book = await Book.objects.get_async(
    pk=book_id
    )
    return render(
    request,
    "book.html",
    {"book": book}
    )
    

    View Slide

11. Andrew Godwin / @andrewgodwin
    SynchronousOnlyOperation: You cannot call
    this from an async context - use a thread or
    sync_to_async
    

    View Slide

12. Andrew Godwin / @andrewgodwin
    async def my_view(request, book_id):
    book = Book.objects.get_async(
    pk=book_id
    )
    return render(
    request,
    "book.html",
    {"book": book}
    )
    

    View Slide

13. Andrew Godwin / @andrewgodwin
    Django's got your back
    ...this time, anyway.
    

    View Slide

14. Andrew Godwin / @andrewgodwin
    async def my_view(request):
    ...
    asyncio.wait({
    create_user_account(),
    send_created_email(),
    })
    ...
    

    View Slide

15. Andrew Godwin / @andrewgodwin
    There's no ordering guarantee!
    Or even a guarantee it will execute.
    

    View Slide

16. Andrew Godwin / @andrewgodwin
    async def my_view(request):
    ...
    await create_user_account()
    await send_created_email()
    ...
    

    View Slide

17. Andrew Godwin / @andrewgodwin
    Race conditions!
    Names are power - now you can research it!
    

    View Slide

18. Andrew Godwin / @andrewgodwin
    # This is how you need to do DB access in 3.1
    @sync_to_async
    def create_user_account():
    User.objects.create(...)
    

    View Slide

19. Andrew Godwin / @andrewgodwin
    async def my_view(request):
    with transaction.atomic():
    await verify_valid_user(...)
    await create_user_account(...)
    

    View Slide

20. Andrew Godwin / @andrewgodwin
    async def my_view(request):
    with transaction.atomic():
    await verify_valid_user(...)
    await create_user_account(...)
    Runs in async thread
    In different, sync thread
    

    View Slide

21. Andrew Godwin / @andrewgodwin
    You don't actually have a transaction.
    We need to address this when we make the ORM async-aware!
    

    View Slide

22. Andrew Godwin / @andrewgodwin
    stats = {"alive": 0, "dead": 0}
    async def fetch_site(site):
    alive = stats["alive"]
    response = await client.get(site)
    if response.status_code < 400:
    stats["alive"] = alive + 1
    ...
    asyncio.wait([
    fetch_site(site) for site in sites
    ])
    

    View Slide

23. Andrew Godwin / @andrewgodwin
    stats = {"alive": 0, "dead": 0}
    async def fetch_site(site):
    response = await client.get(site)
    if response.status_code < 400:
    alive = stats["alive"]
    stats["alive"] = alive + 1
    ...
    asyncio.wait([
    fetch_site(site) for site in sites
    ])
    

    View Slide

24. Andrew Godwin / @andrewgodwin
    Bad with threads. Good with async!
    The await model brings us atomic sections for free
    

    View Slide

25. Andrew Godwin / @andrewgodwin
    ready = False
    async def run():
    await client.get("some-url")
    ready = True
    async def notify():
    while not ready:
    time.sleep(0.01)
    send_email()
    asyncio.wait({run, notify})
    

    View Slide

26. Andrew Godwin / @andrewgodwin
    ready = False
    async def run():
    await client.get("some-url")
    ready = True
    async def notify():
    while not ready:
    await time.sleep(0.01)
    send_email()
    asyncio.wait({run, notify})
    

    View Slide

27. Andrew Godwin / @andrewgodwin
    ready = False
    async def run():
    await client.get("some-url")
    ready = True
    async def notify():
    while not ready:
    await asyncio.sleep(0.01)
    send_email()
    asyncio.wait({run, notify})
    

    View Slide

28. Andrew Godwin / @andrewgodwin
    This maybe is a Python thing
    We can't have functions be both sync and async.
    

    View Slide

29. Andrew Godwin / @andrewgodwin
    @sync_to_async
    def setup_conn():
    connection.execute("SET SESSION CHARACTERISTICS…")
    @sync_to_async
    def deactivate_users():
    with transaction.atomic():
    for user in User.objects.filter(old=True):
    ...
    async def my_view(request):
    await setup_conn()
    await deactivate_users(users)
    

    View Slide

30. Andrew Godwin / @andrewgodwin
    @sync_to_async
    def setup_conn():
    connection.execute("SET SESSION CHARACTERISTICS…")
    @sync_to_async
    def deactivate_users():
    with transaction.atomic():
    for user in User.objects.filter(old=True):
    ...
    async def my_view(request):
    await setup_conn()
    await deactivate_users(users)
    

    View Slide

31. Andrew Godwin / @andrewgodwin
    This one's on me.
    Fix is already in the works!
    

    View Slide

32. Andrew Godwin / @andrewgodwin
    How do we defend against these?
    Seems awfully easy to slip up, doesn't it.
    

    View Slide

33. Andrew Godwin / @andrewgodwin
    It's an env variable. Set it to 1!
    PYTHONASYNCIODEBUG
    

    View Slide

34. Andrew Godwin / @andrewgodwin
    Detects slow coroutines
    Like ones that are calling synchronous code!
    Detects unawaited coroutines
    Because we all forget await sometimes
    Slow I/O & thread-safety too
    Though these are not usually your fault
    

    View Slide

35. Andrew Godwin / @andrewgodwin
    SynchronousOnlyOperation
    Django's here to save you from the worst errors, where we can
    

    View Slide

36. Andrew Godwin / @andrewgodwin
    Again, async programming is hard
    This is not just Python!
    

    View Slide

37. Andrew Godwin / @andrewgodwin
    Write things sync first, with tests
    As they say, "make it work, then make it fast"
    

    View Slide

38. Andrew Godwin / @andrewgodwin
    Django will do our part - where we can
    We want to provide a place where you can quickly & safely add async.
    

    View Slide

39. Andrew Godwin / @andrewgodwin
    We'll get better patterns!
    Until then, be safe out there
    

    View Slide

40. Thanks.
    Andrew Godwin
    @andrewgodwin // aeracode.org
    

    View Slide