Slide 1

Slide 1 text

TAKING Andrew Godwin @andrewgodwin DJANGO ASYNC

Slide 2

Slide 2 text

Hi, I’m Andrew Godwin • Django core team member • Senior Software Engineer at • Does network programming for "fun"

Slide 3

Slide 3 text

2015: Channels 0.1

Slide 4

Slide 4 text

Adds async protocol support to Django Headline protocol is WebSockets Also allows background jobs

Slide 5

Slide 5 text

2017: Channels 1.0

Slide 6

Slide 6 text

Python 2.7 compatible Twisted for webserver Synchronous Django code

Slide 7

Slide 7 text

Webserver (Twisted) Django (Sync worker process) Client Channel Layer (Redis or other)

Slide 8

Slide 8 text

Webserver (Twisted) Django (Sync worker process) Clients Channel Layer (Redis or other) Webserver (Twisted) Django (Sync worker process)

Slide 9

Slide 9 text

Too many moving pieces No asyncio support Easy to shoot yourself in the foot

Slide 10

Slide 10 text

I was wrong.

Slide 11

Slide 11 text

2018: Channels 2.0

Slide 12

Slide 12 text

Asyncio-native Python 3.5 and up Supports async coroutines and sync threads

Slide 13

Slide 13 text

Webserver (asyncio-native) Django (coroutines/threads) Client

Slide 14

Slide 14 text

Webserver (Twisted) Django (Sync worker process) Clients Webserver (Twisted) Django (Sync worker process) Channel Layer (Redis or other)

Slide 15

Slide 15 text

75% rewrite

Slide 16

Slide 16 text

Async interfaces are separate to sync You can't provide both through one API

Slide 17

Slide 17 text

bit.ly/py-async-simp https://www.aeracode.org/2018/02/19/python-async-simplified/

Slide 18

Slide 18 text

Django had to be partially async

Slide 19

Slide 19 text

WSGI Handler (WSGI-to-request translator) URL Routing (path to view mapping) Django Middleware (auth, sessions, etc) Views (logic and presentation) ORM (database kerfuffling)

Slide 20

Slide 20 text

ASGI Handler (ASGI-to-request translator) URL Routing (path to view mapping) Django Middleware (auth, sessions, etc.) Views (logic and presentation) ORM (database kerfuffling)

Slide 21

Slide 21 text

ASGI Handler (ASGI-to-request translator) URL Routing (path to view mapping) Django Middleware (auth, sessions, etc.) Views (logic and presentation) ORM (database kerfuffling) ASGI Routing (path/protocol/etc. mapping) ASGI Middleware (auth, sessions, etc.) Consumers (logic and presentation) synchronous

Slide 22

Slide 22 text

Async-native most of the way

Slide 23

Slide 23 text

But how?

Slide 24

Slide 24 text

sync_to_async async_to_sync

Slide 25

Slide 25 text

sync_to_async async_to_sync

Slide 26

Slide 26 text

Synchronous code has to run in threads ThreadPoolExecutor does most of the hard work

Slide 27

Slide 27 text

loop = asyncio.get_event_loop() future = loop.run_in_executor( self.threadpool, func, ) return await future

Slide 28

Slide 28 text

Calling the ORM Rendering templates Handing off to Django views

Slide 29

Slide 29 text

sync_to_async async_to_sync

Slide 30

Slide 30 text

Async code runs on the event loop We need to go find it! Or make our own.

Slide 31

Slide 31 text

# Make a future for the return information call_result = Future() # Use call_soon_threadsafe to schedule a synchronous callback on the # main event loop's thread if not (self.main_event_loop and self.main_event_loop.is_running()): # Make our own event loop and run inside that. loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) try: loop.run_until_complete(self.main_wrap(args, kwargs, call_result)) finally: try: if hasattr(loop, "shutdown_asyncgens"): loop.run_until_complete(loop.shutdown_asyncgens()) finally: loop.close() asyncio.set_event_loop(self.main_event_loop) else: self.main_event_loop.call_soon_threadsafe( self.main_event_loop.create_task, self.main_wrap( args, kwargs, call_result, ), ) # Wait for results from the future. return call_result.result()

Slide 32

Slide 32 text

Make a Future Jump to the main thread and add the coroutine Tie the coroutine's end to triggering the Future Block the thread on the Future

Slide 33

Slide 33 text

Lets us provide async-native APIs And still let synchronous code use them

Slide 34

Slide 34 text

Both async and sync code are useful Channels lets you write your code as both

Slide 35

Slide 35 text

ASGI Handler (ASGI-to-request translator) URL Routing (path to view mapping) Django Middleware (auth, sessions, etc.) Views (logic and presentation) ORM (database kerfuffling) ASGI Routing (path/protocol/etc. mapping) ASGI Middleware (auth, sessions, etc.) Consumers (logic and presentation) synchronous

Slide 36

Slide 36 text

How to make WSGI async?

Slide 37

Slide 37 text

ASGI: It's like WSGI but with an A in it Also asyncio and stuff like that

Slide 38

Slide 38 text

def application(environ, start_response): ... start_response(...) yield data

Slide 39

Slide 39 text

class Application: def __init__(self, scope): ... async def __call__(self, receive, send): event = await receive() ... await send(data)

Slide 40

Slide 40 text

"Turtles all the way down" Routing is an ASGI app. Middleware is too.

Slide 41

Slide 41 text

What does this mean for Django?

Slide 42

Slide 42 text

How much can we make async? Can we keep sync compatability?

Slide 43

Slide 43 text

What does an async ORM look like? Is it even a sensible endeavour?

Slide 44

Slide 44 text

Can we make Django more pluggable? Finally get to that shareable-middleware goal

Slide 45

Slide 45 text

Do we need to replace WSGI? Or does nobody want long-polling or websockets?

Slide 46

Slide 46 text

Multiple servers (daphne, uvicorn) Multiple frameworks (Django, …?)

Slide 47

Slide 47 text

Do we want to have everyone writing async? What's the balance? How do we be flexible enough?

Slide 48

Slide 48 text

What is Django?

Slide 49

Slide 49 text

Thanks. Andrew Godwin @andrewgodwin aeracode.org