2016 - Andrew Godwin - Django, Channels and Distributed Systems

​Description: Learn about the Django Channels project, how it makes WebSockets easy, how it's not just limited to Django, and the difficulty of building WebSocket and other stateful protocol handling at scale.
​
Abstract: Django Channels' headline feature is bringing WebSocket support to Django, but what it provides is far more useful than that. Underlying it is a robust, generic cross-process communication mechanism, built to support and scale with stateful protocols like WebSockets.

This talk will look at the design of this mechanism - codenamed ASGI - and the difficulties of building an entire system to support WebSockets and broadcast systems across a large number of servers, and how Django encapsulates this to provide you a simple but powerful interface with good performance characteristics.

We'll also take a brief look at how parts of Channels are useable outside of Django with other web frameworks or pure Python code, and how it lets us build better systems overall.
​
Bio: Andrew is a Python programmer, Django core developer and Senior Engineer at Eventbrite. He's behind Django's migration and channels systems, and in his spare time enjoys mountains, archery, and cheese.

https://youtu.be/lw9NA5Ea5wA

Transcript

2. Andrew Godwin Hi, I'm Django core developer Senior Software Engineer at Used to complain about migrations a lot
3. Channels
4. Born from WebSockets
5. Born from WebSockets Expanded to be more
6. The "real hard problem"
7. Asynchronous coordination
8. You take a request... ...and return a response.
9. HTTP 1 request response Browser Server request response request response request response
10. HTTP 2 request response Browser Server request response request 1 response 2 request 2 response 1
11. WebSockets receive send Browser Server send receive send send receive
12. ???
13. Server Client 1 Client 2 Client 3 Client 4
14. Server Client 1 Client 2 Client 3 Client 4 Server
15. Server Client 1 Client 2 Client 3 Client 4 Server The "hard problem"
16. Broadcast
17. We need to coordinate between servers
18. Channels is a foundation for runnng 'async' at scale
19. Architecture is about tradeoffs
20. At most once / At least once Ordered / Unordered FIFO / FILO Expiry / Persistence
21. WebSockets Service-oriented Architecture Chat/Email integration IoT protocols
22. What makes it hard?
23. DISTRIBUTED SYSTEMS
24. Stateful connections Internal network Bottlenecks Machines dying
25. Server Server Django Script ASGI "Send to channel X" "Receive from channel X" "Send to group Y" "Add channel X to group Y" "Remove channel X from group Y"
26. Server Server Django Script ASGI ASGI ASGI ASGI Redis
27. Server Server Django Script ASGI ASGI ASGI ASGI Shared Memory
28. Server Server Django Script ASGI ASGI ASGI ASGI Shared Memory Redis Redis
29. bit.ly/asgi-spec
30. Channels wraps the low-level ASGI operations
31. Think of it as the "Django bit" of a larger whole.
32. Daphne HTTP/WebSocket Server Channels Django integration asgi-redis Redis backend asgi-ipc Local memory backend asgiref Shared code and libs
33. What does Channels provide? Routing Consumers Sessions Auth Helpers By channel, URL, etc. Standardised message handling Cross-network persistence on sockets Including HTTP cookies on WebSocket runserver, runworker, debugging info
34. Putting it to use Let's make a chat!
35. Consumers def on_connect(message): Group("chat").add(message.reply_channel) def on_receive(message): Group("chat").send({"text": message["text"]}) def on_disconnect(message): Group("chat").discard(message.reply_channel) websocket.connect websocket.receive websocket.disconnect
36. Routing from channels import route routing = [ route("websocket.connect", on_connect), route("websocket.receive", on_receive), route("websocket.disconnect", on_disconnect) ]
37. Class-based from channels import route_class routing = [ route_class(ChatConsumer), ] from channels.generic.websockets class ChatConsumer(WebsocketConsumer): def connection_groups(self): return ["chat"] def receive(self, text): self.group_send("chat", text=text) Routing
38. Full worked example github.com/andrewgodwin/channels-examples
39. Ignoring Django
40. 1. Take a channel layer daphne myproject.asgi:channel_layer
41. 2. Tie it into an event loop Twisted, asyncio, or while-True
42. 3. Call send/receive It's a communication channel!
43. Example: SOA Services receive()-block waiting for tasks Clients use send() with a reply-channel to call an endpoint Servers process and send() the reply
44. IRC Worker ASGI Worker Worker Email MQTT Scheduler HTTP/WS Custom Daemon
45. Channels is a tool for you to use
46. There's more to be done (and some funding for it)
47. The tradeoffs may not be for you! (Especially as you specialise)
48. 1995 You are a desktop app 2005 You are a website 2015 You are a rich web/mobile app 2025 ?
49. What are the goals of a framework? Do we adapt?
50. Thanks. Andrew Godwin @andrewgodwin channels.readthedocs.io github.com/andrewgodwin/channels-examples