Reinventing Django for the Real-Time Web

A talk I gave at PyCon US 2016.

Transcript

1. Andrew Godwin @andrewgodwin Reinventing REAL-TIME WEB for the
2. Andrew Godwin Hi, I'm Django core developer Senior Software Engineer at Likes complaining about networking
3. Django, HTTP and WSGI
4. You take a request... ...and return a response.
5. Browser HTTP Webserver Django WSGI View Handler
6. HTTP 1 request response Browser Server request response request response request response
7. HTTP 2 request response Browser Server request response request 1 response 2 request 2 response 1
8. WebSockets receive send Browser Server send receive send send receive
9. ???
10. Easy to use Secure by default Hard to break/deadlock Python 2 & 3 compatible Optional
11. Python Concurrency & Message-passing Django not being async
12. WSGI Server HTTP WSGI WSGI App WSGI Server HTTP WSGI WSGI App Process 1 Process 2
13. Worker Protocol Server WebSocket Process 1 Worker Worker Worker Protocol Server WebSocket Process 2 Worker
14. Protocol Server WebSocket Process 1 Worker Protocol Server WebSocket Process 2
15. Channel Layer Interface Server Worker Server Process 1 ASGI ASGI Asynchronous socket handling Synchronous Django project Interface Server Worker Server ASGI ASGI Worker Server ASGI Process 2 Process 3 Process 4
16. Channel Layer Worker Server Interface Server Channel Layer Worker Server Interface Server Channel Layer Worker Server Interface Server Channel Layer Worker 1 Worker 2 Worker 3 Redis
17. Groups Channels Named FIFO task queues Named sets of channels you broadcast to
18. send("channel_name", {"ponies": True}) receive_many(["channel_name", "channel_two"]) send_group("group_name", {"ponies": True}) group_add("group_name", "channel_name") group_discard("group_name", "channel_name")
19. You take a request... ...and return a response.
20. You receive a message... ...and send zero or more messages.
21. View Request Response Consumer Message Messages
22. Consumer websocket.connect Consumer websocket.receive websocket.send
23. Consumer websocket.connect Consumer websocket.receive websocket.send!jq3x41
24. Consumer http.request http.response!a34cxw
25. websocket.receive Worker Worker Worker websocket.send!a1b2c3 Socket
26. Receive events from channels, and send events to them/groups. WebSocket/HTTP messages come with a reply_channel
27. Using it
28. Installation pip install channels Add channels to INSTALLED_APPS
29. Liveblog We want people to get new blog posts as they are published, without refreshing
30. Liveblog People open a WebSocket when they open the page Their WebSocket is added to a group When the BlogPost model is saved, we send the post to that group
31. Their WebSocket is added to a group When the BlogPost model is saved, we send the post to that group def ws_connect(message): Group("liveblog").add(message.reply_channel) class BlogPost(models.Model): ... def save(self, *args, **kwargs): ... Group("liveblog").send({ "text": json.dumps({"id": self.id}), })
32. Chat People can send messages, and they get sent to everyone connected.
33. Chat When people connect they join a chat group When we receive a message we send it to the group
34. When people connect they join a chat group def ws_connect(message): Group("chat").add(message.reply_channel) When we receive a message we send it to the group def ws_receive(message): Group("chat").send({ "text": message["text"], })
35. ...and we tell Django what consumers are joined to which actions. # in routing.py routing = [ route("websocket.connect", consumers.ws_connect), route("websocket.receive", consumers.ws_receive), ] ...and we tell Django what consumers are joined to which actions.
36. Important notes Runserver just works with WebSockets now Generic Consumers exist Fully worked versions of these two are at github.com/andrewgodwin/channels-examples Django sessions + auth work with WebSockets
37. FIFO queue with send and receive_many operations, named with a string. Channel Group Named set of channels with add/remove/send operations Cross-process Messages Representations for HTTP and WebSocket sessions
38. ASGI API specification for channel layer backends Message formats for HTTP and WebSocket &
39. Redis POSIX IPC In-memory Reference network layer For single-machine installs For testing or single-process installs
40. Channel Layer Interface Server Worker Server Process 1 ASGI ASGI Asynchronous socket handling Synchronous Django project Interface Server Worker Server ASGI ASGI Worker Server ASGI Process 2 Process 3 Process 4
41. Channel Layer Interface Server Worker Server Channel Layer Daphne HTTP + WS WSGI Adapter HTTP asgi_redis Cross-network, shards asgi_ipc Single-machine Django Consumer system WSGI Adapter Most WSGI apps
42. WSGI and/or ASGI An ASGI system can serve HTTP and WebSocket ...or your WSGI system can send onto channels
43. Channel Layer Worker Server Interface Server Channel Layer Worker Server Interface Server WSGI App WSGI Server Channel Layer Worker 1 Worker 2 Worker 3 Redis
44. Scaling?
45. Interface servers scale horizontally Worker servers scale horizontally Channel layer has to as well
46. Consistent hash sharding designed in Other approaches possible
47. Looking Ahead
48. Being Django Official external app Merge in 1.11 or 2.0
49. Maturity Load tests and tweaking Learning from production installs
50. Community Third-party tools and extensions Tutorials, write-ups and case studies
51. Co-existence Not everyone needs WebSockets! Make sure it's easy to add or remove.
52. Expansion Email, chat and other message formats Add more events to code against!
53. Specification ASGI is not Django-specific WSGI needs supplementing for new protocols
54. channels.readthedocs.io github.com/andrewgodwin/channels-examples
55. Thanks. Andrew Godwin @andrewgodwin