Reinventing Django for the Real-Time Web

Transcript

1. Andrew Godwin @andrewgodwin Reinventing REAL-TIME WEB for the

2. Andrew Godwin Hi, I'm Django core developer Senior Software Engineer

   at Likes complaining about networking

3. Django, HTTP and WSGI

4. You take a request... ...and return a response.

5. Browser HTTP Webserver Django WSGI View Handler

6. HTTP 1 request response Browser Server request response request response

   request response

7. HTTP 2 request response Browser Server request response request 1

   response 2 request 2 response 1

8. WebSockets receive send Browser Server send receive send send receive

9. ???

10. Easy to use Secure by default Hard to break/deadlock Python

    2 & 3 compatible Optional

11. Python Concurrency & Message-passing Django not being async

12. WSGI Server HTTP WSGI WSGI App WSGI Server HTTP WSGI

    WSGI App Process 1 Process 2

13. Worker Protocol Server WebSocket Process 1 Worker Worker Worker Protocol

    Server WebSocket Process 2 Worker

14. Protocol Server WebSocket Process 1 Worker Protocol Server WebSocket Process

    2

15. Channel Layer Interface Server Worker Server Process 1 ASGI ASGI

    Asynchronous socket handling Synchronous Django project Interface Server Worker Server ASGI ASGI Worker Server ASGI Process 2 Process 3 Process 4

16. Channel Layer Worker Server Interface Server Channel Layer Worker Server

    Interface Server Channel Layer Worker Server Interface Server Channel Layer Worker 1 Worker 2 Worker 3 Redis

17. Groups Channels Named FIFO task queues Named sets of channels

    you broadcast to

18. send("channel_name", {"ponies": True}) receive_many(["channel_name", "channel_two"]) send_group("group_name", {"ponies": True}) group_add("group_name", "channel_name")

    group_discard("group_name", "channel_name")

19. You take a request... ...and return a response.

20. You receive a message... ...and send zero or more messages.

21. View Request Response Consumer Message Messages

22. Consumer websocket.connect Consumer websocket.receive websocket.send

23. Consumer websocket.connect Consumer websocket.receive websocket.send!jq3x41

24. Consumer http.request http.response!a34cxw

25. websocket.receive Worker Worker Worker websocket.send!a1b2c3 Socket

26. Receive events from channels, and send events to them/groups. WebSocket/HTTP

    messages come with a reply_channel

27. Using it

28. Installation pip install channels Add channels to INSTALLED_APPS

29. Liveblog We want people to get new blog posts as

    they are published, without refreshing

30. Liveblog People open a WebSocket when they open the page

    Their WebSocket is added to a group When the BlogPost model is saved, we send the post to that group

31. Their WebSocket is added to a group When the BlogPost

    model is saved, we send the post to that group def ws_connect(message): Group("liveblog").add(message.reply_channel) class BlogPost(models.Model): ... def save(self, *args, **kwargs): ... Group("liveblog").send({ "text": json.dumps({"id": self.id}), })

32. Chat People can send messages, and they get sent to

    everyone connected.

33. Chat When people connect they join a chat group When

    we receive a message we send it to the group

34. When people connect they join a chat group def ws_connect(message):

    Group("chat").add(message.reply_channel) When we receive a message we send it to the group def ws_receive(message): Group("chat").send({ "text": message["text"], })

35. ...and we tell Django what consumers are joined to which

    actions. # in routing.py routing = [ route("websocket.connect", consumers.ws_connect), route("websocket.receive", consumers.ws_receive), ] ...and we tell Django what consumers are joined to which actions.

36. Important notes Runserver just works with WebSockets now Generic Consumers

    exist Fully worked versions of these two are at github.com/andrewgodwin/channels-examples Django sessions + auth work with WebSockets

37. FIFO queue with send and receive_many operations, named with a

    string. Channel Group Named set of channels with add/remove/send operations Cross-process Messages Representations for HTTP and WebSocket sessions

38. ASGI API specification for channel layer backends Message formats for

    HTTP and WebSocket &

39. Redis POSIX IPC In-memory Reference network layer For single-machine installs

    For testing or single-process installs

40. Channel Layer Interface Server Worker Server Process 1 ASGI ASGI

    Asynchronous socket handling Synchronous Django project Interface Server Worker Server ASGI ASGI Worker Server ASGI Process 2 Process 3 Process 4

41. Channel Layer Interface Server Worker Server Channel Layer Daphne HTTP

    + WS WSGI Adapter HTTP asgi_redis Cross-network, shards asgi_ipc Single-machine Django Consumer system WSGI Adapter Most WSGI apps

42. WSGI and/or ASGI An ASGI system can serve HTTP and

    WebSocket ...or your WSGI system can send onto channels

43. Channel Layer Worker Server Interface Server Channel Layer Worker Server

    Interface Server WSGI App WSGI Server Channel Layer Worker 1 Worker 2 Worker 3 Redis

44. Scaling?

45. Interface servers scale horizontally Worker servers scale horizontally Channel layer

    has to as well

46. Consistent hash sharding designed in Other approaches possible

47. Looking Ahead

48. Being Django Official external app Merge in 1.11 or 2.0

49. Maturity Load tests and tweaking Learning from production installs

50. Community Third-party tools and extensions Tutorials, write-ups and case studies

51. Co-existence Not everyone needs WebSockets! Make sure it's easy to

    add or remove.

52. Expansion Email, chat and other message formats Add more events

    to code against!

53. Specification ASGI is not Django-specific WSGI needs supplementing for new

    protocols

54. channels.readthedocs.io github.com/andrewgodwin/channels-examples

55. Thanks. Andrew Godwin @andrewgodwin