Reinventing Django for the Real-Time Web

Reinventing Django for the Real-Time Web

A talk I gave at PyCon US 2016.

077e9a0cb34fa3eba2699240c9509717?s=128

Andrew Godwin

May 30, 2016
Tweet

Transcript

  1. Andrew Godwin @andrewgodwin Reinventing REAL-TIME WEB for the

  2. Andrew Godwin Hi, I'm Django core developer Senior Software Engineer

    at Likes complaining about networking
  3. Django, HTTP and WSGI

  4. You take a request... ...and return a response.

  5. Browser HTTP Webserver Django WSGI View Handler

  6. HTTP 1 request response Browser Server request response request response

    request response
  7. HTTP 2 request response Browser Server request response request 1

    response 2 request 2 response 1
  8. WebSockets receive send Browser Server send receive send send receive

  9. ???

  10. Easy to use Secure by default Hard to break/deadlock Python

    2 & 3 compatible Optional
  11. Python Concurrency & Message-passing Django not being async

  12. WSGI Server HTTP WSGI WSGI App WSGI Server HTTP WSGI

    WSGI App Process 1 Process 2
  13. Worker Protocol Server WebSocket Process 1 Worker Worker Worker Protocol

    Server WebSocket Process 2 Worker
  14. Protocol Server WebSocket Process 1 Worker Protocol Server WebSocket Process

    2
  15. Channel Layer Interface Server Worker Server Process 1 ASGI ASGI

    Asynchronous socket handling Synchronous Django project Interface Server Worker Server ASGI ASGI Worker Server ASGI Process 2 Process 3 Process 4
  16. Channel Layer Worker Server Interface Server Channel Layer Worker Server

    Interface Server Channel Layer Worker Server Interface Server Channel Layer Worker 1 Worker 2 Worker 3 Redis
  17. Groups Channels Named FIFO task queues Named sets of channels

    you broadcast to
  18. send("channel_name", {"ponies": True}) receive_many(["channel_name", "channel_two"]) send_group("group_name", {"ponies": True}) group_add("group_name", "channel_name")

    group_discard("group_name", "channel_name")
  19. You take a request... ...and return a response.

  20. You receive a message... ...and send zero or more messages.

  21. View Request Response Consumer Message Messages

  22. Consumer websocket.connect Consumer websocket.receive websocket.send

  23. Consumer websocket.connect Consumer websocket.receive websocket.send!jq3x41

  24. Consumer http.request http.response!a34cxw

  25. websocket.receive Worker Worker Worker websocket.send!a1b2c3 Socket

  26. Receive events from channels, and send events to them/groups. WebSocket/HTTP

    messages come with a reply_channel
  27. Using it

  28. Installation pip install channels Add channels to INSTALLED_APPS

  29. Liveblog We want people to get new blog posts as

    they are published, without refreshing
  30. Liveblog People open a WebSocket when they open the page

    Their WebSocket is added to a group When the BlogPost model is saved, we send the post to that group
  31. Their WebSocket is added to a group When the BlogPost

    model is saved, we send the post to that group def ws_connect(message): Group("liveblog").add(message.reply_channel) class BlogPost(models.Model): ... def save(self, *args, **kwargs): ... Group("liveblog").send({ "text": json.dumps({"id": self.id}), })
  32. Chat People can send messages, and they get sent to

    everyone connected.
  33. Chat When people connect they join a chat group When

    we receive a message we send it to the group
  34. When people connect they join a chat group def ws_connect(message):

    Group("chat").add(message.reply_channel) When we receive a message we send it to the group def ws_receive(message): Group("chat").send({ "text": message["text"], })
  35. ...and we tell Django what consumers are joined to which

    actions. # in routing.py routing = [ route("websocket.connect", consumers.ws_connect), route("websocket.receive", consumers.ws_receive), ] ...and we tell Django what consumers are joined to which actions.
  36. Important notes Runserver just works with WebSockets now Generic Consumers

    exist Fully worked versions of these two are at github.com/andrewgodwin/channels-examples Django sessions + auth work with WebSockets
  37. FIFO queue with send and receive_many operations, named with a

    string. Channel Group Named set of channels with add/remove/send operations Cross-process Messages Representations for HTTP and WebSocket sessions
  38. ASGI API specification for channel layer backends Message formats for

    HTTP and WebSocket &
  39. Redis POSIX IPC In-memory Reference network layer For single-machine installs

    For testing or single-process installs
  40. Channel Layer Interface Server Worker Server Process 1 ASGI ASGI

    Asynchronous socket handling Synchronous Django project Interface Server Worker Server ASGI ASGI Worker Server ASGI Process 2 Process 3 Process 4
  41. Channel Layer Interface Server Worker Server Channel Layer Daphne HTTP

    + WS WSGI Adapter HTTP asgi_redis Cross-network, shards asgi_ipc Single-machine Django Consumer system WSGI Adapter Most WSGI apps
  42. WSGI and/or ASGI An ASGI system can serve HTTP and

    WebSocket ...or your WSGI system can send onto channels
  43. Channel Layer Worker Server Interface Server Channel Layer Worker Server

    Interface Server WSGI App WSGI Server Channel Layer Worker 1 Worker 2 Worker 3 Redis
  44. Scaling?

  45. Interface servers scale horizontally Worker servers scale horizontally Channel layer

    has to as well
  46. Consistent hash sharding designed in Other approaches possible

  47. Looking Ahead

  48. Being Django Official external app Merge in 1.11 or 2.0

  49. Maturity Load tests and tweaking Learning from production installs

  50. Community Third-party tools and extensions Tutorials, write-ups and case studies

  51. Co-existence Not everyone needs WebSockets! Make sure it's easy to

    add or remove.
  52. Expansion Email, chat and other message formats Add more events

    to code against!
  53. Specification ASGI is not Django-specific WSGI needs supplementing for new

    protocols
  54. channels.readthedocs.io github.com/andrewgodwin/channels-examples

  55. Thanks. Andrew Godwin @andrewgodwin