Andrew Godwin - Reinventing Django for the Real-Time Web

Django has long been tied to the request-response pattern, but the upcoming "channels" project changes this and allows Django to natively support WebSockets, running tasks after responses, easily handle long-polling and more. Come and learn about the design, how we're trying to keep things as Django-like as possible, and how you can use it in your projects.

https://us.pycon.org/2016/schedule/presentation/1820/

Transcript

1. Andrew Godwin
   @andrewgodwin
   Reinventing
   REAL-TIME WEB
   for the
   

   View Slide

2. Andrew Godwin
   Hi, I'm
   Django core developer
   Senior Software Engineer at
   Likes complaining about networking
   

   View Slide

3. Django, HTTP and WSGI
   

   View Slide

4. You take a request...
   ...and return a response.
   

   View Slide

5. Browser
   HTTP
   Webserver
   Django
   WSGI
   View
   Handler
   

   View Slide

6. HTTP 1
   request
   response
   Browser Server
   request
   response
   request
   response
   request
   response
   

   View Slide

7. HTTP 2
   request
   response
   Browser Server
   request
   response
   request 1
   response 2
   request 2
   response 1
   

   View Slide

8. WebSockets
   receive
   send
   Browser Server
   send
   receive
   send
   send
   receive
   

   View Slide

9. ???
   

   View Slide

10. Easy to use
    Secure by default
    Hard to break/deadlock
    Python 2 & 3 compatible
    Optional
    

    View Slide

11. Python Concurrency &
    Message-passing
    Django not being async
    

    View Slide

12. WSGI
    Server
    HTTP WSGI
    WSGI
    App
    WSGI
    Server
    HTTP WSGI
    WSGI
    App
    Process 1
    Process 2
    

    View Slide

13. Worker
    Protocol
    Server
    WebSocket
    Process 1
    Worker
    Worker
    Worker
    Protocol
    Server
    WebSocket
    Process 2
    Worker
    

    View Slide

14. Protocol
    Server
    WebSocket
    Process 1
    Worker
    Protocol
    Server
    WebSocket
    Process 2
    

    View Slide

15. Channel
    Layer
    Interface
    Server
    Worker
    Server
    Process 1
    ASGI ASGI
    Asynchronous
    socket handling
    Synchronous
    Django project
    Interface
    Server
    Worker
    Server
    ASGI ASGI
    Worker
    Server
    ASGI
    Process 2 Process 3
    Process 4
    

    View Slide

16. Channel Layer
    Worker
    Server
    Interface
    Server
    Channel Layer
    Worker
    Server
    Interface
    Server
    Channel Layer
    Worker
    Server
    Interface
    Server
    Channel Layer
    Worker 1 Worker 2 Worker 3
    Redis
    

    View Slide

17. Groups
    Channels
    Named FIFO task queues
    Named sets of channels you broadcast to
    

    View Slide

18. send("channel_name", {"ponies": True})
    receive_many(["channel_name", "channel_two"])
    send_group("group_name", {"ponies": True})
    group_add("group_name", "channel_name")
    group_discard("group_name", "channel_name")
    

    View Slide

19. You take a request...
    ...and return a response.
    

    View Slide

20. You receive a message...
    ...and send zero or more messages.
    

    View Slide

21. View
    Request
    Response
    Consumer
    Message
    Messages
    

    View Slide

22. Consumer
    websocket.connect
    Consumer
    websocket.receive
    websocket.send
    

    View Slide

23. Consumer
    websocket.connect
    Consumer
    websocket.receive
    websocket.send!jq3x41
    

    View Slide

24. Consumer
    http.request
    http.response!a34cxw
    

    View Slide

25. websocket.receive
    Worker
    Worker
    Worker
    websocket.send!a1b2c3 Socket
    

    View Slide

26. Receive events from channels,
    and send events to them/groups.
    WebSocket/HTTP messages come
    with a reply_channel
    

    View Slide

27. Using it
    

    View Slide

28. Installation
    pip install channels
    Add channels to INSTALLED_APPS
    

    View Slide

29. Liveblog
    We want people to get new blog posts as they
    are published, without refreshing
    

    View Slide

30. Liveblog
    People open a WebSocket when they open the page
    Their WebSocket is added to a group
    When the BlogPost model is saved, we send the
    post to that group
    

    View Slide

31. Their WebSocket is added to a group
    When the BlogPost model is saved, we send the
    post to that group
    def ws_connect(message):
    Group("liveblog").add(message.reply_channel)
    class BlogPost(models.Model):
    ...
    def save(self, *args, **kwargs):
    ...
    Group("liveblog").send({
    "text": json.dumps({"id": self.id}),
    })
    

    View Slide

32. Chat
    People can send messages, and they
    get sent to everyone connected.
    

    View Slide

33. Chat
    When people connect they join a chat group
    When we receive a message we send it to the group
    

    View Slide

34. When people connect they join a chat group
    def ws_connect(message):
    Group("chat").add(message.reply_channel)
    When we receive a message we send it to the group
    def ws_receive(message):
    Group("chat").send({
    "text": message["text"],
    })
    

    View Slide

35. ...and we tell Django what consumers are
    joined to which actions.
    # in routing.py
    routing = [
    route("websocket.connect", consumers.ws_connect),
    route("websocket.receive", consumers.ws_receive),
    ]
    ...and we tell Django what consumers are
    joined to which actions.
    

    View Slide

36. Important notes
    Runserver just works with WebSockets now
    Generic Consumers exist
    Fully worked versions of these two are at
    github.com/andrewgodwin/channels-examples
    Django sessions + auth work with WebSockets
    

    View Slide

37. FIFO queue with send and
    receive_many operations,
    named with a string.
    Channel
    Group Named set of channels with
    add/remove/send operations
    Cross-process
    Messages Representations for HTTP
    and WebSocket sessions
    

    View Slide

38. ASGI
    API specification for
    channel layer backends
    Message formats for
    HTTP and WebSocket
    &
    

    View Slide

39. Redis
    POSIX IPC
    In-memory
    Reference network layer
    For single-machine installs
    For testing or single-process installs
    

    View Slide

40. Channel
    Layer
    Interface
    Server
    Worker
    Server
    Process 1
    ASGI ASGI
    Asynchronous
    socket handling
    Synchronous
    Django project
    Interface
    Server
    Worker
    Server
    ASGI ASGI
    Worker
    Server
    ASGI
    Process 2 Process 3
    Process 4
    

    View Slide

41. Channel
    Layer
    Interface
    Server
    Worker
    Server
    Channel
    Layer
    Daphne
    HTTP + WS
    WSGI Adapter
    HTTP
    asgi_redis
    Cross-network, shards
    asgi_ipc
    Single-machine
    Django
    Consumer system
    WSGI Adapter
    Most WSGI apps
    

    View Slide

42. WSGI and/or ASGI
    An ASGI system can serve HTTP and WebSocket
    ...or your WSGI system can send onto channels
    

    View Slide

43. Channel Layer
    Worker
    Server
    Interface
    Server
    Channel Layer
    Worker
    Server
    Interface
    Server
    WSGI
    App
    WSGI
    Server
    Channel Layer
    Worker 1 Worker 2 Worker 3
    Redis
    

    View Slide

44. Scaling?
    

    View Slide

45. Interface servers scale horizontally
    Worker servers scale horizontally
    Channel layer has to as well
    

    View Slide

46. Consistent hash sharding designed in
    Other approaches possible
    

    View Slide

47. Looking Ahead
    

    View Slide

48. Being Django
    Official external app
    Merge in 1.11 or 2.0
    

    View Slide

49. Maturity
    Load tests and tweaking
    Learning from production installs
    

    View Slide

50. Community
    Third-party tools and extensions
    Tutorials, write-ups and case studies
    

    View Slide

51. Co-existence
    Not everyone needs WebSockets!
    Make sure it's easy to add or remove.
    

    View Slide

52. Expansion
    Email, chat and other message formats
    Add more events to code against!
    

    View Slide

53. Specification
    ASGI is not Django-specific
    WSGI needs supplementing for new protocols
    

    View Slide

54. channels.readthedocs.io
    github.com/andrewgodwin/channels-examples
    

    View Slide

55. Thanks.
    Andrew Godwin
    @andrewgodwin
    

    View Slide