Django, Channels, and Distributed Systems

A talk I gave at PyBay 2016

Transcript

1. View Slide

2. Andrew Godwin
   Hi, I'm
   Django core developer
   Senior Software Engineer at
   Used to complain about migrations a lot
   

   View Slide

3. Channels
   

   View Slide

4. Born from WebSockets
   

   View Slide

5. Born from WebSockets
   Expanded to be more
   

   View Slide

6. The "real hard problem"
   

   View Slide

7. Asynchronous coordination
   

   View Slide

8. You take a request...
   ...and return a response.
   

   View Slide

9. HTTP 1
   request
   response
   Browser Server
   request
   response
   request
   response
   request
   response
   

   View Slide

10. HTTP 2
    request
    response
    Browser Server
    request
    response
    request 1
    response 2
    request 2
    response 1
    

    View Slide

11. WebSockets
    receive
    send
    Browser Server
    send
    receive
    send
    send
    receive
    

    View Slide

12. ???
    

    View Slide

13. Server
    Client 1
    Client 2
    Client 3
    Client 4
    

    View Slide

14. Server
    Client 1
    Client 2
    Client 3
    Client 4
    Server
    

    View Slide

15. Server
    Client 1
    Client 2
    Client 3
    Client 4
    Server
    The "hard problem"
    

    View Slide

16. Broadcast
    

    View Slide

17. We need to coordinate
    between servers
    

    View Slide

18. Channels is a foundation
    for runnng 'async' at scale
    

    View Slide

19. Architecture is
    about tradeoffs
    

    View Slide

20. At most once / At least once
    Ordered / Unordered
    FIFO / FILO
    Expiry / Persistence
    

    View Slide

21. WebSockets
    Service-oriented Architecture
    Chat/Email integration
    IoT protocols
    

    View Slide

22. What makes it hard?
    

    View Slide

23. DISTRIBUTED SYSTEMS
    

    View Slide

24. Stateful connections
    Internal network
    Bottlenecks
    Machines dying
    

    View Slide

25. Server Server Django Script
    ASGI
    "Send to channel X"
    "Receive from channel X"
    "Send to group Y"
    "Add channel X to group Y"
    "Remove channel X from group Y"
    

    View Slide

26. Server Server Django Script
    ASGI ASGI ASGI ASGI
    Redis
    

    View Slide

27. Server Server Django Script
    ASGI ASGI ASGI ASGI
    Shared
    Memory
    

    View Slide

28. Server Server Django Script
    ASGI ASGI ASGI ASGI
    Shared
    Memory
    Redis Redis
    

    View Slide

29. bit.ly/asgi-spec
    

    View Slide

30. Channels wraps the low-level
    ASGI operations
    

    View Slide

31. Think of it as the "Django bit"
    of a larger whole.
    

    View Slide

32. Daphne
    HTTP/WebSocket Server
    Channels
    Django integration
    asgi-redis
    Redis backend
    asgi-ipc
    Local memory backend
    asgiref
    Shared code and libs
    

    View Slide

33. What does Channels provide?
    Routing
    Consumers
    Sessions
    Auth
    Helpers
    By channel, URL, etc.
    Standardised message handling
    Cross-network persistence on sockets
    Including HTTP cookies on WebSocket
    runserver, runworker, debugging info
    

    View Slide

34. Putting it to use
    Let's make a chat!
    

    View Slide

35. Consumers
    def on_connect(message):
    Group("chat").add(message.reply_channel)
    def on_receive(message):
    Group("chat").send({"text": message["text"]})
    def on_disconnect(message):
    Group("chat").discard(message.reply_channel)
    websocket.connect
    websocket.receive
    websocket.disconnect
    

    View Slide

36. Routing
    from channels import route
    routing = [
    route("websocket.connect", on_connect),
    route("websocket.receive", on_receive),
    route("websocket.disconnect", on_disconnect)
    ]
    

    View Slide

37. Class-based
    from channels import route_class
    routing = [
    route_class(ChatConsumer),
    ]
    from channels.generic.websockets
    class ChatConsumer(WebsocketConsumer):
    def connection_groups(self):
    return ["chat"]
    def receive(self, text):
    self.group_send("chat", text=text)
    Routing
    

    View Slide

38. Full worked example
    github.com/andrewgodwin/channels-examples
    

    View Slide

39. Ignoring Django
    

    View Slide

40. 1. Take a channel layer
    daphne myproject.asgi:channel_layer
    

    View Slide

41. 2. Tie it into an event loop
    Twisted, asyncio, or while-True
    

    View Slide

42. 3. Call send/receive
    It's a communication channel!
    

    View Slide

43. Example: SOA
    Services receive()-block waiting for tasks
    Clients use send() with a reply-channel
    to call an endpoint
    Servers process and send() the reply
    

    View Slide

44. IRC
    Worker
    ASGI Worker
    Worker
    Email
    MQTT
    Scheduler
    HTTP/WS
    Custom
    Daemon
    

    View Slide

45. Channels is a tool
    for you to use
    

    View Slide

46. There's more to be done
    (and some funding for it)
    

    View Slide

47. The tradeoffs may not be for you!
    (Especially as you specialise)
    

    View Slide

48. 1995 You are a desktop app
    2005 You are a website
    2015 You are a rich web/mobile app
    2025 ?
    

    View Slide

49. What are the goals of a framework?
    Do we adapt?
    

    View Slide

50. Thanks.
    Andrew Godwin
    @andrewgodwin
    channels.readthedocs.io
    github.com/andrewgodwin/channels-examples
    

    View Slide