Django, Channels, and Distributed Systems

Django, Channels, and Distributed Systems

A talk I gave at PyBay 2016

077e9a0cb34fa3eba2699240c9509717?s=128

Andrew Godwin

August 21, 2016
Tweet

Transcript

  1. None
  2. Andrew Godwin Hi, I'm Django core developer Senior Software Engineer

    at Used to complain about migrations a lot
  3. Channels

  4. Born from WebSockets

  5. Born from WebSockets Expanded to be more

  6. The "real hard problem"

  7. Asynchronous coordination

  8. You take a request... ...and return a response.

  9. HTTP 1 request response Browser Server request response request response

    request response
  10. HTTP 2 request response Browser Server request response request 1

    response 2 request 2 response 1
  11. WebSockets receive send Browser Server send receive send send receive

  12. ???

  13. Server Client 1 Client 2 Client 3 Client 4

  14. Server Client 1 Client 2 Client 3 Client 4 Server

  15. Server Client 1 Client 2 Client 3 Client 4 Server

    The "hard problem"
  16. Broadcast

  17. We need to coordinate between servers

  18. Channels is a foundation for runnng 'async' at scale

  19. Architecture is about tradeoffs

  20. At most once / At least once Ordered / Unordered

    FIFO / FILO Expiry / Persistence
  21. WebSockets Service-oriented Architecture Chat/Email integration IoT protocols

  22. What makes it hard?

  23. DISTRIBUTED SYSTEMS

  24. Stateful connections Internal network Bottlenecks Machines dying

  25. Server Server Django Script ASGI "Send to channel X" "Receive

    from channel X" "Send to group Y" "Add channel X to group Y" "Remove channel X from group Y"
  26. Server Server Django Script ASGI ASGI ASGI ASGI Redis

  27. Server Server Django Script ASGI ASGI ASGI ASGI Shared Memory

  28. Server Server Django Script ASGI ASGI ASGI ASGI Shared Memory

    Redis Redis
  29. bit.ly/asgi-spec

  30. Channels wraps the low-level ASGI operations

  31. Think of it as the "Django bit" of a larger

    whole.
  32. Daphne HTTP/WebSocket Server Channels Django integration asgi-redis Redis backend asgi-ipc

    Local memory backend asgiref Shared code and libs
  33. What does Channels provide? Routing Consumers Sessions Auth Helpers By

    channel, URL, etc. Standardised message handling Cross-network persistence on sockets Including HTTP cookies on WebSocket runserver, runworker, debugging info
  34. Putting it to use Let's make a chat!

  35. Consumers def on_connect(message): Group("chat").add(message.reply_channel) def on_receive(message): Group("chat").send({"text": message["text"]}) def on_disconnect(message):

    Group("chat").discard(message.reply_channel) websocket.connect websocket.receive websocket.disconnect
  36. Routing from channels import route routing = [ route("websocket.connect", on_connect),

    route("websocket.receive", on_receive), route("websocket.disconnect", on_disconnect) ]
  37. Class-based from channels import route_class routing = [ route_class(ChatConsumer), ]

    from channels.generic.websockets class ChatConsumer(WebsocketConsumer): def connection_groups(self): return ["chat"] def receive(self, text): self.group_send("chat", text=text) Routing
  38. Full worked example github.com/andrewgodwin/channels-examples

  39. Ignoring Django

  40. 1. Take a channel layer daphne myproject.asgi:channel_layer

  41. 2. Tie it into an event loop Twisted, asyncio, or

    while-True
  42. 3. Call send/receive It's a communication channel!

  43. Example: SOA Services receive()-block waiting for tasks Clients use send()

    with a reply-channel to call an endpoint Servers process and send() the reply
  44. IRC Worker ASGI Worker Worker Email MQTT Scheduler HTTP/WS Custom

    Daemon
  45. Channels is a tool for you to use

  46. There's more to be done (and some funding for it)

  47. The tradeoffs may not be for you! (Especially as you

    specialise)
  48. 1995 You are a desktop app 2005 You are a

    website 2015 You are a rich web/mobile app 2025 ?
  49. What are the goals of a framework? Do we adapt?

  50. Thanks. Andrew Godwin @andrewgodwin channels.readthedocs.io github.com/andrewgodwin/channels-examples