Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Django, Channels, and Distributed Systems

Django, Channels, and Distributed Systems

A talk I gave at PyBay 2016

Andrew Godwin

August 21, 2016
Tweet

More Decks by Andrew Godwin

Other Decks in Programming

Transcript

  1. View Slide

  2. Andrew Godwin
    Hi, I'm
    Django core developer
    Senior Software Engineer at
    Used to complain about migrations a lot

    View Slide

  3. Channels

    View Slide

  4. Born from WebSockets

    View Slide

  5. Born from WebSockets
    Expanded to be more

    View Slide

  6. The "real hard problem"

    View Slide

  7. Asynchronous coordination

    View Slide

  8. You take a request...
    ...and return a response.

    View Slide

  9. HTTP 1
    request
    response
    Browser Server
    request
    response
    request
    response
    request
    response

    View Slide

  10. HTTP 2
    request
    response
    Browser Server
    request
    response
    request 1
    response 2
    request 2
    response 1

    View Slide

  11. WebSockets
    receive
    send
    Browser Server
    send
    receive
    send
    send
    receive

    View Slide

  12. ???

    View Slide

  13. Server
    Client 1
    Client 2
    Client 3
    Client 4

    View Slide

  14. Server
    Client 1
    Client 2
    Client 3
    Client 4
    Server

    View Slide

  15. Server
    Client 1
    Client 2
    Client 3
    Client 4
    Server
    The "hard problem"

    View Slide

  16. Broadcast

    View Slide

  17. We need to coordinate
    between servers

    View Slide

  18. Channels is a foundation
    for runnng 'async' at scale

    View Slide

  19. Architecture is
    about tradeoffs

    View Slide

  20. At most once / At least once
    Ordered / Unordered
    FIFO / FILO
    Expiry / Persistence

    View Slide

  21. WebSockets
    Service-oriented Architecture
    Chat/Email integration
    IoT protocols

    View Slide

  22. What makes it hard?

    View Slide

  23. DISTRIBUTED SYSTEMS

    View Slide

  24. Stateful connections
    Internal network
    Bottlenecks
    Machines dying

    View Slide

  25. Server Server Django Script
    ASGI
    "Send to channel X"
    "Receive from channel X"
    "Send to group Y"
    "Add channel X to group Y"
    "Remove channel X from group Y"

    View Slide

  26. Server Server Django Script
    ASGI ASGI ASGI ASGI
    Redis

    View Slide

  27. Server Server Django Script
    ASGI ASGI ASGI ASGI
    Shared
    Memory

    View Slide

  28. Server Server Django Script
    ASGI ASGI ASGI ASGI
    Shared
    Memory
    Redis Redis

    View Slide

  29. bit.ly/asgi-spec

    View Slide

  30. Channels wraps the low-level
    ASGI operations

    View Slide

  31. Think of it as the "Django bit"
    of a larger whole.

    View Slide

  32. Daphne
    HTTP/WebSocket Server
    Channels
    Django integration
    asgi-redis
    Redis backend
    asgi-ipc
    Local memory backend
    asgiref
    Shared code and libs

    View Slide

  33. What does Channels provide?
    Routing
    Consumers
    Sessions
    Auth
    Helpers
    By channel, URL, etc.
    Standardised message handling
    Cross-network persistence on sockets
    Including HTTP cookies on WebSocket
    runserver, runworker, debugging info

    View Slide

  34. Putting it to use
    Let's make a chat!

    View Slide

  35. Consumers
    def on_connect(message):
    Group("chat").add(message.reply_channel)
    def on_receive(message):
    Group("chat").send({"text": message["text"]})
    def on_disconnect(message):
    Group("chat").discard(message.reply_channel)
    websocket.connect
    websocket.receive
    websocket.disconnect

    View Slide

  36. Routing
    from channels import route
    routing = [
    route("websocket.connect", on_connect),
    route("websocket.receive", on_receive),
    route("websocket.disconnect", on_disconnect)
    ]

    View Slide

  37. Class-based
    from channels import route_class
    routing = [
    route_class(ChatConsumer),
    ]
    from channels.generic.websockets
    class ChatConsumer(WebsocketConsumer):
    def connection_groups(self):
    return ["chat"]
    def receive(self, text):
    self.group_send("chat", text=text)
    Routing

    View Slide

  38. Full worked example
    github.com/andrewgodwin/channels-examples

    View Slide

  39. Ignoring Django

    View Slide

  40. 1. Take a channel layer
    daphne myproject.asgi:channel_layer

    View Slide

  41. 2. Tie it into an event loop
    Twisted, asyncio, or while-True

    View Slide

  42. 3. Call send/receive
    It's a communication channel!

    View Slide

  43. Example: SOA
    Services receive()-block waiting for tasks
    Clients use send() with a reply-channel
    to call an endpoint
    Servers process and send() the reply

    View Slide

  44. IRC
    Worker
    ASGI Worker
    Worker
    Email
    MQTT
    Scheduler
    HTTP/WS
    Custom
    Daemon

    View Slide

  45. Channels is a tool
    for you to use

    View Slide

  46. There's more to be done
    (and some funding for it)

    View Slide

  47. The tradeoffs may not be for you!
    (Especially as you specialise)

    View Slide

  48. 1995 You are a desktop app
    2005 You are a website
    2015 You are a rich web/mobile app
    2025 ?

    View Slide

  49. What are the goals of a framework?
    Do we adapt?

    View Slide

  50. Thanks.
    Andrew Godwin
    @andrewgodwin
    channels.readthedocs.io
    github.com/andrewgodwin/channels-examples

    View Slide