What Django Can Learn From Twisted (DjangoConAU 2015 Keynote)

What Django Can Learn From Twisted (DjangoConAU 2015 Keynote)

Web frameworks like Django are designed around the traditional request-response cycle — a request comes in, a response is generated, and that is delivered to the client. In the day of “single page” applications, where only sections of the page are updated through requests, doing real-time can be clunky. Twisted, and things that build off it, like Django Channels, might be worth thinking about, and this is what this talk will be about.

3d37232726396a1d3c7412dd915095ea?s=128

Amber Brown (HawkOwl)

July 31, 2015
Tweet

Transcript

  1. 11.

    Russell offered me this slot to talk about Twisted &

    Django, and since I kept ranting about it, why not?
  2. 16.
  3. 19.

    What is Twisted? • Asynchronous networking framework with a stable

    & mature codebase • Includes primitives for asynchronous I/O and many protocol implementations (e.g. HTTP, SSH, SMTP) • Works on Python 2.7, with base functionality on Python 3.3+
  4. 24.

    We tell it that once it has a result, run

    a callback. This time, it’ll be calling treq.content on the Deferred’s result
  5. 25.

    Then we return the Deferred. Klein will then wait for

    the Deferred to fire all its callbacks and then write the response.
  6. 26.
  7. 31.

    request -> get viewing user account -> check permissions (eg.

    blocked or not) -> get profile of target user -> render page -> respond to client
  8. 35.

    Asynchronous I/O (“non- blocking”) is when functions that do I/O

    instead give a promise of a result at some future point in time
  9. 37.

    Benefits of async • If you don’t need the result,

    just don’t wait for it (eg. incrementing a ‘pageviews’ counter) • Handling many concurrent “quiet” connections is more efficient (eg. WebSockets) • Asynchronous code is broken up over I/O barriers, making it easier to test
  10. 44.

    So, she puts the water on to boil, setting a

    timer, and while it’s boiling, she collects the ingredients.
  11. 46.

    While the eggs are cooking, she turns the oven on

    to preheat (and sets a timer), then chops some vegetables.
  12. 47.

    The egg timer goes off, so she stops chopping the

    vegetables, and takes it off the boil
  13. 48.
  14. 49.

    She serves the eggs while the vegetables are cooking, and

    returns when the timer goes off to serve them too.
  15. 51.

    Well, the chef is only one woman, but she did

    several things “at once”.
  16. 52.

    The chef is Twisted, only capable of doing one thing

    at once, but knowing that not everything needs her attention right now
  17. 59.

    Synchronous Upsides • Code flow is easier to understand —

    do x, then y • Only one “thread” of execution, for simplicity • Many libraries are synchronous
  18. 60.

    Synchronous Downsides • You can only do one thing at

    once • Although suited to the request/response cycle, it can only really do that • Persistent connections are not simple to implement
  19. 61.

    Asynchronous Upsides • Multiple “threads” of execution — the code

    handling the request doesn’t have to finish after the request is written • Handling persistent/evented connections is super easy • Reactor model async is threadless, so no thread overhead for concurrency
  20. 62.

    Asynchronous Downsides • You have to be a good citizen

    — blocking in the reactor loop is disastrous for performance • Doing I/O is “harder” because you have to be explicit about it • In Python, things need to be written to be async
  21. 65.

    Hendrix • Hendrix is a “Twisted Django” • WSGI server

    using Twisted, plus WebSockets • Multiprocessing, multithreaded • https://github.com/hangarunderground/hendrix
  22. 66.

    Crochet • Run Twisted code side-by-side with blocking code •

    Runs a Twisted reactor in another thread, rather than Twisted calling Django • https://github.com/itamarst/crochet
  23. 71.

    To go back to the cooking metaphor, Twisted is just

    the head chef putting orders on the board
  24. 76.

    Channels Upsides • It allows you to use WebSockets! •

    If you don’t care about the response (eg. a page view counter), it can be sent by a channel and run by a worker without blocking the current event • The workers don’t have to be on the same machine, allowing distribution
  25. 77.

    Channels Downsides • You can’t get the results of events

    you create in your code • Your code can still only “do” one thing at a time • Your code is a few steps removed from the real WebSocket or HTTP connections, which makes it less flexible
  26. 79.
  27. 82.

    You are given a channel to send the result of

    your consumer when it is called
  28. 83.

    In the case of a HTTP request, you send back

    a “channel encoded” response object
  29. 97.
  30. 99.

    There is a lot of uses of Django where it

    can end up as a maze of mixins and super()
  31. 100.

    Composition is when larger blocks of logic are made up

    of smaller, loosely-coupled parts of logic.
  32. 103.

    A new generic view system on top of Django Channels

    might be a step in the right direction
  33. 131.

    Next time you write a new View, try and write

    your logic to take data rather than directly calling the ORM.