Django & Twisted (Django Under The Hood 2015)

Django & Twisted (Django Under The Hood 2015)


Amber Brown (HawkOwl)

November 06, 2015


  1. 9.
  2. 17.
  3. 22.
  4. 30.

    Python suspends at the yield point, and can run other

    things — purely syntactic sugar
  5. 31.

    Blocking, for the purposes of this talk, means that Python

    cannot run absolutely anything else during that period due to I/O operations
  6. 35.

    PING ( 56 data bytes 64 bytes: icmp_seq=0 ttl=60

    time=13.217 ms 64 bytes: icmp_seq=1 ttl=60 time=18.227 ms 64 bytes: icmp_seq=2 ttl=60 time=13.117 ms
  7. 50.

    This secondary send buffering is taken care of by the

    Twisted Protocol class (socket.write() is never called directly by user code)
  8. 52.
  9. 54.

    Takes a list of file descriptors (eg. sockets) and returns

    the ones that can have further data written/read
  10. 56.

    If more data can be read, Protocol reads it and

    gives it to user code with the overridden dataReceived method
  11. 60.
  12. 61.
  13. 68.
  14. 73.

    >>> d = Deferred() >>> d.addCallback(lambda t: t + 1)

    <Deferred at 0x100a03c50> >>> d.addCallback(lambda t: print(t)) <Deferred at 0x100a03c50> >>> d.callback(12) 13
  15. 79.

    import treq from twisted.internet.task import react def get(reactor): d =

    treq.get("") d.addCallback(treq.content) d.addCallback(lambda _: print(_)) return d react(get)
  16. 82.

    import treq from twisted.internet.task import react from twisted.internet.defer import inlineCallbacks

    @inlineCallbacks def get(reactor): request = yield treq.get( "") content = yield treq.content(request) print(content) react(get)
  17. 96.

    Synchronous Upsides • Code flow is easier to understand —

    do x, then y • Only one “thread” of execution, for simplicity • Many libraries are synchronous
  18. 97.

    Synchronous Downsides • You can only do one thing at

    once • Although suited to the request/response cycle, it can only really do that • Persistent connections are not simple to implement
  19. 98.

    Asynchronous Upsides • Massively scalable network concurrency • Multiple “threads”

    of execution — the code handling the request doesn’t have to finish after the request is written • Handling persistent/evented connections is super easy • Reactor model async is threadless • Python 3 adds some syntactic sugar that makes it easier to write
  20. 99.

    Asynchronous Downsides • “Callback hell” when using raw futures/deferreds •

    You have to be a good citizen — blocking in the reactor loop is disastrous for performance • Doing I/O is “harder” because you have to be explicit about it • Python 2 lacks a bunch of async syntactic sugar
  21. 102.

    Threaded WSGI Runner • The standard Django deployment method —

    run lots of threads, so it doesn’t matter if it blocks • Each thread is blocking, so it can’t run multiple I/O operations at once • To handle many concurrent requests, you need many threads
  22. 103.

    Hendrix • Hendrix is a “Twisted Django” • WSGI server

    using Twisted, plus WebSockets • Multiprocessing, multithreaded •
  23. 104.

    Crochet • Run Twisted code side-by-side with blocking code •

    Runs a Twisted reactor in another thread, rather than Twisted calling Django •
  24. 113.

    Channels Upsides • It allows you to use WebSockets! •

    If you don’t care about the response (eg. a page view counter), it can be sent by a channel and run by a worker without blocking the current event • The workers don’t have to be on the same machine, allowing distribution
  25. 114.

    Channels Downsides • You can’t get the results of events

    you create in your code • Your code can still only “do” one thing at a time • Your code is a few steps removed from the real WebSocket or HTTP connections, which makes it less flexible
  26. 116.
  27. 117.
  28. 119.

    You are given a channel to send the result of

    your consumer when it is called
  29. 120.

    In the case of a HTTP request, you send back

    a “channel encoded” response object
  30. 134.
  31. 146.
  32. 151.

    This is Django… …with async views… …with an async ORM…

    …running on Twisted Web… …with no WSGI.
  33. 152.
  34. 153.
  35. 154.

    Caveats: I wrote this on a plane, the ORM runs

    in a threadpool, the tests fail hilariously
  36. 183.

    Like I said earlier, you cannot get the upsides of

    async and sync code at the same time
  37. 185.

    Features like yield from and async def can be adopted

    by Twisted, even though they’re targeted at asyncio
  38. 192.

    — Glyph “Despite the fact that implicit coroutines masquerade under

    many different names, many of which don’t include the word “thread” – for example, “greenlets”, “coroutines”, “fibers”, “tasks” – green or lightweight threads are indeed threads … In the long run, when you build a system that relies upon them, you eventually have all the pitfalls and dangers of full-blown preemptive threads.”
  39. 196.
  40. 197.