What Django Can Learn From Twisted (DjangoConAU 2015 Keynote)

What Django Can Learn From Twisted (DjangoConAU 2015 Keynote)

Web frameworks like Django are designed around the traditional request-response cycle — a request comes in, a response is generated, and that is delivered to the client. In the day of “single page” applications, where only sections of the page are updated through requests, doing real-time can be clunky. Twisted, and things that build off it, like Django Channels, might be worth thinking about, and this is what this talk will be about.

3d37232726396a1d3c7412dd915095ea?s=128

Amber Brown (HawkOwl)

July 31, 2015
Tweet

Transcript

  1. What Django Can Learn From Twisted DjangoCon Australia, 2015

  2. Hello, I’m Amber Brown (HawkOwl)

  3. I live in Perth, Western Australia

  4. I organise Django Girls events!

  5. omg it’s russ I organise Django Girls events!

  6. I serve on the Django Code of Conduct Committee.

  7. Otherwise, I have nothing to do with Django.

  8. I’m a Twisted core developer! …and release manager (get hype

    for 15.4!)
  9. (image by isometri.cc)

  10. We all remember last year’s DjangoCon AU Keynote…

  11. Russell offered me this slot to talk about Twisted &

    Django, and since I kept ranting about it, why not?
  12. Talks are only worthwhile if they educate or entertain

  13. So I’m going to say this upfront, with no ambiguity:

  14. This talk’s conclusion is NOT “Django sucks”.

  15. This talk’s conclusion is NOT “using (Twisted/Haskell/ Clojure/Ruby) makes you

    a better programmer”.
  16. This talk’s conclusion is that the future of Python web

    development is working together.
  17. >>> Django == good True >>> Twisted == good True

  18. An Intro to Twisted

  19. What is Twisted? • Asynchronous networking framework with a stable

    & mature codebase • Includes primitives for asynchronous I/O and many protocol implementations (e.g. HTTP, SSH, SMTP) • Works on Python 2.7, with base functionality on Python 3.3+
  20. Example using Flask Simple google.com proxy (Flask+requests)

  21. Example of Twisted Simple google.com proxy (Klein+treq)

  22. Let’s break it down!

  23. treq.get returns a Deferred and will fire at a later

    time
  24. We tell it that once it has a result, run

    a callback. This time, it’ll be calling treq.content on the Deferred’s result
  25. Then we return the Deferred. Klein will then wait for

    the Deferred to fire all its callbacks and then write the response.
  26. Deferreds

  27. Deferreds are an object which holds a result at some

    point in time
  28. Callbacks mean ‘when you have a result, call this function

    with the result’
  29. You chain these functions to create your application

  30. Example: Profile page for a Twitter clone

  31. request -> get viewing user account -> check permissions (eg.

    blocked or not) -> get profile of target user -> render page -> respond to client
  32. Deferreds make writing applications that use asynchronous I/O easier.

  33. Asynchronous I/O

  34. Synchronous I/O (“blocking”) is when functions that do I/O return

    with the result.
  35. Asynchronous I/O (“non- blocking”) is when functions that do I/O

    instead give a promise of a result at some future point in time
  36. Twisted uses Deferreds, but similar things (“Promises”) are in JavaScript

  37. Benefits of async • If you don’t need the result,

    just don’t wait for it (eg. incrementing a ‘pageviews’ counter) • Handling many concurrent “quiet” connections is more efficient (eg. WebSockets) • Asynchronous code is broken up over I/O barriers, making it easier to test
  38. Twisted uses the “reactor model” to provide concurrency via async

    I/O
  39. Twisted has a single loop, and each portion works cooperatively

  40. Efficient for low-CPU, high-I/O use cases
 (web servers, IRC, DNS,

    SMTP)
  41. However, let’s talk about cooking.

  42. The wise chef knows she can only do one thing

    at once
  43. …but some things take time!

  44. So, she puts the water on to boil, setting a

    timer, and while it’s boiling, she collects the ingredients.
  45. Once the water has boiled, she puts the eggs in,

    and sets the timer.
  46. While the eggs are cooking, she turns the oven on

    to preheat (and sets a timer), then chops some vegetables.
  47. The egg timer goes off, so she stops chopping the

    vegetables, and takes it off the boil
  48. The oven timer goes off, so she puts the vegetables

    in the oven, and sets a timer
  49. She serves the eggs while the vegetables are cooking, and

    returns when the timer goes off to serve them too.
  50. …so what did that have to do with Twisted?

  51. Well, the chef is only one woman, but she did

    several things “at once”.
  52. The chef is Twisted, only capable of doing one thing

    at once, but knowing that not everything needs her attention right now
  53. Instead of timers, it’s the reactor saying that “something can

    be done with this socket”.
  54. Making Django Asynchronous

  55. Django is synchronous at its core

  56. asynchronousness can’t be bolted on

  57. “Common Sense” async == hard sync == easy

  58. In reality, each approach has tradeoffs

  59. Synchronous Upsides • Code flow is easier to understand —

    do x, then y • Only one “thread” of execution, for simplicity • Many libraries are synchronous
  60. Synchronous Downsides • You can only do one thing at

    once • Although suited to the request/response cycle, it can only really do that • Persistent connections are not simple to implement
  61. Asynchronous Upsides • Multiple “threads” of execution — the code

    handling the request doesn’t have to finish after the request is written • Handling persistent/evented connections is super easy • Reactor model async is threadless, so no thread overhead for concurrency
  62. Asynchronous Downsides • You have to be a good citizen

    — blocking in the reactor loop is disastrous for performance • Doing I/O is “harder” because you have to be explicit about it • In Python, things need to be written to be async
  63. You can’t get the upsides of both

  64. But you can try!

  65. Hendrix • Hendrix is a “Twisted Django” • WSGI server

    using Twisted, plus WebSockets • Multiprocessing, multithreaded • https://github.com/hangarunderground/hendrix
  66. Crochet • Run Twisted code side-by-side with blocking code •

    Runs a Twisted reactor in another thread, rather than Twisted calling Django • https://github.com/itamarst/crochet
  67. Introducing Django Channels

  68. Brainchild of Andrew Godwin (1.7 Migrations author)

  69. Django Channels makes Django event-driven

  70. Asynchronous server (Twisted) + Synchronous “workers”

  71. To go back to the cooking metaphor, Twisted is just

    the head chef putting orders on the board
  72. Requests and WebSocket events are now “events” sent through “channels”

  73. You write synchronous code which handles these events

  74. Channel events go on a queue, and are picked up

    by workers
  75. Workers can also put things on the queue (but can’t

    get the result)
  76. Channels Upsides • It allows you to use WebSockets! •

    If you don’t care about the response (eg. a page view counter), it can be sent by a channel and run by a worker without blocking the current event • The workers don’t have to be on the same machine, allowing distribution
  77. Channels Downsides • You can’t get the results of events

    you create in your code • Your code can still only “do” one thing at a time • Your code is a few steps removed from the real WebSocket or HTTP connections, which makes it less flexible
  78. So, what does Channels look like?

  79. None
  80. When a HTTP/WebSocket event comes in from a client, it

    sends a message to a channel
  81. You implement consumers for these channels

  82. You are given a channel to send the result of

    your consumer when it is called
  83. In the case of a HTTP request, you send back

    a “channel encoded” response object
  84. In the case of Websockets, you send back content

  85. This content is then returned to the client

  86. WebSocket clients can be put into “Groups”

  87. You can then broadcast a message out to a Group

  88. What makes it different?

  89. Channels doesn’t actually make your code asynchronous, it just adds

    async runners for your sync code
  90. It doesn’t tackle the “hard” problem of running Django asynchronously

  91. So it doesn’t get all the benefits as if it

    did
  92. Maybe that’s enough?

  93. It’s a positive development for Django

  94. It supports Python 2.7 and Python 3.3+

  95. Check it out: http://git.io/vYEbp

  96. Composition

  97. Tomorrow @ 10:30am Slow Down, Compose Yourself How Composition Can

    Help You Write Modular, Testable Code
  98. Django relies on inheritance to customise behaviour

  99. There is a lot of uses of Django where it

    can end up as a maze of mixins and super()
  100. Composition is when larger blocks of logic are made up

    of smaller, loosely-coupled parts of logic.
  101. A database driver might meet some interface, and can be

    swapped out
  102. Move the ORM one layer deeper

  103. A new generic view system on top of Django Channels

    might be a step in the right direction
  104. Unit Testing

  105. COMPLAIN ABOUT HOW THE ORM MAKES DJANGO CODE UNTESTABLE HERE

  106. Test database dumps can get outdated

  107. Composing the ORM- touching sections can make it easier to

    test
  108. Don’t write to the database

  109. Don’t have any external databases in your tests

  110. Make stand-ins

  111. Make assertions of what happened

  112. Use a real database for integration and acceptance tests

  113. …but that should be driven by Selenium or such

  114. Unit tests + integration tests == dependability

  115. Global State

  116. Django is a big ball of mutable global state.

  117. Many parts of Django could have use outside of the

    project
  118. Django Forms is good, I want to use it in

    Twisted apps
  119. So, why not just use Twisted?

  120. Twisted has its place, so does Django

  121. Django is opinionated

  122. Twisted is a pile of bricks

  123. But Django has good opinions

  124. The future is projects like Hendrix and Django Channels

  125. Twisted isn’t hard, it just forces you to deal with

    hard concepts
  126. Each usage requires assessing the tradeoffs

  127. Picking and choosing the best from both worlds

  128. The future is working together

  129. So, in short…

  130. Next time you need WebSockets, experiment with Django Channels.

  131. Next time you write a new View, try and write

    your logic to take data rather than directly calling the ORM.
  132. Investigate composition, see if it can help make your application

    easier to understand
  133. And most of all…

  134. Don’t be scared of Twisted!

  135. One last thing

  136. Django Girls Australia 2015
 Monday, 3rd of August djangogirls.com/brisbane Still

    time to apply/mentor!
  137. Questions! (non-questions will be ignored)