What Django Can Learn From Twisted (DjangoConAU 2015 Keynote)

Web frameworks like Django are designed around the traditional request-response cycle — a request comes in, a response is generated, and that is delivered to the client. In the day of “single page” applications, where only sections of the page are updated through requests, doing real-time can be clunky. Twisted, and things that build off it, like Django Channels, might be worth thinking about, and this is what this talk will be about.

Transcript

1. What Django Can Learn From Twisted DjangoCon Australia, 2015
2. Hello, I’m Amber Brown (HawkOwl)
3. I live in Perth, Western Australia
4. I organise Django Girls events!
5. omg it’s russ I organise Django Girls events!
6. I serve on the Django Code of Conduct Committee.
7. Otherwise, I have nothing to do with Django.
8. I’m a Twisted core developer! …and release manager (get hype for 15.4!)
9. (image by isometri.cc)
10. We all remember last year’s DjangoCon AU Keynote…
11. Russell offered me this slot to talk about Twisted & Django, and since I kept ranting about it, why not?
12. Talks are only worthwhile if they educate or entertain
13. So I’m going to say this upfront, with no ambiguity:
14. This talk’s conclusion is NOT “Django sucks”.
15. This talk’s conclusion is NOT “using (Twisted/Haskell/ Clojure/Ruby) makes you a better programmer”.
16. This talk’s conclusion is that the future of Python web development is working together.
17. >>> Django == good True >>> Twisted == good True
18. An Intro to Twisted
19. What is Twisted? • Asynchronous networking framework with a stable & mature codebase • Includes primitives for asynchronous I/O and many protocol implementations (e.g. HTTP, SSH, SMTP) • Works on Python 2.7, with base functionality on Python 3.3+
20. Example using Flask Simple google.com proxy (Flask+requests)
21. Example of Twisted Simple google.com proxy (Klein+treq)
22. Let’s break it down!
23. treq.get returns a Deferred and will fire at a later time
24. We tell it that once it has a result, run a callback. This time, it’ll be calling treq.content on the Deferred’s result
25. Then we return the Deferred. Klein will then wait for the Deferred to fire all its callbacks and then write the response.
26. Deferreds
27. Deferreds are an object which holds a result at some point in time
28. Callbacks mean ‘when you have a result, call this function with the result’
29. You chain these functions to create your application
30. Example: Profile page for a Twitter clone
31. request -> get viewing user account -> check permissions (eg. blocked or not) -> get profile of target user -> render page -> respond to client
32. Deferreds make writing applications that use asynchronous I/O easier.
33. Asynchronous I/O
34. Synchronous I/O (“blocking”) is when functions that do I/O return with the result.
35. Asynchronous I/O (“non- blocking”) is when functions that do I/O instead give a promise of a result at some future point in time
36. Twisted uses Deferreds, but similar things (“Promises”) are in JavaScript
37. Benefits of async • If you don’t need the result, just don’t wait for it (eg. incrementing a ‘pageviews’ counter) • Handling many concurrent “quiet” connections is more efficient (eg. WebSockets) • Asynchronous code is broken up over I/O barriers, making it easier to test
38. Twisted uses the “reactor model” to provide concurrency via async I/O
39. Twisted has a single loop, and each portion works cooperatively
40. Efficient for low-CPU, high-I/O use cases
 (web servers, IRC, DNS, SMTP)
41. However, let’s talk about cooking.
42. The wise chef knows she can only do one thing at once
43. …but some things take time!
44. So, she puts the water on to boil, setting a timer, and while it’s boiling, she collects the ingredients.
45. Once the water has boiled, she puts the eggs in, and sets the timer.
46. While the eggs are cooking, she turns the oven on to preheat (and sets a timer), then chops some vegetables.
47. The egg timer goes off, so she stops chopping the vegetables, and takes it off the boil
48. The oven timer goes off, so she puts the vegetables in the oven, and sets a timer
49. She serves the eggs while the vegetables are cooking, and returns when the timer goes off to serve them too.
50. …so what did that have to do with Twisted?
51. Well, the chef is only one woman, but she did several things “at once”.
52. The chef is Twisted, only capable of doing one thing at once, but knowing that not everything needs her attention right now
53. Instead of timers, it’s the reactor saying that “something can be done with this socket”.
54. Making Django Asynchronous
55. Django is synchronous at its core
56. asynchronousness can’t be bolted on
57. “Common Sense” async == hard sync == easy
58. In reality, each approach has tradeoffs
59. Synchronous Upsides • Code flow is easier to understand — do x, then y • Only one “thread” of execution, for simplicity • Many libraries are synchronous
60. Synchronous Downsides • You can only do one thing at once • Although suited to the request/response cycle, it can only really do that • Persistent connections are not simple to implement
61. Asynchronous Upsides • Multiple “threads” of execution — the code handling the request doesn’t have to finish after the request is written • Handling persistent/evented connections is super easy • Reactor model async is threadless, so no thread overhead for concurrency
62. Asynchronous Downsides • You have to be a good citizen — blocking in the reactor loop is disastrous for performance • Doing I/O is “harder” because you have to be explicit about it • In Python, things need to be written to be async
63. You can’t get the upsides of both
64. But you can try!
65. Hendrix • Hendrix is a “Twisted Django” • WSGI server using Twisted, plus WebSockets • Multiprocessing, multithreaded • https://github.com/hangarunderground/hendrix
66. Crochet • Run Twisted code side-by-side with blocking code • Runs a Twisted reactor in another thread, rather than Twisted calling Django • https://github.com/itamarst/crochet
67. Introducing Django Channels
68. Brainchild of Andrew Godwin (1.7 Migrations author)
69. Django Channels makes Django event-driven
70. Asynchronous server (Twisted) + Synchronous “workers”
71. To go back to the cooking metaphor, Twisted is just the head chef putting orders on the board
72. Requests and WebSocket events are now “events” sent through “channels”
73. You write synchronous code which handles these events
74. Channel events go on a queue, and are picked up by workers
75. Workers can also put things on the queue (but can’t get the result)
76. Channels Upsides • It allows you to use WebSockets! • If you don’t care about the response (eg. a page view counter), it can be sent by a channel and run by a worker without blocking the current event • The workers don’t have to be on the same machine, allowing distribution
77. Channels Downsides • You can’t get the results of events you create in your code • Your code can still only “do” one thing at a time • Your code is a few steps removed from the real WebSocket or HTTP connections, which makes it less flexible
78. So, what does Channels look like?
80. When a HTTP/WebSocket event comes in from a client, it sends a message to a channel
81. You implement consumers for these channels
82. You are given a channel to send the result of your consumer when it is called
83. In the case of a HTTP request, you send back a “channel encoded” response object
84. In the case of Websockets, you send back content
85. This content is then returned to the client
86. WebSocket clients can be put into “Groups”
87. You can then broadcast a message out to a Group
88. What makes it different?
89. Channels doesn’t actually make your code asynchronous, it just adds async runners for your sync code
90. It doesn’t tackle the “hard” problem of running Django asynchronously
91. So it doesn’t get all the benefits as if it did
92. Maybe that’s enough?
93. It’s a positive development for Django
94. It supports Python 2.7 and Python 3.3+
95. Check it out: http://git.io/vYEbp
96. Composition
97. Tomorrow @ 10:30am Slow Down, Compose Yourself How Composition Can Help You Write Modular, Testable Code
98. Django relies on inheritance to customise behaviour
99. There is a lot of uses of Django where it can end up as a maze of mixins and super()
100. Composition is when larger blocks of logic are made up of smaller, loosely-coupled parts of logic.
101. A database driver might meet some interface, and can be swapped out
102. Move the ORM one layer deeper
103. A new generic view system on top of Django Channels might be a step in the right direction
104. Unit Testing
105. COMPLAIN ABOUT HOW THE ORM MAKES DJANGO CODE UNTESTABLE HERE
106. Test database dumps can get outdated
107. Composing the ORM- touching sections can make it easier to test
108. Don’t write to the database
109. Don’t have any external databases in your tests
110. Make stand-ins
111. Make assertions of what happened
112. Use a real database for integration and acceptance tests
113. …but that should be driven by Selenium or such
114. Unit tests + integration tests == dependability
115. Global State
116. Django is a big ball of mutable global state.
117. Many parts of Django could have use outside of the project
118. Django Forms is good, I want to use it in Twisted apps
119. So, why not just use Twisted?
120. Twisted has its place, so does Django
121. Django is opinionated
122. Twisted is a pile of bricks
123. But Django has good opinions
124. The future is projects like Hendrix and Django Channels
125. Twisted isn’t hard, it just forces you to deal with hard concepts
126. Each usage requires assessing the tradeoffs
127. Picking and choosing the best from both worlds
128. The future is working together
129. So, in short…
130. Next time you need WebSockets, experiment with Django Channels.
131. Next time you write a new View, try and write your logic to take data rather than directly calling the ORM.
132. Investigate composition, see if it can help make your application easier to understand
133. And most of all…
134. Don’t be scared of Twisted!
135. One last thing
136. Django Girls Australia 2015
 Monday, 3rd of August djangogirls.com/brisbane Still time to apply/mentor!
137. Questions! (non-questions will be ignored)