What Django Can Learn From Twisted (DjangoConAU 2015 Keynote)

Web frameworks like Django are designed around the traditional request-response cycle — a request comes in, a response is generated, and that is delivered to the client. In the day of “single page” applications, where only sections of the page are updated through requests, doing real-time can be clunky. Twisted, and things that build off it, like Django Channels, might be worth thinking about, and this is what this talk will be about.

Transcript

1. What Django Can Learn From Twisted
   DjangoCon Australia, 2015
   

   View Slide

2. Hello, I’m
   Amber Brown
   (HawkOwl)
   

   View Slide

3. I live in Perth, Western Australia
   

   View Slide

4. I organise Django Girls events!
   

   View Slide

5. omg it’s russ
   I organise Django Girls events!
   

   View Slide

6. I serve on the Django Code of Conduct
   Committee.
   

   View Slide

7. Otherwise, I have nothing to do with
   Django.
   

   View Slide

8. I’m a Twisted core
   developer!
   …and release manager
   (get hype for 15.4!)
   

   View Slide

9. (image by isometri.cc)
   

   View Slide

10. We all remember last year’s
    DjangoCon AU Keynote…
    

    View Slide

11. Russell offered me this slot to talk
    about Twisted & Django, and since
    I kept ranting about it, why not?
    

    View Slide

12. Talks are only worthwhile
    if they educate or entertain
    

    View Slide

13. So I’m going to say this
    upfront, with no ambiguity:
    

    View Slide

14. This talk’s conclusion is NOT
    “Django sucks”.
    

    View Slide

15. This talk’s conclusion is NOT
    “using (Twisted/Haskell/
    Clojure/Ruby) makes you a
    better programmer”.
    

    View Slide

16. This talk’s conclusion is
    that the future of Python
    web development is
    working together.
    

    View Slide

17. >>> Django == good
    True
    >>> Twisted == good
    True
    

    View Slide

18. An Intro to Twisted
    

    View Slide

19. What is Twisted?
    • Asynchronous networking framework with a
    stable & mature codebase
    • Includes primitives for asynchronous I/O and
    many protocol implementations (e.g. HTTP, SSH,
    SMTP)
    • Works on Python 2.7, with base functionality on
    Python 3.3+
    

    View Slide

20. Example using Flask
    Simple google.com proxy
    (Flask+requests)
    

    View Slide

21. Example of Twisted
    Simple google.com proxy (Klein+treq)
    

    View Slide

22. Let’s break it down!
    

    View Slide

23. treq.get returns a Deferred and will fire
    at a later time
    

    View Slide

24. We tell it that once it has a result, run a
    callback. This time, it’ll be calling
    treq.content on the Deferred’s result
    

    View Slide

25. Then we return the Deferred. Klein will
    then wait for the Deferred to fire all its
    callbacks and then write the response.
    

    View Slide

26. Deferreds
    

    View Slide

27. Deferreds are an object
    which holds a result at
    some point in time
    

    View Slide

28. Callbacks mean ‘when you
    have a result, call this
    function with the result’
    

    View Slide

29. You chain these functions
    to create your application
    

    View Slide

30. Example: Profile page
    for a Twitter clone
    

    View Slide

31. request ->
    get viewing user account ->
    check permissions (eg. blocked or not) ->
    get profile of target user ->
    render page ->
    respond to client
    

    View Slide

32. Deferreds make writing
    applications that use
    asynchronous I/O easier.
    

    View Slide

33. Asynchronous I/O
    

    View Slide

34. Synchronous I/O
    (“blocking”) is when
    functions that do I/O
    return with the result.
    

    View Slide

35. Asynchronous I/O (“non-
    blocking”) is when
    functions that do I/O
    instead give a promise of a
    result at some future point
    in time
    

    View Slide

36. Twisted uses Deferreds, but
    similar things (“Promises”)
    are in JavaScript
    

    View Slide

37. Benefits of async
    • If you don’t need the result, just don’t wait for it
    (eg. incrementing a ‘pageviews’ counter)
    • Handling many concurrent “quiet” connections
    is more efficient (eg. WebSockets)
    • Asynchronous code is broken up over I/O
    barriers, making it easier to test
    

    View Slide

38. Twisted uses the
    “reactor model” to provide
    concurrency via async I/O
    

    View Slide

39. Twisted has a single
    loop, and each portion
    works cooperatively
    

    View Slide

40. Efficient for low-CPU,
    high-I/O use cases

    (web servers, IRC, DNS, SMTP)
    

    View Slide

41. However, let’s talk
    about cooking.
    

    View Slide

42. The wise chef knows she
    can only do one thing at
    once
    

    View Slide

43. …but some things take
    time!
    

    View Slide

44. So, she puts the water on to
    boil, setting a timer, and
    while it’s boiling, she
    collects the ingredients.
    

    View Slide

45. Once the water has boiled,
    she puts the eggs in, and
    sets the timer.
    

    View Slide

46. While the eggs are cooking,
    she turns the oven on to
    preheat (and sets a timer),
    then chops some vegetables.
    

    View Slide

47. The egg timer goes off, so she
    stops chopping the vegetables,
    and takes it off the boil
    

    View Slide

48. The oven timer goes off, so
    she puts the vegetables in
    the oven, and sets a timer
    

    View Slide

49. She serves the eggs while
    the vegetables are cooking,
    and returns when the timer
    goes off to serve them too.
    

    View Slide

50. …so what did that have
    to do with Twisted?
    

    View Slide

51. Well, the chef is only one
    woman, but she did several
    things “at once”.
    

    View Slide

52. The chef is Twisted, only
    capable of doing one thing at
    once, but knowing that not
    everything needs her attention
    right now
    

    View Slide

53. Instead of timers, it’s the
    reactor saying that “something
    can be done with this socket”.
    

    View Slide

54. Making Django
    Asynchronous
    

    View Slide

55. Django is synchronous
    at its core
    

    View Slide

56. asynchronousness
    can’t be bolted on
    

    View Slide

57. “Common Sense”
    async == hard
    sync == easy
    

    View Slide

58. In reality, each
    approach has tradeoffs
    

    View Slide

59. Synchronous Upsides
    • Code flow is easier to understand — do x, then y
    • Only one “thread” of execution, for simplicity
    • Many libraries are synchronous
    

    View Slide

60. Synchronous Downsides
    • You can only do one thing at once
    • Although suited to the request/response cycle, it
    can only really do that
    • Persistent connections are not simple to
    implement
    

    View Slide

61. Asynchronous Upsides
    • Multiple “threads” of execution — the code
    handling the request doesn’t have to finish after
    the request is written
    • Handling persistent/evented connections is
    super easy
    • Reactor model async is threadless, so no thread
    overhead for concurrency
    

    View Slide

62. Asynchronous Downsides
    • You have to be a good citizen — blocking in the
    reactor loop is disastrous for performance
    • Doing I/O is “harder” because you have to be
    explicit about it
    • In Python, things need to be written to be async
    

    View Slide

63. You can’t get the
    upsides of both
    

    View Slide

64. But you can try!
    

    View Slide

65. Hendrix
    • Hendrix is a “Twisted Django”
    • WSGI server using Twisted, plus WebSockets
    • Multiprocessing, multithreaded
    • https://github.com/hangarunderground/hendrix
    

    View Slide

66. Crochet
    • Run Twisted code side-by-side with blocking
    code
    • Runs a Twisted reactor in another thread, rather
    than Twisted calling Django
    • https://github.com/itamarst/crochet
    

    View Slide

67. Introducing Django
    Channels
    

    View Slide

68. Brainchild of
    Andrew Godwin
    (1.7 Migrations author)
    

    View Slide

69. Django Channels makes
    Django event-driven
    

    View Slide

70. Asynchronous server
    (Twisted)
    +
    Synchronous “workers”
    

    View Slide

71. To go back to the cooking
    metaphor, Twisted is just
    the head chef putting
    orders on the board
    

    View Slide

72. Requests and WebSocket
    events are now “events”
    sent through “channels”
    

    View Slide

73. You write synchronous
    code which handles
    these events
    

    View Slide

74. Channel events go on a
    queue, and are picked
    up by workers
    

    View Slide

75. Workers can also put
    things on the queue
    (but can’t get the result)
    

    View Slide

76. Channels Upsides
    • It allows you to use WebSockets!
    • If you don’t care about the response (eg. a page
    view counter), it can be sent by a channel and
    run by a worker without blocking the current
    event
    • The workers don’t have to be on the same
    machine, allowing distribution
    

    View Slide

77. Channels Downsides
    • You can’t get the results of events you create in
    your code
    • Your code can still only “do” one thing at a time
    • Your code is a few steps removed from the real
    WebSocket or HTTP connections, which makes
    it less flexible
    

    View Slide

78. So, what does
    Channels look like?
    

    View Slide

79. View Slide

80. When a HTTP/WebSocket
    event comes in from a client, it
    sends a message to a channel
    

    View Slide

81. You implement consumers
    for these channels
    

    View Slide

82. You are given a channel to
    send the result of your
    consumer when it is called
    

    View Slide

83. In the case of a HTTP request,
    you send back a “channel
    encoded” response object
    

    View Slide

84. In the case of Websockets,
    you send back content
    

    View Slide

85. This content is then
    returned to the client
    

    View Slide

86. WebSocket clients can
    be put into “Groups”
    

    View Slide

87. You can then broadcast a
    message out to a Group
    

    View Slide

88. What makes it
    different?
    

    View Slide

89. Channels doesn’t actually
    make your code asynchronous,
    it just adds async runners for
    your sync code
    

    View Slide

90. It doesn’t tackle the
    “hard” problem of running
    Django asynchronously
    

    View Slide

91. So it doesn’t get all the
    benefits as if it did
    

    View Slide

92. Maybe that’s enough?
    

    View Slide

93. It’s a positive
    development for Django
    

    View Slide

94. It supports Python 2.7
    and Python 3.3+
    

    View Slide

95. Check it out:
    http://git.io/vYEbp
    

    View Slide

96. Composition
    

    View Slide

97. Tomorrow @ 10:30am
    Slow Down,
    Compose Yourself
    How Composition Can Help You
    Write Modular, Testable Code
    

    View Slide

98. Django relies on inheritance
    to customise behaviour
    

    View Slide

99. There is a lot of uses of Django
    where it can end up as a maze
    of mixins and super()
    

    View Slide

100. Composition is when larger
     blocks of logic are made up
     of smaller, loosely-coupled
     parts of logic.
     

     View Slide

101. A database driver might
     meet some interface, and
     can be swapped out
     

     View Slide

102. Move the ORM one
     layer deeper
     

     View Slide

103. A new generic view system on
     top of Django Channels might
     be a step in the right direction
     

     View Slide

104. Unit Testing
     

     View Slide

105. COMPLAIN ABOUT HOW THE
     ORM MAKES DJANGO CODE
     UNTESTABLE HERE
     

     View Slide

106. Test database dumps
     can get outdated
     

     View Slide

107. Composing the ORM-
     touching sections can make
     it easier to test
     

     View Slide

108. Don’t write to the
     database
     

     View Slide

109. Don’t have any external
     databases in your tests
     

     View Slide

110. Make stand-ins
     

     View Slide

111. Make assertions of
     what happened
     

     View Slide

112. Use a real database for
     integration and acceptance
     tests
     

     View Slide

113. …but that should be
     driven by Selenium or such
     

     View Slide

114. Unit tests
     +
     integration tests
     ==
     dependability
     

     View Slide

115. Global State
     

     View Slide

116. Django is a big ball of
     mutable global state.
     

     View Slide

117. Many parts of Django
     could have use outside
     of the project
     

     View Slide

118. Django Forms is good, I
     want to use it in Twisted
     apps
     

     View Slide

119. So, why not just use
     Twisted?
     

     View Slide

120. Twisted has its place,
     so does Django
     

     View Slide

121. Django is opinionated
     

     View Slide

122. Twisted is a pile of
     bricks
     

     View Slide

123. But Django has good
     opinions
     

     View Slide

124. The future is projects like
     Hendrix and
     Django Channels
     

     View Slide

125. Twisted isn’t hard, it
     just forces you to deal
     with hard concepts
     

     View Slide

126. Each usage requires
     assessing the tradeoffs
     

     View Slide

127. Picking and choosing the
     best from both worlds
     

     View Slide

128. The future is working
     together
     

     View Slide

129. So, in short…
     

     View Slide

130. Next time you need
     WebSockets,
     experiment with
     Django Channels.
     

     View Slide

131. Next time you write a new
     View, try and write your logic
     to take data rather than
     directly calling the ORM.
     

     View Slide

132. Investigate composition, see
     if it can help make your
     application easier to
     understand
     

     View Slide

133. And most of all…
     

     View Slide

134. Don’t be scared of
     Twisted!
     

     View Slide

135. One last thing
     

     View Slide

136. Django Girls Australia 2015

     Monday, 3rd of August
     djangogirls.com/brisbane
     Still time to apply/mentor!
     

     View Slide

137. Questions!
     (non-questions will be ignored)
     

     View Slide