Amber Brown - The Report Of Twisted’s Death or: Why Twisted and Tornado Are Relevant In The Asyncio Age

With asyncio on the scene, the question has been asked: is there any point in having Twisted or Tornado around?

https://us.pycon.org/2016/schedule/presentation/2114/

Transcript

1. The Report of Twisted’s Death
   …or why Twisted and Tornado are
   relevant in the asyncio age
   

   View Slide

2. Hello, I’m
   Amber Brown
   (HawkOwl)
   

   View Slide

3. I live in Perth, Western Australia
   

   View Slide

4. View Slide

5. Core Developer
   Release Manager
   Ported 40KLoC+ to Python 3
   

   View Slide

6. (image by isometri.cc)
   

   View Slide

7. View Slide

8. Binary release management
   across 3 distros
   Ported Autobahn|Python (Tx)
   and Crossbar.io to Python 3
   Web API/REST integration in CB
   

   View Slide

9. Russell Keith-Magee Glyph Lefkowitz
   Credit where credit is due…
   

   View Slide

10. https://goo.gl/89FqCM
    "The Report Of Our Death"
    by Glyph
    

    View Slide

11. So you want to
    do some I/O…
    

    View Slide

12. View Slide

13. Synchronous I/O frameworks
    (Django, Pyramid, Flask)
    serve one request at a time
    

    View Slide

14. Deployed with runners that
    run many copies using
    threads or processes
    

    View Slide

15. In Python,
    threads or processes
    won’t help with C10K
    (10,000 concurrent connections)
    

    View Slide

16. View Slide

17. View Slide

18. Threads are hard to:
    use safely
    (without race conditions)
    scale with
    (1 thread per connection)
    

    View Slide

19. Thread memory overhead:
    32kB to 8MB per thread
    

    View Slide

20. 128kB of per-thread stack
    x 10,000 threads
    = 1.3GiB of overhead
    

    View Slide

21. Python’s Global Interpreter Lock
    ==
    no parallelism
    

    View Slide

22. You will not do
    threads properly.
    

    View Slide

23. Microthreads/green threads
    (eventlet, gevent)
    are no better
    

    View Slide

24. https://goo.gl/7xE4d1
    "Unyielding", by Glyph
    

    View Slide

25. Non-Threaded
    Asynchronous I/O
    

    View Slide

26. The approach of Twisted,
    Tornado, asyncio, curio
    

    View Slide

27. Twisted was one of the first
    

    View Slide

28. (of SVN history)
    

    View Slide

29. asyncio was introduced
    much later
    

    View Slide

30. View Slide

31. Identical system call at their
    core: Selector functions
    

    View Slide

32. select() and friends
    (poll, epoll, kqueue)
    (or IOCP, but that's different)
    

    View Slide

33. Selector functions take a
    list of file descriptors
    (e.g. sockets, open files)
    and tell you what is ready
    for reading or writing
    

    View Slide

34. Selector loops can handle
    thousands of open sockets
    and events
    

    View Slide

35. 
    

    View Slide

36. Data is channeled through a
    transport to a protocol
    implementation (e.g. HTTP)
    

    View Slide

37. Sending data is queued until
    the network is ready
    

    View Slide

38. Nothing blocks!
    It just waits until the network
    is ready for more data to be
    sent, or more data has arrived
    

    View Slide

39. “I/O loops”
    or
    “reactors”
    (after the “reactor pattern”)
    

    View Slide

40. Higher density per core
    No threads required!
    Concurrency, not parallelism
    

    View Slide

41. Best case:
    high I/O throughput,
    high-latency clients,
    low CPU processing
    

    View Slide

42. You’re probably waiting on
    the client or the database
    

    View Slide

43. Some implementations come
    with abstractions to help you
    

    View Slide

44. They provide an object which
    is a standin for some future
    result, and a way of being
    notified of that result
    

    View Slide

45. Twisted uses a Deferred
    asyncio uses a Future
    

    View Slide

46. d = Deferred()
    d.addCallback(print)
    # some time later...
    d.callback("Hi!")
    

    View Slide

47. f = Future()
    f.add_done_callback(print)
    # some time later...
    f.set_result("Hi!")
    

    View Slide

48. Deferreds run callbacks as
    soon as they are able
    Futures schedule a callback for
    the next reactor loop
    

    View Slide

49. Why a new solution?
    

    View Slide

50. asynchronous I/O in Python
    circa 2012 was a total mess
    

    View Slide

51. no gevent or eventlet
    little Twisted ported
    most of Tornado ported
    

    View Slide

52. Elsewhere:
    node.js exploding in popularity
    async/await shipped
    in .NET 4.5
    

    View Slide

53. Python 3 needed its
    “killer feature”
    

    View Slide

54. Why asyncio?
    

    View Slide

55. A framework designed around
    “coroutines” from the start
    

    View Slide

56. coroutines are a
    special generator
    

    View Slide

57. Python 3.5 contains syntax
    that makes Futures act like
    coroutines
    

    View Slide

58. import asyncio
    import datetime
    async def display_date(loop):
    end_time = loop.time() + 5.0
    while True:
    print(datetime.datetime.now())
    if (loop.time() + 1.0) >= end_time:
    break
    await asyncio.sleep(1)
    loop = asyncio.get_event_loop()
    # Blocking call which returns when the display_date()
    # coroutine is done
    loop.run_until_complete(display_date(loop))
    loop.close()
    

    View Slide

59. asyncio.sleep(1) returns a
    Future which is awaited on in
    the coroutine
    

    View Slide

60. Repairing library API
    fragmentation
    

    View Slide

61. “It should be easy for (Python 3.3 ports
    of) frameworks like Twisted, Tornado, or
    even gevent to either adapt the default
    event loop implementation to their needs
    using a lightweight adapter or proxy, or
    to replace the default event loop
    implementation with an adaptation of
    their own event loop implementation.”
    “Interoperability - asyncio”

    https://www.python.org/dev/peps/pep-3156/
    

    View Slide

62. Reducing duplication
    

    View Slide

63. “For this interoperability to be effective,
    the preferred direction of adaptation in
    third party frameworks is to keep the
    default event loop and adapt it to the
    framework's API. Ideally all third party
    frameworks would give up their own
    event loop implementation in favor of
    the standard implementation.”
    “Interoperability - asyncio”

    https://www.python.org/dev/peps/pep-3156/
    

    View Slide

64. So doesn’t asyncio
    replace Twisted?
    

    View Slide

65. cooperative, single-threaded
    multitasking
    primitives for supporting
    asynchronous programming
    (Futures are like Deferreds,
    coroutines are like inlineCallbacks)
    

    View Slide

66. Same system APIs
    select, poll, epoll(Linux),
    kqueue (BSDs), IOCP(Windows)
    

    View Slide

67. Protocols and transports are
    directly inspired by Twisted
    

    View Slide

68. I/O loop is architecturally
    similar to Twisted’s
    

    View Slide

69. Newer, “standard” API
    Just there in Python 3.4+!
    

    View Slide

70. “Twisted is an async I/O thing, asyncio
    is an async I/O thing. Therefore they are
    the same kind of thing. I only need one
    kind of thing in each category of thing.
    Therefore I only need one of them, and
    the “standard” one is probably the
    better one to depend on. So I guess
    nobody will need Twisted any more!”
    https://glyph.twistedmatrix.com/2014/05/the-report-of-our-death.html
    

    View Slide

71. asyncio is an apple
    Twisted is a fruit salad
    

    View Slide

72. 0
    75000
    150000
    225000
    300000
    Twisted 16.1 asyncio
    3,352
    107,612
    21,902
    176,927
    Code (lines) Comments (lines)
    Lines of Code (Python & C)
    With Tests
    

    View Slide

73. 0
    35000
    70000
    105000
    140000
    Twisted 16.1 asyncio
    2,355
    54,242
    8,452
    74,250
    Code (lines) Comments (lines)
    Lines of Code (Python & C)
    Without Tests
    

    View Slide

74. Twisted did a lot of things
    because none of these things
    were otherwise available
    

    View Slide

75. One big package was easier to:
    distribute
    install
    use
    

    View Slide

76. 0
    35000
    70000
    105000
    140000
    Twisted 16.1 Django 1.9
    25,625
    54,242
    74,033
    74,250
    Code (lines) Comments (lines)
    Lines of Code (Python & C)
    Without Tests
    

    View Slide

77. Asynchronous I/O primitives
    Tools for doing async I/O
    Python utilities
    Protocols using all of the above
    

    View Slide

78. 0
    7500
    15000
    22500
    30000
    Twisted 16.1

    (asyncio parity)
    asyncio

    (+ concurrent.futures)
    2,712
    8,434
    9,143
    13,722
    Code (lines) Comments (lines)
    Lines of Code (Python & C)
    Without Tests
    

    View Slide

79. twisted.internet
    twisted.web.http/wsgi
    twisted.words.protocols.irc
    twisted.protocols.memcache
    twisted.names.resolve
    twisted.conch.ssh
    twisted.mail.smtp
    twisted.words.protocols.jabber
    twisted.python.threadpool
    asyncio
    aiohttp
    irc3
    aiomemcached
    aiodns
    AsyncSSH
    aiosmtpd
    aioxmpp
    concurrent.futures
    

    View Slide

80. Twisted also contains
    protocols that don’t have
    asyncio implementations
    

    View Slide

81. Tornado
    

    View Slide

82. Asynchronous web
    framework in Python
    by FriendFeed/Facebook
    

    View Slide

83. IOStream is similar to
    Twisted/asyncio Transports
    

    View Slide

84. Protocols are not as
    well defined
    

    View Slide

85. Implements its own I/O loop
    Twisted & asyncio integration
    (yield Deferreds or Futures)
    

    View Slide

86. Ultimately, Tornado may
    remove their own I/O loop
    

    View Slide

87. 1.0 (2009): callback-based
    2.1 (2011): Generator-based
    3.0 (2013): concurrent.Futures
    4.3 (Nov 2015): Python 3.5 coroutines
    

    View Slide

88. Tornado is a great example
    of interoperation
    

    View Slide

89. An example for Twisted?
    

    View Slide

90. In where Amber discovers
    that interoperation is hard
    

    View Slide

91. asyncio is similar but
    not the same
    

    View Slide

92. My focus: async/await
    

    View Slide

93. Introduced in Python 3.5
    Detailed in PEP-0492
    

    View Slide

94. async def read_data(db):
    data = await db.fetch('SELECT ...')
    ...
    

    View Slide

95. await gets the result
    of a coroutine
    coroutines are a special
    kind of generator
    

    View Slide

96. Similar to yield from,
    delegates to subgenerator
    Asynchronous code executed in a
    synchronous style
    Yielding for other things while it
    waits for a new result
    

    View Slide

97. @inlineCallbacks
    def loadData(url):
    response = yield makeRequest(url)
    return json.loads(response)
    Twisted has had a trampoline
    to turn Deferreds into a
    generator since 2006
    

    View Slide

98. Coming soon:
    @deferredCoroutine
    

    View Slide

99. await on Deferreds
    coroutines wrapped in Deferreds
    

    View Slide

100. import treq
     from twisted.internet.defer import deferredCoroutine
     from twisted.internet.task import react
     @deferredCoroutine
     async def main(reactor):
     pages = [
     "http://google.com/",
     "http://twistedmatrix.com",
     ]
     results = {}
     for page in pages:
     results[page] = await treq.content(await treq.get(page))
     print(results)
     react(main)
     

     View Slide

101. Coming soon:
     asyncioreactor
     twistd --reactor=asyncio
     

     View Slide

102. Twisted reactor on
     top of asyncio
     

     View Slide

103. Coming soon:
     deferredToFuture
     

     View Slide

104. import asyncio
     import uvloop
     asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
     from twisted.internet.asyncioreactor import install
     install()
     import treq
     from aiohttp import web
     from twisted.internet import reactor, defer
     reactor.startRunning(True)
     user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1'
     async def handle(request):
     url = "http://www.google.com" + request.path_qs
     headers = dict(request.headers)
     try:
     headers.pop('HOST')
     headers.pop('CONNECTION')
     except:
     pass
     get = await defer.deferredToFuture(treq.get(url, headers=headers))
     text = await defer.deferredToFuture(treq.content(get))
     return web.Response(
     body=text,
     content_type=get.headers.getRawHeaders("content-type")[0])
     app = web.Application()
     app.router.add_route('GET', '/{tail:.*}', handle)
     web.run_app(app)
     

     View Slide

105. text = await defer.deferredToFuture(treq.content(get))
     

     View Slide

106. Not reviewed/merged
     Requires asyncio patches
     

     View Slide

107. Why Twisted is still
     worth using
     

     View Slide

108. Released often
     

     View Slide

109. 3+ times a year
     2016 is set to have 5 releases
     

     View Slide

110. Time based releases, taken
     off our trunk branch
     

     View Slide

111. Need the cutting edge?
     Our trunk branch is stable!
     

     View Slide

112. Lots of protocols,
     out of the box!
     

     View Slide

113. HTTP/1.0+1.1, SMTP, DNS,
     IRC, NMEA, FTP, SSH2,
     Memcache, Finger, Telnet,
     SOCKS, POP3, IMAP4
     

     View Slide

114. Super easy to make
     your own protocols!
     

     View Slide

115. from twisted.internet import reactor
     from twisted.internet.endpoints import serverFromString
     from twisted.internet.protocol import Factory
     from twisted.protocols.basic import LineReceiver
     class MyProtocol(LineReceiver):
     def lineReceived(self, line):
     print("I got this line!")
     print(line)
     factory = Factory.forProtocol(MyProtocol)
     endpoint = serverFromString(reactor, "tcp:7000")
     endpoint.listen(factory)
     reactor.run()
     

     View Slide

116. HTTP/2 coming soon!
     

     View Slide

117. Established library support
     

     View Slide

118. Example:
     txacme and txsni
     Python interface to Let’s Encrypt
     Automatic certificate renewal
     

     View Slide

119. Example:
     hendrix
     WSGI runner on top of Twisted
     Websockets, TLS, run Twisted code
     github.com/hendrix/hendrix
     

     View Slide

120. Example:
     Autobahn|Python
     Websockets for Twisted and asyncio
     WS + WAMP, a RPC & PubSub framework
     Super fast under PyPy!
     I work on it :D
     github.com/crossbario/autobahnpython
     

     View Slide

121. We're a dependable base
     

     View Slide

122. We try not to break your code
     

     View Slide

123. Deprecation cycles mean you
     have a year to notice that
     we’re removing something
     

     View Slide

124. Upgrade with impunity!
     

     View Slide

125. Code review
     Automated testing
     Thousands of tests
     

     View Slide

126. Fast, just add PyPy!
     

     View Slide

127. View Slide

128. Many officially supported
     platforms
     

     View Slide

129. Officially supported means
     the tests pass, and must pass
     before branches are merged
     

     View Slide

130. Ubuntu 12.04/14.04/15.04/15.10
     Debian 8
     CentOS 7
     Fedora 22/23
     FreeBSD 10.1
     Windows 7
     OS X 10.10
     

     View Slide

131. Python 2.7 (all platforms)
     Python 3.4 (Linux, FreeBSD)
     Python 3.5 (Linux, FreeBSD)
     Earlier versions support
     2.6 and 3.3
     

     View Slide

132. PyPy is close, only a handful
     of tests remain, nearly all are
     due to CPython assumptions
     

     View Slide

133. Python 3.4/3.5 is coming to
     Windows soon!
     

     View Slide

134. Competition is good!
     

     View Slide

135. Twisted and Tornado fit in
     this ecosystem, if only as
     competitors
     

     View Slide

136. Where to from here?
     

     View Slide

137. Twisted calling asyncio
     asyncio calling Twisted
     

     View Slide

138. async-sig mailing list
     

     View Slide

139. See:
     "Building Protocol Libraries
     the Right Way"
     by Cory Benfield
     "Thinking in Coroutines" by Lukasz Langa
     

     View Slide

140. Questions?
     (pls no statements, save them for after)
     

     View Slide