The Report of Twisted's Death; or Twisted & Tornado in the Asyncio Age (PyCon US 2016)

The Report of Twisted's Death; or Twisted & Tornado in the Asyncio Age (PyCon US 2016)


Amber Brown (HawkOwl)

May 31, 2016


  1. The Report of Twisted’s Death …or why Twisted and Tornado

    are relevant in the asyncio age
  2. Hello, I’m Amber Brown (HawkOwl)

  3. I live in Perth, Western Australia

  4. None
  5. Core Developer Release Manager Ported 40KLoC+ to Python 3

  6. (image by

  7. None
  8. Binary release management across 3 distros Ported Autobahn|Python (Tx) and to Python 3 Web API/REST integration in CB
  9. Russell Keith-Magee Glyph Lefkowitz Credit where credit is due…

  10. "The Report Of Our Death" by Glyph

  11. So you want to do some I/O…

  12. None
  13. Synchronous I/O frameworks (Django, Pyramid, Flask) serve one request at

    a time
  14. Deployed with runners that run many copies using threads or

  15. In Python, threads or processes won’t help with C10K (10,000

    concurrent connections)
  16. None
  17. None
  18. Threads are hard to: use safely (without race conditions) scale

    with (1 thread per connection)
  19. Thread memory overhead: 32kB to 8MB per thread

  20. 128kB of per-thread stack x 10,000 threads = 1.3GiB of

  21. Python’s Global Interpreter Lock == no parallelism

  22. You will not do threads properly.

  23. Microthreads/green threads (eventlet, gevent) are no better

  24. "Unyielding", by Glyph

  25. Non-Threaded Asynchronous I/O

  26. The approach of Twisted, Tornado, asyncio, curio

  27. Twisted was one of the first

  28. (of SVN history)

  29. asyncio was introduced much later

  30. None
  31. Identical system call at their core: Selector functions

  32. select() and friends (poll, epoll, kqueue) (or IOCP, but that's

  33. Selector functions take a list of file descriptors (e.g. sockets,

    open files) and tell you what is ready for reading or writing
  34. Selector loops can handle thousands of open sockets and events

  35. <demo>

  36. Data is channeled through a transport to a protocol implementation

    (e.g. HTTP)
  37. Sending data is queued until the network is ready

  38. Nothing blocks! It just waits until the network is ready

    for more data to be sent, or more data has arrived
  39. “I/O loops” or “reactors” (after the “reactor pattern”)

  40. Higher density per core No threads required! Concurrency, not parallelism

  41. Best case: high I/O throughput, high-latency clients, low CPU processing

  42. You’re probably waiting on the client or the database

  43. Some implementations come with abstractions to help you

  44. They provide an object which is a standin for some

    future result, and a way of being notified of that result
  45. Twisted uses a Deferred asyncio uses a Future

  46. d = Deferred() d.addCallback(print) # some time later... d.callback("Hi!")

  47. f = Future() f.add_done_callback(print) # some time later... f.set_result("Hi!")

  48. Deferreds run callbacks as soon as they are able Futures

    schedule a callback for the next reactor loop
  49. Why a new solution?

  50. asynchronous I/O in Python circa 2012 was a total mess

  51. no gevent or eventlet little Twisted ported most of Tornado

  52. Elsewhere: node.js exploding in popularity async/await shipped in .NET 4.5

  53. Python 3 needed its “killer feature”

  54. Why asyncio?

  55. A framework designed around “coroutines” from the start

  56. coroutines are a special generator

  57. Python 3.5 contains syntax that makes Futures act like coroutines

  58. import asyncio import datetime async def display_date(loop): end_time = loop.time()

    + 5.0 while True: print( if (loop.time() + 1.0) >= end_time: break await asyncio.sleep(1) loop = asyncio.get_event_loop() # Blocking call which returns when the display_date() # coroutine is done loop.run_until_complete(display_date(loop)) loop.close()
  59. asyncio.sleep(1) returns a Future which is awaited on in the

  60. Repairing library API fragmentation

  61. “It should be easy for (Python 3.3 ports of) frameworks

    like Twisted, Tornado, or even gevent to either adapt the default event loop implementation to their needs using a lightweight adapter or proxy, or to replace the default event loop implementation with an adaptation of their own event loop implementation.” “Interoperability - asyncio”
  62. Reducing duplication

  63. “For this interoperability to be effective, the preferred direction of

    adaptation in third party frameworks is to keep the default event loop and adapt it to the framework's API. Ideally all third party frameworks would give up their own event loop implementation in favor of the standard implementation.” “Interoperability - asyncio”
  64. So doesn’t asyncio replace Twisted?

  65. cooperative, single-threaded multitasking primitives for supporting asynchronous programming (Futures are

    like Deferreds, coroutines are like inlineCallbacks)
  66. Same system APIs select, poll, epoll(Linux), kqueue (BSDs), IOCP(Windows)

  67. Protocols and transports are directly inspired by Twisted

  68. I/O loop is architecturally similar to Twisted’s

  69. Newer, “standard” API Just there in Python 3.4+!

  70. “Twisted is an async I/O thing, asyncio is an async

    I/O thing. Therefore they are the same kind of thing. I only need one kind of thing in each category of thing. Therefore I only need one of them, and the “standard” one is probably the better one to depend on. So I guess nobody will need Twisted any more!”
  71. asyncio is an apple Twisted is a fruit salad

  72. 0 75000 150000 225000 300000 Twisted 16.1 asyncio 3,352 107,612

    21,902 176,927 Code (lines) Comments (lines) Lines of Code (Python & C) With Tests
  73. 0 35000 70000 105000 140000 Twisted 16.1 asyncio 2,355 54,242

    8,452 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
  74. Twisted did a lot of things because none of these

    things were otherwise available
  75. One big package was easier to: distribute install use

  76. 0 35000 70000 105000 140000 Twisted 16.1 Django 1.9 25,625

    54,242 74,033 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
  77. Asynchronous I/O primitives Tools for doing async I/O Python utilities

    Protocols using all of the above
  78. 0 7500 15000 22500 30000 Twisted 16.1
 (asyncio parity) asyncio

    (+ concurrent.futures) 2,712 8,434 9,143 13,722 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
  79. twisted.internet twisted.web.http/wsgi twisted.words.protocols.irc twisted.protocols.memcache twisted.names.resolve twisted.conch.ssh twisted.mail.smtp twisted.words.protocols.jabber twisted.python.threadpool asyncio

    aiohttp irc3 aiomemcached aiodns AsyncSSH aiosmtpd aioxmpp concurrent.futures
  80. Twisted also contains protocols that don’t have asyncio implementations

  81. Tornado

  82. Asynchronous web framework in Python by FriendFeed/Facebook

  83. IOStream is similar to Twisted/asyncio Transports

  84. Protocols are not as well defined

  85. Implements its own I/O loop Twisted & asyncio integration (yield

    Deferreds or Futures)
  86. Ultimately, Tornado may remove their own I/O loop

  87. 1.0 (2009): callback-based 2.1 (2011): Generator-based 3.0 (2013): concurrent.Futures 4.3

    (Nov 2015): Python 3.5 coroutines
  88. Tornado is a great example of interoperation

  89. An example for Twisted?

  90. In where Amber discovers that interoperation is hard

  91. asyncio is similar but not the same

  92. My focus: async/await

  93. Introduced in Python 3.5 Detailed in PEP-0492

  94. async def read_data(db): data = await db.fetch('SELECT ...') ...

  95. await gets the result of a coroutine coroutines are a

    special kind of generator
  96. Similar to yield from, delegates to subgenerator Asynchronous code executed

    in a synchronous style Yielding for other things while it waits for a new result
  97. @inlineCallbacks def loadData(url): response = yield makeRequest(url) return json.loads(response) Twisted

    has had a trampoline to turn Deferreds into a generator since 2006
  98. Coming soon: @deferredCoroutine

  99. await on Deferreds coroutines wrapped in Deferreds

  100. import treq from twisted.internet.defer import deferredCoroutine from twisted.internet.task import react

    @deferredCoroutine async def main(reactor): pages = [ "", "", ] results = {} for page in pages: results[page] = await treq.content(await treq.get(page)) print(results) react(main)
  101. Coming soon: asyncioreactor twistd --reactor=asyncio

  102. Twisted reactor on top of asyncio

  103. Coming soon: deferredToFuture

  104. import asyncio import uvloop asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) from twisted.internet.asyncioreactor import install install()

    import treq from aiohttp import web from twisted.internet import reactor, defer reactor.startRunning(True) user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1' async def handle(request): url = "" + request.path_qs headers = dict(request.headers) try: headers.pop('HOST') headers.pop('CONNECTION') except: pass get = await defer.deferredToFuture(treq.get(url, headers=headers)) text = await defer.deferredToFuture(treq.content(get)) return web.Response( body=text, content_type=get.headers.getRawHeaders("content-type")[0]) app = web.Application() app.router.add_route('GET', '/{tail:.*}', handle) web.run_app(app)
  105. text = await defer.deferredToFuture(treq.content(get))

  106. Not reviewed/merged Requires asyncio patches

  107. Why Twisted is still worth using

  108. Released often

  109. 3+ times a year 2016 is set to have 5

  110. Time based releases, taken off our trunk branch

  111. Need the cutting edge? Our trunk branch is stable!

  112. Lots of protocols, out of the box!

  113. HTTP/1.0+1.1, SMTP, DNS, IRC, NMEA, FTP, SSH2, Memcache, Finger, Telnet,

  114. Super easy to make your own protocols!

  115. from twisted.internet import reactor from twisted.internet.endpoints import serverFromString from twisted.internet.protocol

    import Factory from twisted.protocols.basic import LineReceiver class MyProtocol(LineReceiver): def lineReceived(self, line): print("I got this line!") print(line) factory = Factory.forProtocol(MyProtocol) endpoint = serverFromString(reactor, "tcp:7000") endpoint.listen(factory)
  116. HTTP/2 coming soon!

  117. Established library support

  118. Example: txacme and txsni Python interface to Let’s Encrypt Automatic

    certificate renewal
  119. Example: hendrix WSGI runner on top of Twisted Websockets, TLS,

    run Twisted code
  120. Example: Autobahn|Python Websockets for Twisted and asyncio WS + WAMP,

    a RPC & PubSub framework Super fast under PyPy! I work on it :D
  121. We're a dependable base

  122. We try not to break your code

  123. Deprecation cycles mean you have a year to notice that

    we’re removing something
  124. Upgrade with impunity!

  125. Code review Automated testing Thousands of tests

  126. Fast, just add PyPy!

  127. None
  128. Many officially supported platforms

  129. Officially supported means the tests pass, and must pass before

    branches are merged
  130. Ubuntu 12.04/14.04/15.04/15.10 Debian 8 CentOS 7 Fedora 22/23 FreeBSD 10.1

    Windows 7 OS X 10.10
  131. Python 2.7 (all platforms) Python 3.4 (Linux, FreeBSD) Python 3.5

    (Linux, FreeBSD) Earlier versions support 2.6 and 3.3
  132. PyPy is close, only a handful of tests remain, nearly

    all are due to CPython assumptions
  133. Python 3.4/3.5 is coming to Windows soon!

  134. Competition is good!

  135. Twisted and Tornado fit in this ecosystem, if only as

  136. Where to from here?

  137. Twisted calling asyncio asyncio calling Twisted

  138. async-sig mailing list

  139. See: "Building Protocol Libraries the Right Way" by Cory Benfield

    "Thinking in Coroutines" by Lukasz Langa
  140. Questions? (pls no statements, save them for after)