Amber Brown - The Report Of Twisted’s Death or: Why Twisted and Tornado Are Relevant In The Asyncio Age

With asyncio on the scene, the question has been asked: is there any point in having Twisted or Tornado around?

https://us.pycon.org/2016/schedule/presentation/2114/

Transcript

1. The Report of Twisted’s Death …or why Twisted and Tornado are relevant in the asyncio age
2. Hello, I’m Amber Brown (HawkOwl)
3. I live in Perth, Western Australia
5. Core Developer Release Manager Ported 40KLoC+ to Python 3
6. (image by isometri.cc)
8. Binary release management across 3 distros Ported Autobahn|Python (Tx) and Crossbar.io to Python 3 Web API/REST integration in CB
9. Russell Keith-Magee Glyph Lefkowitz Credit where credit is due…
10. https://goo.gl/89FqCM "The Report Of Our Death" by Glyph
11. So you want to do some I/O…
13. Synchronous I/O frameworks (Django, Pyramid, Flask) serve one request at a time
14. Deployed with runners that run many copies using threads or processes
15. In Python, threads or processes won’t help with C10K (10,000 concurrent connections)
18. Threads are hard to: use safely (without race conditions) scale with (1 thread per connection)
19. Thread memory overhead: 32kB to 8MB per thread
20. 128kB of per-thread stack x 10,000 threads = 1.3GiB of overhead
21. Python’s Global Interpreter Lock == no parallelism
22. You will not do threads properly.
23. Microthreads/green threads (eventlet, gevent) are no better
24. https://goo.gl/7xE4d1 "Unyielding", by Glyph
25. Non-Threaded Asynchronous I/O
26. The approach of Twisted, Tornado, asyncio, curio
27. Twisted was one of the first
28. (of SVN history)
29. asyncio was introduced much later
31. Identical system call at their core: Selector functions
32. select() and friends (poll, epoll, kqueue) (or IOCP, but that's different)
33. Selector functions take a list of file descriptors (e.g. sockets, open files) and tell you what is ready for reading or writing
34. Selector loops can handle thousands of open sockets and events
35. <demo>
36. Data is channeled through a transport to a protocol implementation (e.g. HTTP)
37. Sending data is queued until the network is ready
38. Nothing blocks! It just waits until the network is ready for more data to be sent, or more data has arrived
39. “I/O loops” or “reactors” (after the “reactor pattern”)
40. Higher density per core No threads required! Concurrency, not parallelism
41. Best case: high I/O throughput, high-latency clients, low CPU processing
42. You’re probably waiting on the client or the database
43. Some implementations come with abstractions to help you
44. They provide an object which is a standin for some future result, and a way of being notified of that result
45. Twisted uses a Deferred asyncio uses a Future
46. d = Deferred() d.addCallback(print) # some time later... d.callback("Hi!")
47. f = Future() f.add_done_callback(print) # some time later... f.set_result("Hi!")
48. Deferreds run callbacks as soon as they are able Futures schedule a callback for the next reactor loop
49. Why a new solution?
50. asynchronous I/O in Python circa 2012 was a total mess
51. no gevent or eventlet little Twisted ported most of Tornado ported
52. Elsewhere: node.js exploding in popularity async/await shipped in .NET 4.5
53. Python 3 needed its “killer feature”
54. Why asyncio?
55. A framework designed around “coroutines” from the start
56. coroutines are a special generator
57. Python 3.5 contains syntax that makes Futures act like coroutines
58. import asyncio import datetime async def display_date(loop): end_time = loop.time() + 5.0 while True: print(datetime.datetime.now()) if (loop.time() + 1.0) >= end_time: break await asyncio.sleep(1) loop = asyncio.get_event_loop() # Blocking call which returns when the display_date() # coroutine is done loop.run_until_complete(display_date(loop)) loop.close()
59. asyncio.sleep(1) returns a Future which is awaited on in the coroutine
60. Repairing library API fragmentation
61. “It should be easy for (Python 3.3 ports of) frameworks like Twisted, Tornado, or even gevent to either adapt the default event loop implementation to their needs using a lightweight adapter or proxy, or to replace the default event loop implementation with an adaptation of their own event loop implementation.” “Interoperability - asyncio”
 https://www.python.org/dev/peps/pep-3156/
62. Reducing duplication
63. “For this interoperability to be effective, the preferred direction of adaptation in third party frameworks is to keep the default event loop and adapt it to the framework's API. Ideally all third party frameworks would give up their own event loop implementation in favor of the standard implementation.” “Interoperability - asyncio”
 https://www.python.org/dev/peps/pep-3156/
64. So doesn’t asyncio replace Twisted?
65. cooperative, single-threaded multitasking primitives for supporting asynchronous programming (Futures are like Deferreds, coroutines are like inlineCallbacks)
66. Same system APIs select, poll, epoll(Linux), kqueue (BSDs), IOCP(Windows)
67. Protocols and transports are directly inspired by Twisted
68. I/O loop is architecturally similar to Twisted’s
69. Newer, “standard” API Just there in Python 3.4+!
70. “Twisted is an async I/O thing, asyncio is an async I/O thing. Therefore they are the same kind of thing. I only need one kind of thing in each category of thing. Therefore I only need one of them, and the “standard” one is probably the better one to depend on. So I guess nobody will need Twisted any more!” https://glyph.twistedmatrix.com/2014/05/the-report-of-our-death.html
71. asyncio is an apple Twisted is a fruit salad
72. 0 75000 150000 225000 300000 Twisted 16.1 asyncio 3,352 107,612 21,902 176,927 Code (lines) Comments (lines) Lines of Code (Python & C) With Tests
73. 0 35000 70000 105000 140000 Twisted 16.1 asyncio 2,355 54,242 8,452 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
74. Twisted did a lot of things because none of these things were otherwise available
75. One big package was easier to: distribute install use
76. 0 35000 70000 105000 140000 Twisted 16.1 Django 1.9 25,625 54,242 74,033 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
77. Asynchronous I/O primitives Tools for doing async I/O Python utilities Protocols using all of the above
78. 0 7500 15000 22500 30000 Twisted 16.1
 (asyncio parity) asyncio
 (+ concurrent.futures) 2,712 8,434 9,143 13,722 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
79. twisted.internet twisted.web.http/wsgi twisted.words.protocols.irc twisted.protocols.memcache twisted.names.resolve twisted.conch.ssh twisted.mail.smtp twisted.words.protocols.jabber twisted.python.threadpool asyncio aiohttp irc3 aiomemcached aiodns AsyncSSH aiosmtpd aioxmpp concurrent.futures
80. Twisted also contains protocols that don’t have asyncio implementations
81. Tornado
82. Asynchronous web framework in Python by FriendFeed/Facebook
83. IOStream is similar to Twisted/asyncio Transports
84. Protocols are not as well defined
85. Implements its own I/O loop Twisted & asyncio integration (yield Deferreds or Futures)
86. Ultimately, Tornado may remove their own I/O loop
87. 1.0 (2009): callback-based 2.1 (2011): Generator-based 3.0 (2013): concurrent.Futures 4.3 (Nov 2015): Python 3.5 coroutines
88. Tornado is a great example of interoperation
89. An example for Twisted?
90. In where Amber discovers that interoperation is hard
91. asyncio is similar but not the same
92. My focus: async/await
93. Introduced in Python 3.5 Detailed in PEP-0492
94. async def read_data(db): data = await db.fetch('SELECT ...') ...
95. await gets the result of a coroutine coroutines are a special kind of generator
96. Similar to yield from, delegates to subgenerator Asynchronous code executed in a synchronous style Yielding for other things while it waits for a new result
97. @inlineCallbacks def loadData(url): response = yield makeRequest(url) return json.loads(response) Twisted has had a trampoline to turn Deferreds into a generator since 2006
98. Coming soon: @deferredCoroutine
99. await on Deferreds coroutines wrapped in Deferreds
100. import treq from twisted.internet.defer import deferredCoroutine from twisted.internet.task import react @deferredCoroutine async def main(reactor): pages = [ "http://google.com/", "http://twistedmatrix.com", ] results = {} for page in pages: results[page] = await treq.content(await treq.get(page)) print(results) react(main)
101. Coming soon: asyncioreactor twistd --reactor=asyncio
102. Twisted reactor on top of asyncio
103. Coming soon: deferredToFuture
104. import asyncio import uvloop asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) from twisted.internet.asyncioreactor import install install() import treq from aiohttp import web from twisted.internet import reactor, defer reactor.startRunning(True) user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1' async def handle(request): url = "http://www.google.com" + request.path_qs headers = dict(request.headers) try: headers.pop('HOST') headers.pop('CONNECTION') except: pass get = await defer.deferredToFuture(treq.get(url, headers=headers)) text = await defer.deferredToFuture(treq.content(get)) return web.Response( body=text, content_type=get.headers.getRawHeaders("content-type")[0]) app = web.Application() app.router.add_route('GET', '/{tail:.*}', handle) web.run_app(app)
105. text = await defer.deferredToFuture(treq.content(get))
106. Not reviewed/merged Requires asyncio patches
107. Why Twisted is still worth using
108. Released often
109. 3+ times a year 2016 is set to have 5 releases
110. Time based releases, taken off our trunk branch
111. Need the cutting edge? Our trunk branch is stable!
112. Lots of protocols, out of the box!
113. HTTP/1.0+1.1, SMTP, DNS, IRC, NMEA, FTP, SSH2, Memcache, Finger, Telnet, SOCKS, POP3, IMAP4
114. Super easy to make your own protocols!
115. from twisted.internet import reactor from twisted.internet.endpoints import serverFromString from twisted.internet.protocol import Factory from twisted.protocols.basic import LineReceiver class MyProtocol(LineReceiver): def lineReceived(self, line): print("I got this line!") print(line) factory = Factory.forProtocol(MyProtocol) endpoint = serverFromString(reactor, "tcp:7000") endpoint.listen(factory) reactor.run()
116. HTTP/2 coming soon!
117. Established library support
118. Example: txacme and txsni Python interface to Let’s Encrypt Automatic certificate renewal
119. Example: hendrix WSGI runner on top of Twisted Websockets, TLS, run Twisted code github.com/hendrix/hendrix
120. Example: Autobahn|Python Websockets for Twisted and asyncio WS + WAMP, a RPC & PubSub framework Super fast under PyPy! I work on it :D github.com/crossbario/autobahnpython
121. We're a dependable base
122. We try not to break your code
123. Deprecation cycles mean you have a year to notice that we’re removing something
124. Upgrade with impunity!
125. Code review Automated testing Thousands of tests
126. Fast, just add PyPy!
128. Many officially supported platforms
129. Officially supported means the tests pass, and must pass before branches are merged
130. Ubuntu 12.04/14.04/15.04/15.10 Debian 8 CentOS 7 Fedora 22/23 FreeBSD 10.1 Windows 7 OS X 10.10
131. Python 2.7 (all platforms) Python 3.4 (Linux, FreeBSD) Python 3.5 (Linux, FreeBSD) Earlier versions support 2.6 and 3.3
132. PyPy is close, only a handful of tests remain, nearly all are due to CPython assumptions
133. Python 3.4/3.5 is coming to Windows soon!
134. Competition is good!
135. Twisted and Tornado fit in this ecosystem, if only as competitors
136. Where to from here?
137. Twisted calling asyncio asyncio calling Twisted
138. async-sig mailing list
139. See: "Building Protocol Libraries the Right Way" by Cory Benfield "Thinking in Coroutines" by Lukasz Langa
140. Questions? (pls no statements, save them for after)