The Report of Twisted's Death; or Twisted & Tornado in the Asyncio Age (Perth Django Meetup)

Given at the Perth Django Meetup

Transcript

1. This is an alpha of a talk to be given at PyCon US 2016 Please excuse any mistakes or formatting weirdness :)
2. The Report of Twisted’s Death …or why Twisted and Tornado are relevant in the asyncio age
3. Hello, I’m Amber Brown (HawkOwl)
4. I live in Perth, Western Australia
6. Release Manager since 2013 (10+ major releases) Ported 40,000+ lines of code to Python 3
7. (image by isometri.cc)
9. Binary release management across 3 distros Ported Autobahn|Python and Crossbar.io to Python 3 Web API/REST integration in CB
10. Russell Keith Magee Glyph Lefkowitz Credit where credit is due…
11. So you want to do some I/O…
13. Synchronous I/O frameworks (Django, Pyramid, Flask) serve one request at a time
14. Deployed with runners that run many copies using threads or processes
15. In Python, threads or processes won’t help with C10K (10,000 concurrent connections)
18. Threads are hard to: use safely (without race conditions) scale with (1 thread per connection)
19. Thread memory overhead: 32kB to 8MB per thread
20. 128kB of per-thread stack x 10,000 threads = 1.3GB of overhead
21. Python’s Global Interpreter Lock == no parallelism
22. You will not do threads properly.
23. Microthreads/green threads (eventlet, gevent)
24. You still won’t do those properly, they’re still threads
25. Asynchronous I/O
26. select() and friends (poll, epoll, kqueue)
27. Selector functions take a list of file descriptors (e.g. sockets, open files) and tell you what is ready for reading or writing
28. Selector loops can handle thousands of open sockets and events
29. Hands received data to the protocol implementation
30. Queues sent data from protocols until the network is ready to send it
31. Nothing blocks! It just waits until the network is ready for more data
32. “I/O loops” or “reactors” (after the “reactor pattern”)
33. Higher density per core No threads required! Still no parallelism
34. Best case: high I/O throughput, high-latency clients, low CPU processing
35. You’re probably waiting on the client or the database
36. Twisted and asyncio are two such selector frameworks
37. Twisted was one of the first
39. asyncio was introduced much later
41. Why a new solution?
42. asynchronous I/O in Python circa 2012 was a total mess
43. no gevent or eventlet little Twisted ported most of Tornado ported
44. Elsewhere: node.js exploding in popularity async/await shipped in .NET 4.5
45. Python 3 needed its “killer feature”
46. Why asyncio?
47. A framework designed around “coroutines” from the start
48. coroutines are a special generator
49. import asyncio import datetime async def display_date(loop): end_time = loop.time() + 5.0 while True: print(datetime.datetime.now()) if (loop.time() + 1.0) >= end_time: break await asyncio.sleep(1) loop = asyncio.get_event_loop() # Blocking call which returns when the display_date() # coroutine is done loop.run_until_complete(display_date(loop)) loop.close()
50. Repairing library API fragmentation
51. “It should be easy for (Python 3.3 ports of) frameworks like Twisted, Tornado, or even gevent to either adapt the default event loop implementation to their needs using a lightweight adapter or proxy, or to replace the default event loop implementation with an adaptation of their own event loop implementation.” “Interoperability - asyncio”
 https://www.python.org/dev/peps/pep-3156/
52. Reducing duplication
53. “For this interoperability to be effective, the preferred direction of adaptation in third party frameworks is to keep the default event loop and adapt it to the framework's API. Ideally all third party frameworks would give up their own event loop implementation in favor of the standard implementation.” “Interoperability - asyncio”
 https://www.python.org/dev/peps/pep-3156/
54. So doesn’t asyncio replace Twisted?
55. cooperative, single-threaded multitasking primitives for supporting asynchronous programming (Futures are like Deferreds, coroutines are like inlineCallbacks)
56. Same system APIs select, poll, epoll(Linux), kqueue (BSDs), IOCP(Windows)
57. Protocols and transports are directly inspired by Twisted
58. I/O loop is architecturally similar to Twisted’s
59. Newer, “standard” API Just there in Python 3.4+!
60. “Twisted is an async I/O thing, asyncio is an async I/O thing. Therefore they are the same kind of thing. I only need one kind of thing in each category of thing. Therefore I only need one of them, and the “standard” one is probably the better one to depend on. So I guess nobody will need Twisted any more!” https://glyph.twistedmatrix.com/2014/05/the-report-of-our-death.html
61. asyncio is an apple Twisted is a fruit salad
62. 0 75000 150000 225000 300000 Twisted asyncio 3,352 107,612 21,902 176,927 Code (lines) Comments (lines) Lines of Code (Python & C) With Tests
63. 0 35000 70000 105000 140000 Twisted asyncio 2,355 54,242 8,452 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
64. Twisted did a lot of things because none of these things were otherwise available
65. One big package was easier to: distribute install use
66. 0 35000 70000 105000 140000 Twisted Django 25,625 54,242 74,033 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
67. Asynchronous I/O primitives Tools for doing async I/O Python utilities Protocols using all of the above
68. 0 7500 15000 22500 30000 Twisted
 (asyncio parity) asyncio
 (+ concurrent.futures) 2,712 8,434 9,143 13,722 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
69. twisted.internet twisted.web.http/wsgi twisted.words.protocols.irc twisted.protocols.memcache twisted.names.resolve twisted.conch.ssh twisted.mail.smtp twisted.words.protocols.jabber twisted.python.threadpool asyncio aiohttp irc3 aiomemcached aiodns AsyncSSH aiosmtpd aioxmpp concurrent.futures
70. Twisted also contains protocols that don’t have asyncio implementations
71. Tornado
72. Asynchronous web framework in Python by FriendFeed/Facebook
73. IOStream is similar to Twisted/asyncio Transports
74. Protocols are not as well defined
75. Implements its own I/O loop Twisted & asyncio integration (yield Deferreds or Futures)
76. Ultimately, Tornado may remove their own I/O loop
77. 1.0 (2009): callback-based 2.1 (2011): Generator-based 3.0 (2013): concurrent.Futures 4.3 (Nov 2015): Python 3.5 coroutines
78. Tornado is a great example of interoperation
79. An example for Twisted?
80. In where Amber discovers that interoperation is hard
81. asyncio is similar but not the same
82. My focus: async/await
83. Introduced in Python 3.5 Detailed in PEP-0492
84. async def read_data(db): data = await db.fetch('SELECT ...') ...
85. await gets the result of a coroutine coroutines are a special kind of generator
86. Similar to yield from, delegates to subgenerator Asynchronous code executed in a synchronous style Yielding for other things while it waits for a new result
87. @inlineCallbacks def loadData(url): response = yield makeRequest(url) return json.loads(response) Twisted has had a trampoline to turn Deferreds into a generator since 2006
88. def _inlineCallbacks(result, g, deferred): """ See L{inlineCallbacks}. """ # This function is complicated by the need to prevent unbounded recursion # arising from repeatedly yielding immediately ready deferreds. This while # loop and the waiting variable solve that by manually unfolding the # recursion. waiting = [True, # waiting for result? None] # result while 1: try: # Send the last result back as the result of the yield expression. isFailure = isinstance(result, failure.Failure) if isFailure: result = result.throwExceptionIntoGenerator(g) else: result = g.send(result) except StopIteration as e: # fell off the end, or "return" statement deferred.callback(getattr(e, "value", None)) return deferred except _DefGen_Return as e: # returnValue() was called; time to give a result to the original # Deferred. First though, let's try to identify the potentially # confusing situation which results when returnValue() is # accidentally invoked from a different function, one that wasn't # decorated with @inlineCallbacks. # The traceback starts in this frame (the one for # _inlineCallbacks); the next one down should be the application # code. appCodeTrace = exc_info()[2].tb_next if isFailure: # If we invoked this generator frame by throwing an exception # into it, then throwExceptionIntoGenerator will consume an # additional stack frame itself, so we need to skip that too. appCodeTrace = appCodeTrace.tb_next # Now that we've identified the frame being exited by the # exception, let's figure out if returnValue was called from it # directly. returnValue itself consumes a stack frame, so the # application code will have a tb_next, but it will *not* have a # second tb_next. if appCodeTrace.tb_next.tb_next: # If returnValue was invoked non-local to the frame which it is # exiting, identify the frame that ultimately invoked # returnValue so that we can warn the user, as this behavior is # confusing. ultimateTrace = appCodeTrace while ultimateTrace.tb_next.tb_next: ultimateTrace = ultimateTrace.tb_next filename = ultimateTrace.tb_frame.f_code.co_filename lineno = ultimateTrace.tb_lineno warnings.warn_explicit( "returnValue() in %r causing %r to exit: " "returnValue should only be invoked by functions decorated " "with inlineCallbacks" % ( ultimateTrace.tb_frame.f_code.co_name, appCodeTrace.tb_frame.f_code.co_name), DeprecationWarning, filename, lineno) deferred.callback(e.value) return deferred except: deferred.errback() return deferred if isinstance(result, Deferred): # a deferred was yielded, get the result. def gotResult(r): if waiting[0]: waiting[0] = False waiting[1] = r else: _inlineCallbacks(r, g, deferred) result.addBoth(gotResult) if waiting[0]: # Haven't called back yet, set flag so that we get reinvoked # and return from the loop waiting[0] = False return deferred result = waiting[1] # Reset waiting to initial values for next loop. gotResult uses # waiting, but this isn't a problem because gotResult is only # executed once, and if it hasn't been executed yet, the return # branch above would have been taken. waiting[0] = True waiting[1] = None def inlineCallbacks(f): """ inlineCallbacks helps you write L{Deferred}-using code that looks like a regular sequential function. For example:: @inlineCallbacks def thingummy(): thing = yield makeSomeRequestResultingInDeferred() print(thing) # the result! hoorj! When you call anything that results in a L{Deferred}, you can simply yield it; your generator will automatically be resumed when the Deferred's result is available. The generator will be sent the result of the L{Deferred} with the 'send' method on generators, or if the result was a failure, 'throw'. Things that are not L{Deferred}s may also be yielded, and your generator will be resumed with the same object sent back. This means C{yield} performs an operation roughly equivalent to L{maybeDeferred}. Your inlineCallbacks-enabled generator will return a L{Deferred} object, which will result in the return value of the generator (or will fail with a failure object if your generator raises an unhandled exception). Note that you can't use C{return result} to return a value; use C{returnValue(result)} instead. Falling off the end of the generator, or simply using C{return} will cause the L{Deferred} to have a result of C{None}. Be aware that L{returnValue} will not accept a L{Deferred} as a parameter. If you believe the thing you'd like to return could be a L{Deferred}, do this:: result = yield result returnValue(result) The L{Deferred} returned from your deferred generator may errback if your generator raised an exception:: @inlineCallbacks def thingummy(): thing = yield makeSomeRequestResultingInDeferred() if thing == 'I love Twisted': # will become the result of the Deferred returnValue('TWISTED IS GREAT!') else: # will trigger an errback raise Exception('DESTROY ALL LIFE') If you are using Python 3.3 or later, it is possible to use the C{return} statement instead of L{returnValue}:: @inlineCallbacks def loadData(url): response = yield makeRequest(url) return json.loads(response) """ @wraps(f) def unwindGenerator(*args, **kwargs): try: gen = f(*args, **kwargs) except _DefGen_Return: raise TypeError( "inlineCallbacks requires %r to produce a generator; instead" "caught returnValue being used in a non-generator" % (f,)) if not isinstance(gen, types.GeneratorType): raise TypeError( "inlineCallbacks requires %r to produce a generator; " "instead got %r" % (f, gen)) return _inlineCallbacks(None, gen, Deferred()) return unwindGenerator
89. def inlineCallbacks(f): @wraps(f) def unwindGenerator(*args, **kwargs): gen = f(*args, **kwargs) return _inlineCallbacks(None, gen, Deferred()) return unwindGenerator
90. def inlineCallbacks(f): @wraps(f) def unwindGenerator(*args, **kwargs): gen = f(*args, **kwargs) return _inlineCallbacks(None, gen, Deferred()) return unwindGenerator 1 The wrapped function is called. This returns a generator.
91. def inlineCallbacks(f): @wraps(f) def unwindGenerator(*args, **kwargs): gen = f(*args, **kwargs) return _inlineCallbacks(None, gen, Deferred()) return unwindGenerator 2 _inlineCallbacks is called with an initial result of None, the generator, and a new Deferred.
92. def _inlineCallbacks(result, g, deferred): waiting = [True, None] # waiting for result and the result while 1: try: if isinstance(result, failure.Failure): result = result.throwExceptionIntoGenerator(g) else: result = g.send(result) except StopIteration as e: deferred.callback(getattr(e, "value", None)) return deferred except _DefGen_Return as e: deferred.callback(e.value) return deferred except: deferred.errback() return deferred We send result (None) into the generator, which spins it off and gets to the first Deferred 3
93. deferred.callback(e.value) return deferred except: deferred.errback() return deferred if isinstance(result, Deferred): def gotResult(r): if waiting[0]: waiting = [False, r] else: _inlineCallbacks(r, g, deferred) result.addBoth(gotResult) if waiting[0]: waiting[0] = False return deferred result = waiting[1] waiting = [True, None] We tell the returned Deferred to call gotResult when it fires. 4
94. deferred.callback(e.value) return deferred except: deferred.errback() return deferred if isinstance(result, Deferred): def gotResult(r): if waiting[0]: waiting = [False, r] else: _inlineCallbacks(r, g, deferred) result.addBoth(gotResult) if waiting[0]: waiting[0] = False return deferred result = waiting[1] waiting = [True, None] If we’re waiting for a result (which because this is the first run we are), return the Deferred from the original inlineCallbacks call, and say that we’re not waiting for a result. 5
95. deferred.callback(e.value) return deferred except: deferred.errback() return deferred if isinstance(result, Deferred): def gotResult(r): if waiting[0]: waiting = [False, r] else: _inlineCallbacks(r, g, deferred) result.addBoth(gotResult) if waiting[0]: waiting[0] = False return deferred result = waiting[1] waiting = [True, None] When the Deferred returned by the generator fires, it checks to see if it is waiting for a result. If it is not, it calls _inlineCallbacks again with the new result. 6
96. def _inlineCallbacks(result, g, deferred): waiting = [True, None] # waiting for result and the result while 1: try: if isinstance(result, failure.Failure): result = result.throwExceptionIntoGenerator(g) else: result = g.send(result) except StopIteration as e: deferred.callback(getattr(e, "value", None)) return deferred except _DefGen_Return as e: deferred.callback(e.value) return deferred except: deferred.errback() return deferred Now result is a value we have, so it is sent in to the generator. 7
97. def _inlineCallbacks(result, g, deferred): waiting = [True, None] # waiting for result and the result while 1: try: if isinstance(result, failure.Failure): result = result.throwExceptionIntoGenerator(g) else: result = g.send(result) except StopIteration as e: deferred.callback(getattr(e, "value", None)) return deferred except _DefGen_Return as e: deferred.callback(e.value) return deferred except: deferred.errback() return deferred If this is the final result, we will get either StopIteration, _DefGen_Return, or an exception. 8
98. def _inlineCallbacks(result, g, deferred): waiting = [True, None] # waiting for result and the result while 1: try: if isinstance(result, failure.Failure): result = result.throwExceptionIntoGenerator(g) else: result = g.send(result) except StopIteration as e: deferred.callback(getattr(e, "value", None)) return deferred except _DefGen_Return as e: deferred.callback(e.value) return deferred except: deferred.errback() return deferred StopIteration is when it runs off the “end”, or in Python 3, when you use return in a generator. 9a
99. def _inlineCallbacks(result, g, deferred): waiting = [True, None] # waiting for result and the result while 1: try: if isinstance(result, failure.Failure): result = result.throwExceptionIntoGenerator(g) else: result = g.send(result) except StopIteration as e: deferred.callback(getattr(e, "value", None)) return deferred except _DefGen_Return as e: deferred.callback(e.value) return deferred except: deferred.errback() return deferred On Python 2, you can’t return from a generator. Twisted uses returnValue, which raises this special exception which holds a value. 9b
100. def _inlineCallbacks(result, g, deferred): waiting = [True, None] # waiting for result and the result while 1: try: if isinstance(result, failure.Failure): result = result.throwExceptionIntoGenerator(g) else: result = g.send(result) except StopIteration as e: deferred.callback(getattr(e, "value", None)) return deferred except _DefGen_Return as e: deferred.callback(e.value) return deferred except: deferred.errback() return deferred Sometimes, the generator will raise an exception before it yields its next value. 9c
101. def _inlineCallbacks(result, g, deferred): waiting = [True, None] # waiting for result and the result while 1: try: if isinstance(result, failure.Failure): result = result.throwExceptionIntoGenerator(g) else: result = g.send(result) except StopIteration as e: deferred.callback(getattr(e, "value", None)) return deferred except _DefGen_Return as e: deferred.callback(e.value) return deferred except: deferred.errback() return deferred In all three cases, the Deferred created in the initial inlineCallbacks call is fired with a result (or a failure). 9c 9b 9a
102. If Deferreds were like Futures, this would be the end of it
103. Particularly observant viewers might have noticed some dead code…
104. Deferreds are synchronous constructs Futures are asynchronous constructs
105. class Future: def set_result(self, result): """Mark the future done and set its result. If the future is already done when this method is called, raises InvalidStateError. """ if self._state != _PENDING: raise InvalidStateError('{}: {!r}'.format(self._state, self)) self._result = result self._state = _FINISHED self._schedule_callbacks() def _schedule_callbacks(self): """Internal: Ask the event loop to call all callbacks. The callbacks are scheduled to be called as soon as possible. Also clears the callback list. """ callbacks = self._callbacks[:] if not callbacks: return self._callbacks[:] = [] for callback in callbacks: self._loop.call_soon(callback, self)
106. deferred.callback(e.value) return deferred except: deferred.errback() return deferred if isinstance(result, Deferred): def gotResult(r): if waiting[0]: waiting = [False, r] else: _inlineCallbacks(r, g, deferred) result.addBoth(gotResult) if waiting[0]: waiting[0] = False return deferred result = waiting[1] waiting = [True, None] If the result Deferred already has a result, addBoth will be called synchronously 10
107. deferred.callback(e.value) return deferred except: deferred.errback() return deferred if isinstance(result, Deferred): def gotResult(r): if waiting[0]: waiting = [False, r] else: _inlineCallbacks(r, g, deferred) result.addBoth(gotResult) if waiting[0]: waiting[0] = False return deferred result = waiting[1] waiting = [True, None] We ARE waiting for a result, and we have it, so we set it to say that we don’t, and put the result beside it. 11
108. deferred.callback(e.value) return deferred except: deferred.errback() return deferred if isinstance(result, Deferred): def gotResult(r): if waiting[0]: waiting = [False, r] else: _inlineCallbacks(r, g, deferred) result.addBoth(gotResult) if waiting[0]: waiting[0] = False return deferred result = waiting[1] waiting = [True, None] Because we’re not waiting for a result, this code is ran, and we say that we want a new value. 12
109. def _inlineCallbacks(result, g, deferred): waiting = [True, None] # waiting for result and the result while 1: try: if isinstance(result, failure.Failure): result = result.throwExceptionIntoGenerator(g) else: result = g.send(result) except StopIteration as e: deferred.callback(getattr(e, "value", None)) return deferred except _DefGen_Return as e: deferred.callback(e.value) return deferred except: deferred.errback() return deferred We loop back up, and send the synchronously-got result back into the generator. And so on, so forth. 13
110. Comparatively, the changes for async/await are much the same
111. class Deferred: def __await__(self): awaiter = _MakeDeferredAwaitable(self) return awaiter.__await__() class _MakeDeferredAwaitable(object): """ A subgenerator which is created by L{Deferred.__await__}. """ result = _NO_RESULT def __init__(self, d): self.d = d self.d.addBoth(self._setResult) def _setResult(self, value): self.result = value def __next__(self): if self.result is not _NO_RESULT: raise StopIteration(self.result) return self.d def __await__(self): return self def _awaitTick(result, coro, deferred): try: result = coro.send(result) except StopIteration as e: deferred.callback(e.value) except: deferred.errback() if isinstance(result, Deferred): result.addBoth(_awaitTick, coro=coro, deferred=deferred) return deferred def deferredCoroutine(f): """ A decorator for supporting coroutine-style programming using L{Deferred}s. It implements the awaitable protocol from PEP-0492. When using this decorator, the wrapped function may use the C{await} keyword to suspend execution until the awaited L{Deferred} has fired. For example:: import treq from twisted.internet.defer import deferredCoroutine from twisted.internet.task import react def main(reactor): pages = [ "https://google.com/", "https://twistedmatrix.com", ] d = crawl(pages) d.addCallback(print) return d @deferredCoroutine async def crawl(pages): results = {} for page in pages: results[page] = await treq.content(await treq.get(page)) return results react(main) In the above example, L{treq.get} and L{treq.content} return L{Deferreds}, and the decorated function returns a L{Deferred} itself. """ def wrapped(*args, **kwargs): coro = f(*args, **kwargs) return _awaitTick(None, coro, Deferred()) return wrapped
112. This code allows you to await on Deferreds
113. import treq from twisted.internet.defer import deferredCoroutine from twisted.internet.task import react def main(reactor): pages = [ "https://google.com/", "https://twistedmatrix.com", ] d = crawl(pages) d.addCallback(print) return d @deferredCoroutine async def crawl(pages): results = {} for page in pages: results[page] = await treq.content(await treq.get(page)) return results react(main)
114. Taken six months, two Twisted core developers, and three Twisted contributors
115. No solution for asynchronous iterators or context managers… yet
116. No solution for interop between Futures and Deferreds… yet
117. No fully implemented asyncio + Twisted reactor… yet
118. Why Twisted is still worth using
119. Released often
120. 3+ times a year 2016 is set to have 5 releases
121. Time based releases, taken off our trunk branch
122. Need the cutting edge? Our trunk branch is stable!
123. Lots of protocols, out of the box!
124. HTTP/1.0+1.1, SMTP, DNS, IRC, NMEA, FTP, SSH2
125. Super easy to make your own line-based protocols
126. from twisted.internet import reactor from twisted.internet.endpoints import serverFromString from twisted.internet.protocol import Factory from twisted.protocols.basic import LineReceiver class MyProtocol(LineReceiver): def lineReceived(self, line): print("I got this line!") print(line) factory = Factory.forProtocol(MyProtocol) endpoint = serverFromString(reactor, "tcp:7000") endpoint.listen(factory) reactor.run()
127. Established library support
128. Example: txacme and txsni Python interface to Let’s Encrypt Automatic certificate renewal
129. Example: hendrix WSGI runner on top of Twisted Websockets, TLS, run Twisted code github.com/hendrix/hendrix
130. Example: Autobahn|Python Websockets for Twisted and asyncio WS + WAMP, a RPC & PubSub framework Super fast under PyPy! I work on it :D github.com/crossbario/autobahnpython
131. Robust compatibility policy
132. We try not to break your code
133. Deprecation cycles mean you have a year to notice that we’re removing something
134. Upgrade with impunity!
135. High code quality
136. 88% code & branch coverage Core APIs have 95%+ coverage 100,000+ LoC of tests
137. Code review Automated testing
138. Works excellent on PyPy
140. ~ live demo ~
141. Many officially supported platforms
142. Officially supported means the tests pass, and must pass before branches are merged
143. Ubuntu 12.04/14.04/15.04/15.10 Debian 8 CentOS 7 Fedora 22/23 FreeBSD 10.1 Windows 7 OS X 10.10
144. Python 2.7 (all platforms) Python 3.4 (Linux, FreeBSD) Python 3.5 (Linux, FreeBSD) Earlier versions support 2.6 and 3.3
145. PyPy is close, only a handful of tests remain, nearly all are due to CPython assumptions
146. Python 3.4/3.5 is coming to Windows soon!
147. Competition is good!
148. Twisted and Tornado fit in this ecosystem, if only as competitors
149. Questions? (pls no statements, save them for after)