Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Report of Twisted's Death; or Twisted & Tornado in the Asyncio Age (EuroPython 2016)

The Report of Twisted's Death; or Twisted & Tornado in the Asyncio Age (EuroPython 2016)

3d37232726396a1d3c7412dd915095ea?s=128

Amber Brown (HawkOwl)

July 18, 2016
Tweet

Transcript

  1. The Report of Twisted’s Death …or why Twisted and Tornado

    are relevant in the asyncio age
  2. Hello, I’m Amber Brown (HawkOwl)

  3. @hawkieowl atleastfornow.net

  4. I live in Perth, Western Australia

  5. None
  6. None
  7. @hawkieowl "The Report of Twisted's Death" Core Developer Release Manager

    Ported 40KLoC+ to Python 3
  8. (image by isometri.cc)

  9. None
  10. @hawkieowl "The Report of Twisted's Death" Binary release management across

    3 distros Ported Autobahn|Python (Tx) and Crossbar.io to Python 3 Web API/REST integration in CB
  11. Glyph Lefkowitz Credit where credit is due… Russell Keith-Magee

  12. @hawkieowl "The Report of Twisted's Death" https://goo.gl/89FqCM "The Report Of

    Our Death" by Glyph
  13. So you want to do some I/O…

  14. None
  15. @hawkieowl "The Report of Twisted's Death" Synchronous I/O frameworks (Django,

    Pyramid, Flask) serve one request at a time
  16. @hawkieowl "The Report of Twisted's Death" Deployed with runners that

    run many copies using threads or processes
  17. @hawkieowl "The Report of Twisted's Death" In Python, threads or

    processes won’t help with C10K (10,000 concurrent connections)
  18. @hawkieowl "The Report of Twisted's Death"

  19. @hawkieowl "The Report of Twisted's Death"

  20. @hawkieowl "The Report of Twisted's Death" Threads are hard to:

    use safely (without race conditions) scale with (1 thread per connection)
  21. @hawkieowl "The Report of Twisted's Death" Thread memory overhead: 32kB

    to 8MB per thread
  22. @hawkieowl "The Report of Twisted's Death" 128kB of per-thread stack

    x 10,000 threads = 1.3GiB of overhead
  23. @hawkieowl "The Report of Twisted's Death" Python’s Global Interpreter Lock

    == no parallelism
  24. @hawkieowl "The Report of Twisted's Death" You will not do

    threads properly.
  25. @hawkieowl "The Report of Twisted's Death" Microthreads/green threads (eventlet, gevent)

    are no better
  26. @hawkieowl "The Report of Twisted's Death" https://goo.gl/7xE4d1 "Unyielding", by Glyph

  27. Non-Threaded Asynchronous I/O

  28. @hawkieowl "The Report of Twisted's Death" The approach of Twisted,

    Tornado, asyncio, curio
  29. @hawkieowl "The Report of Twisted's Death" Twisted was one of

    the first
  30. @hawkieowl "The Report of Twisted's Death" SVN Git CVS

  31. @hawkieowl "The Report of Twisted's Death" asyncio was introduced much

    later
  32. @hawkieowl "The Report of Twisted's Death"

  33. @hawkieowl "The Report of Twisted's Death" Identical system call at

    their core: Selector functions
  34. @hawkieowl "The Report of Twisted's Death" select() and friends (poll,

    epoll, kqueue) (or IOCP, but that's different)
  35. @hawkieowl "The Report of Twisted's Death" Selector functions take a

    list of file descriptors (e.g. sockets, open files) and tell you what is ready for reading or writing
  36. @hawkieowl "The Report of Twisted's Death" Selector loops can handle

    thousands of open sockets and events
  37. @hawkieowl "The Report of Twisted's Death" Twisted 16.3 PyPy 5.3

    OS X 10.11
  38. @hawkieowl "The Report of Twisted's Death" Data is channeled through

    a transport to a protocol implementation (e.g. HTTP)
  39. @hawkieowl "The Report of Twisted's Death" Sending data is queued

    until the network is ready
  40. @hawkieowl "The Report of Twisted's Death" Nothing blocks! It just

    waits until the network is ready for more data to be sent, or more data has arrived
  41. @hawkieowl "The Report of Twisted's Death" “I/O loops” or “reactors”

    (after the “reactor pattern”)
  42. @hawkieowl "The Report of Twisted's Death" Higher density per core

    No threads required! Concurrency, not parallelism
  43. @hawkieowl "The Report of Twisted's Death" Best case: high I/O

    throughput, high-latency clients, low CPU processing
  44. @hawkieowl "The Report of Twisted's Death" You’re probably waiting on

    the client or the database
  45. @hawkieowl "The Report of Twisted's Death" Some implementations come with

    abstractions to help you
  46. @hawkieowl "The Report of Twisted's Death" They provide an object

    which is a standin for some future result, and a way of being notified of that result
  47. @hawkieowl "The Report of Twisted's Death" Twisted uses a Deferred

    asyncio uses a Future
  48. @hawkieowl "The Report of Twisted's Death" d = Deferred() d.addCallback(print)

    # some time later... d.callback("Hi!")
  49. @hawkieowl "The Report of Twisted's Death" f = Future() f.add_done_callback(print)

    # some time later... f.set_result("Hi!")
  50. @hawkieowl "The Report of Twisted's Death" Deferreds run callbacks as

    soon as they are able Futures schedule a callback for the next reactor loop
  51. Why a new solution?

  52. @hawkieowl "The Report of Twisted's Death" asynchronous I/O in Python

    circa 2012 was a total mess
  53. @hawkieowl "The Report of Twisted's Death" no gevent or eventlet

    little Twisted ported most of Tornado ported
  54. @hawkieowl "The Report of Twisted's Death" Elsewhere: node.js getting popular

    async/await in .NET 4.5
  55. @hawkieowl "The Report of Twisted's Death" Python 3 adoption was

    chugging along, but there wasn't anything unique to it
  56. Why asyncio?

  57. @hawkieowl "The Report of Twisted's Death" A framework designed around

    “coroutines” from the start
  58. @hawkieowl "The Report of Twisted's Death" coroutines are a special

    generator
  59. @hawkieowl "The Report of Twisted's Death" Python 3.5 contains syntax

    that makes Futures act like coroutines
  60. @hawkieowl "The Report of Twisted's Death" import asyncio import datetime

    async def display_date(loop): end_time = loop.time() + 5.0 while True: print(datetime.datetime.now()) if (loop.time() + 1.0) >= end_time: break await asyncio.sleep(1) loop = asyncio.get_event_loop() # Blocking call which returns when the display_date() # coroutine is done loop.run_until_complete(display_date(loop)) loop.close()
  61. @hawkieowl "The Report of Twisted's Death" asyncio.sleep(1) returns a Future

    which is awaited on in the coroutine
  62. @hawkieowl "The Report of Twisted's Death" Repairing library API fragmentation

  63. “It should be easy for (Python 3.3 ports of) frameworks

    like Twisted, Tornado, or even gevent to either adapt the default event loop implementation to their needs using a lightweight adapter or proxy, or to replace the default event loop implementation with an adaptation of their own event loop implementation.” “Interoperability - asyncio”
 https://www.python.org/dev/peps/pep-3156/
  64. @hawkieowl "The Report of Twisted's Death" Reducing duplication

  65. “For this interoperability to be effective, the preferred direction of

    adaptation in third party frameworks is to keep the default event loop and adapt it to the framework's API. Ideally all third party frameworks would give up their own event loop implementation in favor of the standard implementation.” “Interoperability - asyncio”
 https://www.python.org/dev/peps/pep-3156/
  66. So doesn’t asyncio replace Twisted?

  67. @hawkieowl "The Report of Twisted's Death" cooperative, single-threaded multitasking primitives

    for supporting asynchronous programming (Futures are like Deferreds, coroutines are like inlineCallbacks)
  68. @hawkieowl "The Report of Twisted's Death" Same system APIs select,

    poll, epoll(Linux), kqueue (BSDs), IOCP(Windows)
  69. @hawkieowl "The Report of Twisted's Death" Protocols and transports are

    directly inspired by Twisted
  70. @hawkieowl "The Report of Twisted's Death" I/O loop is architecturally

    similar to Twisted’s
  71. @hawkieowl "The Report of Twisted's Death" Newer, “standard” API Just

    there in Python 3.4+!
  72. “Twisted is an async I/O thing, asyncio is an async

    I/O thing. Therefore they are the same kind of thing. I only need one kind of thing in each category of thing. Therefore I only need one of them, and the “standard” one is probably the better one to depend on. So I guess nobody will need Twisted any more!” https://glyph.twistedmatrix.com/2014/05/the-report-of-our-death.html
  73. asyncio is an apple Twisted is a fruit salad

  74. @hawkieowl "The Report of Twisted's Death" 0 75000 150000 225000

    300000 Twisted 16.1 asyncio 3,352 107,612 21,902 176,927 Code (lines) Comments (lines) Lines of Code (Python & C) With Tests
  75. @hawkieowl "The Report of Twisted's Death" 0 35000 70000 105000

    140000 Twisted 16.1 asyncio 2,355 54,242 8,452 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
  76. @hawkieowl "The Report of Twisted's Death" Twisted did a lot

    of things because none of these things were otherwise available
  77. @hawkieowl "The Report of Twisted's Death" One big package was

    easier to: distribute install use
  78. @hawkieowl "The Report of Twisted's Death" 0 35000 70000 105000

    140000 Twisted 16.1 Django 1.9 25,625 54,242 74,033 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
  79. @hawkieowl "The Report of Twisted's Death" Asynchronous I/O primitives Tools

    for doing async I/O Python utilities Protocols using all of the above
  80. @hawkieowl "The Report of Twisted's Death" 0 7500 15000 22500

    30000 Twisted 16.1
 (asyncio parity) asyncio
 (+ concurrent.futures) 2,712 8,434 9,143 13,722 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
  81. twisted.internet twisted.web.http/wsgi twisted.words.protocols.irc twisted.protocols.memcache twisted.names.resolve twisted.conch.ssh twisted.mail.smtp twisted.words.protocols.jabber twisted.python.threadpool asyncio

    aiohttp irc3 aiomemcached aiodns AsyncSSH aiosmtpd aioxmpp concurrent.futures
  82. @hawkieowl "The Report of Twisted's Death" Twisted also contains protocols

    that don’t have asyncio implementations
  83. Tornado

  84. @hawkieowl "The Report of Twisted's Death" Asynchronous web framework in

    Python by FriendFeed/Facebook
  85. @hawkieowl "The Report of Twisted's Death" IOStream is similar to

    Twisted/asyncio Transports
  86. @hawkieowl "The Report of Twisted's Death" Protocols are not as

    well defined
  87. @hawkieowl "The Report of Twisted's Death" Implements its own I/O

    loop Twisted & asyncio integration (yield Deferreds or Futures)
  88. @hawkieowl "The Report of Twisted's Death" Ultimately, Tornado may remove

    their own I/O loop
  89. @hawkieowl "The Report of Twisted's Death" 1.0 (2009): callback-based 2.1

    (2011): Generator-based 3.0 (2013): concurrent.Futures 4.3 (Nov 2015): Python 3.5 coroutines
  90. @hawkieowl "The Report of Twisted's Death" Tornado is a great

    example of interoperation
  91. @hawkieowl "The Report of Twisted's Death" An example for Twisted?

  92. In where Amber discovers that interoperation is hard

  93. @hawkieowl "The Report of Twisted's Death" asyncio is similar but

    not the same
  94. @hawkieowl "The Report of Twisted's Death" My focus: async/await

  95. @hawkieowl "The Report of Twisted's Death" Introduced in Python 3.5

    Detailed in PEP-0492
  96. @hawkieowl "The Report of Twisted's Death" async def read_data(db): data

    = await db.fetch('SELECT ...') ...
  97. @hawkieowl "The Report of Twisted's Death" await gets the result

    of a coroutine coroutines are a special kind of generator
  98. @hawkieowl "The Report of Twisted's Death" Similar to yield from,

    delegates to subgenerator Asynchronous code executed in a synchronous style Yielding for other things while it waits for a new result
  99. @hawkieowl "The Report of Twisted's Death" @inlineCallbacks def loadData(url): response

    = yield makeRequest(url) return json.loads(response) Twisted has had a trampoline to turn Deferreds into a generator since 2006
  100. @hawkieowl "The Report of Twisted's Death" Coming soon: @ensureDeferred

  101. @hawkieowl "The Report of Twisted's Death" Coroutine -> Deferred The

    coroutine can await on Deferreds!
  102. @hawkieowl "The Report of Twisted's Death" import treq from twisted.internet.defer

    import ensureDeferred from twisted.internet.task import react async def main(reactor): pages = [ "http://google.com/", "http://twistedmatrix.com", ] results = {} for page in pages: results[page] = await treq.content(await treq.get(page)) print(results) react(lambda r: ensureDeferred(main(r)))
  103. @hawkieowl "The Report of Twisted's Death" Coming soon: asyncioreactor twistd

    --reactor=asyncio
  104. @hawkieowl "The Report of Twisted's Death" Twisted reactor on top

    of asyncio
  105. @hawkieowl "The Report of Twisted's Death" Coming soon: deferredToFuture

  106. @hawkieowl "The Report of Twisted's Death" import asyncio import uvloop

    asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) from twisted.internet.asyncioreactor import install install() import treq from aiohttp import web from twisted.internet import reactor, defer reactor.startRunning(True) user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1' async def handle(request): url = "http://www.google.com" + request.path_qs headers = dict(request.headers) try: headers.pop('HOST') headers.pop('CONNECTION') except: pass get = await defer.deferredToFuture(treq.get(url, headers=headers)) text = await defer.deferredToFuture(treq.content(get)) return web.Response( body=text, content_type=get.headers.getRawHeaders("content-type")[0]) app = web.Application() app.router.add_route('GET', '/{tail:.*}', handle) web.run_app(app)
  107. @hawkieowl "The Report of Twisted's Death" text = await defer.deferredToFuture(treq.content(get))

  108. @hawkieowl "The Report of Twisted's Death" Not reviewed/merged Requires asyncio

    patches
  109. Why Twisted is still worth using

  110. Released often

  111. @hawkieowl "The Report of Twisted's Death" 3+ times a year

    2016 is set to have 5 releases
  112. @hawkieowl "The Report of Twisted's Death" Time based releases, taken

    off our trunk branch
  113. @hawkieowl "The Report of Twisted's Death" Need the cutting edge?

    Our trunk branch is stable!
  114. Lots of protocols, out of the box!

  115. @hawkieowl "The Report of Twisted's Death" HTTP/1, HTTP/2, SMTP, DNS,

    IRC, NMEA, FTP, SSH2, Memcache, Finger, Telnet, SOCKS, POP3, IMAP4 Python 2+3, Python 2
  116. @hawkieowl "The Report of Twisted's Death" Super easy to make

    your own protocols!
  117. @hawkieowl "The Report of Twisted's Death" from twisted.internet import reactor

    from twisted.internet.endpoints import serverFromString from twisted.internet.protocol import Factory from twisted.protocols.basic import LineReceiver class MyProtocol(LineReceiver): def lineReceived(self, line): print("I got this line!") print(line) factory = Factory.forProtocol(MyProtocol) endpoint = serverFromString(reactor, "tcp:7000") endpoint.listen(factory) reactor.run()
  118. @hawkieowl "The Report of Twisted's Death" HTTP/2 in 16.3!

  119. Established library support

  120. @hawkieowl "The Report of Twisted's Death" Example: txacme and txsni

    Python interface to Let’s Encrypt Automatic certificate renewal
  121. @hawkieowl "The Report of Twisted's Death" Example: hendrix WSGI runner

    on top of Twisted Websockets, TLS, run Twisted code github.com/hendrix/hendrix
  122. @hawkieowl "The Report of Twisted's Death" Example: Autobahn|Python Websockets for

    Twisted and asyncio WS + WAMP, a RPC & PubSub framework Super fast under PyPy! I work on it :D github.com/crossbario/autobahnpython
  123. We're a dependable base

  124. @hawkieowl "The Report of Twisted's Death" We try not to

    break your code
  125. @hawkieowl "The Report of Twisted's Death" Deprecation cycles mean you

    have a year to notice that we’re removing something
  126. @hawkieowl "The Report of Twisted's Death" Upgrade with impunity!

  127. @hawkieowl "The Report of Twisted's Death" Code review Automated testing

    Thousands of tests
  128. Fast, just add PyPy!

  129. @hawkieowl "The Report of Twisted's Death"

  130. Many officially supported platforms

  131. @hawkieowl "The Report of Twisted's Death" Officially supported means the

    tests pass, and must pass before branches are merged
  132. @hawkieowl "The Report of Twisted's Death" Ubuntu 12.04/14.04/16.04 Debian 8

    CentOS 7 Fedora 22/23 Windows 7 Windows Server 2012 R2 OS X 10.10
  133. @hawkieowl "The Report of Twisted's Death" Python 2.7 (all platforms)

    Python 3.4 (Linux) Python 3.5 (Linux) Earlier versions support 2.6 and 3.3
  134. @hawkieowl "The Report of Twisted's Death" PyPy is close, only

    a handful of tests remain, nearly all are due to CPython assumptions
  135. @hawkieowl "The Report of Twisted's Death" PyPy3 (3.3.5) is getting

    worked on
  136. @hawkieowl "The Report of Twisted's Death" Python 3.4/3.5 is coming

    to Windows soon!
  137. Competition is good!

  138. @hawkieowl "The Report of Twisted's Death" Twisted and Tornado fit

    in this ecosystem, if only as competitors
  139. Where to from here?

  140. @hawkieowl "The Report of Twisted's Death" Twisted calling asyncio asyncio

    calling Twisted
  141. @hawkieowl "The Report of Twisted's Death" async-sig mailing list

  142. @hawkieowl "The Report of Twisted's Death" See: "Building Protocol Libraries

    the Right Way" by Cory Benfield @ PyCon US "Thinking in Coroutines" by Lukasz Langa @ PyCon US
  143. Questions? (pls no statements, save them for after)