Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Amber Brown - The Report Of Twisted’s Death or: Why Twisted and Tornado Are Relevant In The Asyncio Age

Amber Brown - The Report Of Twisted’s Death or: Why Twisted and Tornado Are Relevant In The Asyncio Age

With asyncio on the scene, the question has been asked: is there any point in having Twisted or Tornado around?

https://us.pycon.org/2016/schedule/presentation/2114/

PyCon 2016

May 29, 2016
Tweet

More Decks by PyCon 2016

Other Decks in Programming

Transcript

  1. The Report of Twisted’s Death
    …or why Twisted and Tornado are
    relevant in the asyncio age

    View Slide

  2. Hello, I’m
    Amber Brown
    (HawkOwl)

    View Slide

  3. I live in Perth, Western Australia

    View Slide

  4. View Slide

  5. Core Developer
    Release Manager
    Ported 40KLoC+ to Python 3

    View Slide

  6. (image by isometri.cc)

    View Slide

  7. View Slide

  8. Binary release management
    across 3 distros
    Ported Autobahn|Python (Tx)
    and Crossbar.io to Python 3
    Web API/REST integration in CB

    View Slide

  9. Russell Keith-Magee Glyph Lefkowitz
    Credit where credit is due…

    View Slide

  10. https://goo.gl/89FqCM
    "The Report Of Our Death"
    by Glyph

    View Slide

  11. So you want to
    do some I/O…

    View Slide

  12. View Slide

  13. Synchronous I/O frameworks
    (Django, Pyramid, Flask)
    serve one request at a time

    View Slide

  14. Deployed with runners that
    run many copies using
    threads or processes

    View Slide

  15. In Python,
    threads or processes
    won’t help with C10K
    (10,000 concurrent connections)

    View Slide

  16. View Slide

  17. View Slide

  18. Threads are hard to:
    use safely
    (without race conditions)
    scale with
    (1 thread per connection)

    View Slide

  19. Thread memory overhead:
    32kB to 8MB per thread

    View Slide

  20. 128kB of per-thread stack
    x 10,000 threads
    = 1.3GiB of overhead

    View Slide

  21. Python’s Global Interpreter Lock
    ==
    no parallelism

    View Slide

  22. You will not do
    threads properly.

    View Slide

  23. Microthreads/green threads
    (eventlet, gevent)
    are no better

    View Slide

  24. https://goo.gl/7xE4d1
    "Unyielding", by Glyph

    View Slide

  25. Non-Threaded
    Asynchronous I/O

    View Slide

  26. The approach of Twisted,
    Tornado, asyncio, curio

    View Slide

  27. Twisted was one of the first

    View Slide

  28. (of SVN history)

    View Slide

  29. asyncio was introduced
    much later

    View Slide

  30. View Slide

  31. Identical system call at their
    core: Selector functions

    View Slide

  32. select() and friends
    (poll, epoll, kqueue)
    (or IOCP, but that's different)

    View Slide

  33. Selector functions take a
    list of file descriptors
    (e.g. sockets, open files)
    and tell you what is ready
    for reading or writing

    View Slide

  34. Selector loops can handle
    thousands of open sockets
    and events

    View Slide


  35. View Slide

  36. Data is channeled through a
    transport to a protocol
    implementation (e.g. HTTP)

    View Slide

  37. Sending data is queued until
    the network is ready

    View Slide

  38. Nothing blocks!
    It just waits until the network
    is ready for more data to be
    sent, or more data has arrived

    View Slide

  39. “I/O loops”
    or
    “reactors”
    (after the “reactor pattern”)

    View Slide

  40. Higher density per core
    No threads required!
    Concurrency, not parallelism

    View Slide

  41. Best case:
    high I/O throughput,
    high-latency clients,
    low CPU processing

    View Slide

  42. You’re probably waiting on
    the client or the database

    View Slide

  43. Some implementations come
    with abstractions to help you

    View Slide

  44. They provide an object which
    is a standin for some future
    result, and a way of being
    notified of that result

    View Slide

  45. Twisted uses a Deferred
    asyncio uses a Future

    View Slide

  46. d = Deferred()
    d.addCallback(print)
    # some time later...
    d.callback("Hi!")

    View Slide

  47. f = Future()
    f.add_done_callback(print)
    # some time later...
    f.set_result("Hi!")

    View Slide

  48. Deferreds run callbacks as
    soon as they are able
    Futures schedule a callback for
    the next reactor loop

    View Slide

  49. Why a new solution?

    View Slide

  50. asynchronous I/O in Python
    circa 2012 was a total mess

    View Slide

  51. no gevent or eventlet
    little Twisted ported
    most of Tornado ported

    View Slide

  52. Elsewhere:
    node.js exploding in popularity
    async/await shipped
    in .NET 4.5

    View Slide

  53. Python 3 needed its
    “killer feature”

    View Slide

  54. Why asyncio?

    View Slide

  55. A framework designed around
    “coroutines” from the start

    View Slide

  56. coroutines are a
    special generator

    View Slide

  57. Python 3.5 contains syntax
    that makes Futures act like
    coroutines

    View Slide

  58. import asyncio
    import datetime
    async def display_date(loop):
    end_time = loop.time() + 5.0
    while True:
    print(datetime.datetime.now())
    if (loop.time() + 1.0) >= end_time:
    break
    await asyncio.sleep(1)
    loop = asyncio.get_event_loop()
    # Blocking call which returns when the display_date()
    # coroutine is done
    loop.run_until_complete(display_date(loop))
    loop.close()

    View Slide

  59. asyncio.sleep(1) returns a
    Future which is awaited on in
    the coroutine

    View Slide

  60. Repairing library API
    fragmentation

    View Slide

  61. “It should be easy for (Python 3.3 ports
    of) frameworks like Twisted, Tornado, or
    even gevent to either adapt the default
    event loop implementation to their needs
    using a lightweight adapter or proxy, or
    to replace the default event loop
    implementation with an adaptation of
    their own event loop implementation.”
    “Interoperability - asyncio”

    https://www.python.org/dev/peps/pep-3156/

    View Slide

  62. Reducing duplication

    View Slide

  63. “For this interoperability to be effective,
    the preferred direction of adaptation in
    third party frameworks is to keep the
    default event loop and adapt it to the
    framework's API. Ideally all third party
    frameworks would give up their own
    event loop implementation in favor of
    the standard implementation.”
    “Interoperability - asyncio”

    https://www.python.org/dev/peps/pep-3156/

    View Slide

  64. So doesn’t asyncio
    replace Twisted?

    View Slide

  65. cooperative, single-threaded
    multitasking
    primitives for supporting
    asynchronous programming
    (Futures are like Deferreds,
    coroutines are like inlineCallbacks)

    View Slide

  66. Same system APIs
    select, poll, epoll(Linux),
    kqueue (BSDs), IOCP(Windows)

    View Slide

  67. Protocols and transports are
    directly inspired by Twisted

    View Slide

  68. I/O loop is architecturally
    similar to Twisted’s

    View Slide

  69. Newer, “standard” API
    Just there in Python 3.4+!

    View Slide

  70. “Twisted is an async I/O thing, asyncio
    is an async I/O thing. Therefore they are
    the same kind of thing. I only need one
    kind of thing in each category of thing.
    Therefore I only need one of them, and
    the “standard” one is probably the
    better one to depend on. So I guess
    nobody will need Twisted any more!”
    https://glyph.twistedmatrix.com/2014/05/the-report-of-our-death.html

    View Slide

  71. asyncio is an apple
    Twisted is a fruit salad

    View Slide

  72. 0
    75000
    150000
    225000
    300000
    Twisted 16.1 asyncio
    3,352
    107,612
    21,902
    176,927
    Code (lines) Comments (lines)
    Lines of Code (Python & C)
    With Tests

    View Slide

  73. 0
    35000
    70000
    105000
    140000
    Twisted 16.1 asyncio
    2,355
    54,242
    8,452
    74,250
    Code (lines) Comments (lines)
    Lines of Code (Python & C)
    Without Tests

    View Slide

  74. Twisted did a lot of things
    because none of these things
    were otherwise available

    View Slide

  75. One big package was easier to:
    distribute
    install
    use

    View Slide

  76. 0
    35000
    70000
    105000
    140000
    Twisted 16.1 Django 1.9
    25,625
    54,242
    74,033
    74,250
    Code (lines) Comments (lines)
    Lines of Code (Python & C)
    Without Tests

    View Slide

  77. Asynchronous I/O primitives
    Tools for doing async I/O
    Python utilities
    Protocols using all of the above

    View Slide

  78. 0
    7500
    15000
    22500
    30000
    Twisted 16.1

    (asyncio parity)
    asyncio

    (+ concurrent.futures)
    2,712
    8,434
    9,143
    13,722
    Code (lines) Comments (lines)
    Lines of Code (Python & C)
    Without Tests

    View Slide

  79. twisted.internet
    twisted.web.http/wsgi
    twisted.words.protocols.irc
    twisted.protocols.memcache
    twisted.names.resolve
    twisted.conch.ssh
    twisted.mail.smtp
    twisted.words.protocols.jabber
    twisted.python.threadpool
    asyncio
    aiohttp
    irc3
    aiomemcached
    aiodns
    AsyncSSH
    aiosmtpd
    aioxmpp
    concurrent.futures

    View Slide

  80. Twisted also contains
    protocols that don’t have
    asyncio implementations

    View Slide

  81. Tornado

    View Slide

  82. Asynchronous web
    framework in Python
    by FriendFeed/Facebook

    View Slide

  83. IOStream is similar to
    Twisted/asyncio Transports

    View Slide

  84. Protocols are not as
    well defined

    View Slide

  85. Implements its own I/O loop
    Twisted & asyncio integration
    (yield Deferreds or Futures)

    View Slide

  86. Ultimately, Tornado may
    remove their own I/O loop

    View Slide

  87. 1.0 (2009): callback-based
    2.1 (2011): Generator-based
    3.0 (2013): concurrent.Futures
    4.3 (Nov 2015): Python 3.5 coroutines

    View Slide

  88. Tornado is a great example
    of interoperation

    View Slide

  89. An example for Twisted?

    View Slide

  90. In where Amber discovers
    that interoperation is hard

    View Slide

  91. asyncio is similar but
    not the same

    View Slide

  92. My focus: async/await

    View Slide

  93. Introduced in Python 3.5
    Detailed in PEP-0492

    View Slide

  94. async def read_data(db):
    data = await db.fetch('SELECT ...')
    ...

    View Slide

  95. await gets the result
    of a coroutine
    coroutines are a special
    kind of generator

    View Slide

  96. Similar to yield from,
    delegates to subgenerator
    Asynchronous code executed in a
    synchronous style
    Yielding for other things while it
    waits for a new result

    View Slide

  97. @inlineCallbacks
    def loadData(url):
    response = yield makeRequest(url)
    return json.loads(response)
    Twisted has had a trampoline
    to turn Deferreds into a
    generator since 2006

    View Slide

  98. Coming soon:
    @deferredCoroutine

    View Slide

  99. await on Deferreds
    coroutines wrapped in Deferreds

    View Slide

  100. import treq
    from twisted.internet.defer import deferredCoroutine
    from twisted.internet.task import react
    @deferredCoroutine
    async def main(reactor):
    pages = [
    "http://google.com/",
    "http://twistedmatrix.com",
    ]
    results = {}
    for page in pages:
    results[page] = await treq.content(await treq.get(page))
    print(results)
    react(main)

    View Slide

  101. Coming soon:
    asyncioreactor
    twistd --reactor=asyncio

    View Slide

  102. Twisted reactor on
    top of asyncio

    View Slide

  103. Coming soon:
    deferredToFuture

    View Slide

  104. import asyncio
    import uvloop
    asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
    from twisted.internet.asyncioreactor import install
    install()
    import treq
    from aiohttp import web
    from twisted.internet import reactor, defer
    reactor.startRunning(True)
    user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1'
    async def handle(request):
    url = "http://www.google.com" + request.path_qs
    headers = dict(request.headers)
    try:
    headers.pop('HOST')
    headers.pop('CONNECTION')
    except:
    pass
    get = await defer.deferredToFuture(treq.get(url, headers=headers))
    text = await defer.deferredToFuture(treq.content(get))
    return web.Response(
    body=text,
    content_type=get.headers.getRawHeaders("content-type")[0])
    app = web.Application()
    app.router.add_route('GET', '/{tail:.*}', handle)
    web.run_app(app)

    View Slide

  105. text = await defer.deferredToFuture(treq.content(get))

    View Slide

  106. Not reviewed/merged
    Requires asyncio patches

    View Slide

  107. Why Twisted is still
    worth using

    View Slide

  108. Released often

    View Slide

  109. 3+ times a year
    2016 is set to have 5 releases

    View Slide

  110. Time based releases, taken
    off our trunk branch

    View Slide

  111. Need the cutting edge?
    Our trunk branch is stable!

    View Slide

  112. Lots of protocols,
    out of the box!

    View Slide

  113. HTTP/1.0+1.1, SMTP, DNS,
    IRC, NMEA, FTP, SSH2,
    Memcache, Finger, Telnet,
    SOCKS, POP3, IMAP4

    View Slide

  114. Super easy to make
    your own protocols!

    View Slide

  115. from twisted.internet import reactor
    from twisted.internet.endpoints import serverFromString
    from twisted.internet.protocol import Factory
    from twisted.protocols.basic import LineReceiver
    class MyProtocol(LineReceiver):
    def lineReceived(self, line):
    print("I got this line!")
    print(line)
    factory = Factory.forProtocol(MyProtocol)
    endpoint = serverFromString(reactor, "tcp:7000")
    endpoint.listen(factory)
    reactor.run()

    View Slide

  116. HTTP/2 coming soon!

    View Slide

  117. Established library support

    View Slide

  118. Example:
    txacme and txsni
    Python interface to Let’s Encrypt
    Automatic certificate renewal

    View Slide

  119. Example:
    hendrix
    WSGI runner on top of Twisted
    Websockets, TLS, run Twisted code
    github.com/hendrix/hendrix

    View Slide

  120. Example:
    Autobahn|Python
    Websockets for Twisted and asyncio
    WS + WAMP, a RPC & PubSub framework
    Super fast under PyPy!
    I work on it :D
    github.com/crossbario/autobahnpython

    View Slide

  121. We're a dependable base

    View Slide

  122. We try not to break your code

    View Slide

  123. Deprecation cycles mean you
    have a year to notice that
    we’re removing something

    View Slide

  124. Upgrade with impunity!

    View Slide

  125. Code review
    Automated testing
    Thousands of tests

    View Slide

  126. Fast, just add PyPy!

    View Slide

  127. View Slide

  128. Many officially supported
    platforms

    View Slide

  129. Officially supported means
    the tests pass, and must pass
    before branches are merged

    View Slide

  130. Ubuntu 12.04/14.04/15.04/15.10
    Debian 8
    CentOS 7
    Fedora 22/23
    FreeBSD 10.1
    Windows 7
    OS X 10.10

    View Slide

  131. Python 2.7 (all platforms)
    Python 3.4 (Linux, FreeBSD)
    Python 3.5 (Linux, FreeBSD)
    Earlier versions support
    2.6 and 3.3

    View Slide

  132. PyPy is close, only a handful
    of tests remain, nearly all are
    due to CPython assumptions

    View Slide

  133. Python 3.4/3.5 is coming to
    Windows soon!

    View Slide

  134. Competition is good!

    View Slide

  135. Twisted and Tornado fit in
    this ecosystem, if only as
    competitors

    View Slide

  136. Where to from here?

    View Slide

  137. Twisted calling asyncio
    asyncio calling Twisted

    View Slide

  138. async-sig mailing list

    View Slide

  139. See:
    "Building Protocol Libraries
    the Right Way"
    by Cory Benfield
    "Thinking in Coroutines" by Lukasz Langa

    View Slide

  140. Questions?
    (pls no statements, save them for after)

    View Slide