Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Report of Twisted's Death; or Twisted & Tornado in the Asyncio Age (EuroPython 2016)

The Report of Twisted's Death; or Twisted & Tornado in the Asyncio Age (EuroPython 2016)

Amber Brown (HawkOwl)

July 18, 2016
Tweet

More Decks by Amber Brown (HawkOwl)

Other Decks in Programming

Transcript

  1. @hawkieowl "The Report of Twisted's Death" Binary release management across

    3 distros Ported Autobahn|Python (Tx) and Crossbar.io to Python 3 Web API/REST integration in CB
  2. @hawkieowl "The Report of Twisted's Death" In Python, threads or

    processes won’t help with C10K (10,000 concurrent connections)
  3. @hawkieowl "The Report of Twisted's Death" Threads are hard to:

    use safely (without race conditions) scale with (1 thread per connection)
  4. @hawkieowl "The Report of Twisted's Death" select() and friends (poll,

    epoll, kqueue) (or IOCP, but that's different)
  5. @hawkieowl "The Report of Twisted's Death" Selector functions take a

    list of file descriptors (e.g. sockets, open files) and tell you what is ready for reading or writing
  6. @hawkieowl "The Report of Twisted's Death" Data is channeled through

    a transport to a protocol implementation (e.g. HTTP)
  7. @hawkieowl "The Report of Twisted's Death" Nothing blocks! It just

    waits until the network is ready for more data to be sent, or more data has arrived
  8. @hawkieowl "The Report of Twisted's Death" Higher density per core

    No threads required! Concurrency, not parallelism
  9. @hawkieowl "The Report of Twisted's Death" Best case: high I/O

    throughput, high-latency clients, low CPU processing
  10. @hawkieowl "The Report of Twisted's Death" They provide an object

    which is a standin for some future result, and a way of being notified of that result
  11. @hawkieowl "The Report of Twisted's Death" Deferreds run callbacks as

    soon as they are able Futures schedule a callback for the next reactor loop
  12. @hawkieowl "The Report of Twisted's Death" no gevent or eventlet

    little Twisted ported most of Tornado ported
  13. @hawkieowl "The Report of Twisted's Death" Python 3 adoption was

    chugging along, but there wasn't anything unique to it
  14. @hawkieowl "The Report of Twisted's Death" import asyncio import datetime

    async def display_date(loop): end_time = loop.time() + 5.0 while True: print(datetime.datetime.now()) if (loop.time() + 1.0) >= end_time: break await asyncio.sleep(1) loop = asyncio.get_event_loop() # Blocking call which returns when the display_date() # coroutine is done loop.run_until_complete(display_date(loop)) loop.close()
  15. “It should be easy for (Python 3.3 ports of) frameworks

    like Twisted, Tornado, or even gevent to either adapt the default event loop implementation to their needs using a lightweight adapter or proxy, or to replace the default event loop implementation with an adaptation of their own event loop implementation.” “Interoperability - asyncio”
 https://www.python.org/dev/peps/pep-3156/
  16. “For this interoperability to be effective, the preferred direction of

    adaptation in third party frameworks is to keep the default event loop and adapt it to the framework's API. Ideally all third party frameworks would give up their own event loop implementation in favor of the standard implementation.” “Interoperability - asyncio”
 https://www.python.org/dev/peps/pep-3156/
  17. @hawkieowl "The Report of Twisted's Death" cooperative, single-threaded multitasking primitives

    for supporting asynchronous programming (Futures are like Deferreds, coroutines are like inlineCallbacks)
  18. @hawkieowl "The Report of Twisted's Death" Same system APIs select,

    poll, epoll(Linux), kqueue (BSDs), IOCP(Windows)
  19. “Twisted is an async I/O thing, asyncio is an async

    I/O thing. Therefore they are the same kind of thing. I only need one kind of thing in each category of thing. Therefore I only need one of them, and the “standard” one is probably the better one to depend on. So I guess nobody will need Twisted any more!” https://glyph.twistedmatrix.com/2014/05/the-report-of-our-death.html
  20. @hawkieowl "The Report of Twisted's Death" 0 75000 150000 225000

    300000 Twisted 16.1 asyncio 3,352 107,612 21,902 176,927 Code (lines) Comments (lines) Lines of Code (Python & C) With Tests
  21. @hawkieowl "The Report of Twisted's Death" 0 35000 70000 105000

    140000 Twisted 16.1 asyncio 2,355 54,242 8,452 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
  22. @hawkieowl "The Report of Twisted's Death" Twisted did a lot

    of things because none of these things were otherwise available
  23. @hawkieowl "The Report of Twisted's Death" 0 35000 70000 105000

    140000 Twisted 16.1 Django 1.9 25,625 54,242 74,033 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
  24. @hawkieowl "The Report of Twisted's Death" Asynchronous I/O primitives Tools

    for doing async I/O Python utilities Protocols using all of the above
  25. @hawkieowl "The Report of Twisted's Death" 0 7500 15000 22500

    30000 Twisted 16.1
 (asyncio parity) asyncio
 (+ concurrent.futures) 2,712 8,434 9,143 13,722 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests
  26. @hawkieowl "The Report of Twisted's Death" Implements its own I/O

    loop Twisted & asyncio integration (yield Deferreds or Futures)
  27. @hawkieowl "The Report of Twisted's Death" 1.0 (2009): callback-based 2.1

    (2011): Generator-based 3.0 (2013): concurrent.Futures 4.3 (Nov 2015): Python 3.5 coroutines
  28. @hawkieowl "The Report of Twisted's Death" await gets the result

    of a coroutine coroutines are a special kind of generator
  29. @hawkieowl "The Report of Twisted's Death" Similar to yield from,

    delegates to subgenerator Asynchronous code executed in a synchronous style Yielding for other things while it waits for a new result
  30. @hawkieowl "The Report of Twisted's Death" @inlineCallbacks def loadData(url): response

    = yield makeRequest(url) return json.loads(response) Twisted has had a trampoline to turn Deferreds into a generator since 2006
  31. @hawkieowl "The Report of Twisted's Death" import treq from twisted.internet.defer

    import ensureDeferred from twisted.internet.task import react async def main(reactor): pages = [ "http://google.com/", "http://twistedmatrix.com", ] results = {} for page in pages: results[page] = await treq.content(await treq.get(page)) print(results) react(lambda r: ensureDeferred(main(r)))
  32. @hawkieowl "The Report of Twisted's Death" import asyncio import uvloop

    asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) from twisted.internet.asyncioreactor import install install() import treq from aiohttp import web from twisted.internet import reactor, defer reactor.startRunning(True) user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1' async def handle(request): url = "http://www.google.com" + request.path_qs headers = dict(request.headers) try: headers.pop('HOST') headers.pop('CONNECTION') except: pass get = await defer.deferredToFuture(treq.get(url, headers=headers)) text = await defer.deferredToFuture(treq.content(get)) return web.Response( body=text, content_type=get.headers.getRawHeaders("content-type")[0]) app = web.Application() app.router.add_route('GET', '/{tail:.*}', handle) web.run_app(app)
  33. @hawkieowl "The Report of Twisted's Death" HTTP/1, HTTP/2, SMTP, DNS,

    IRC, NMEA, FTP, SSH2, Memcache, Finger, Telnet, SOCKS, POP3, IMAP4 Python 2+3, Python 2
  34. @hawkieowl "The Report of Twisted's Death" from twisted.internet import reactor

    from twisted.internet.endpoints import serverFromString from twisted.internet.protocol import Factory from twisted.protocols.basic import LineReceiver class MyProtocol(LineReceiver): def lineReceived(self, line): print("I got this line!") print(line) factory = Factory.forProtocol(MyProtocol) endpoint = serverFromString(reactor, "tcp:7000") endpoint.listen(factory) reactor.run()
  35. @hawkieowl "The Report of Twisted's Death" Example: txacme and txsni

    Python interface to Let’s Encrypt Automatic certificate renewal
  36. @hawkieowl "The Report of Twisted's Death" Example: hendrix WSGI runner

    on top of Twisted Websockets, TLS, run Twisted code github.com/hendrix/hendrix
  37. @hawkieowl "The Report of Twisted's Death" Example: Autobahn|Python Websockets for

    Twisted and asyncio WS + WAMP, a RPC & PubSub framework Super fast under PyPy! I work on it :D github.com/crossbario/autobahnpython
  38. @hawkieowl "The Report of Twisted's Death" Deprecation cycles mean you

    have a year to notice that we’re removing something
  39. @hawkieowl "The Report of Twisted's Death" Officially supported means the

    tests pass, and must pass before branches are merged
  40. @hawkieowl "The Report of Twisted's Death" Ubuntu 12.04/14.04/16.04 Debian 8

    CentOS 7 Fedora 22/23 Windows 7 Windows Server 2012 R2 OS X 10.10
  41. @hawkieowl "The Report of Twisted's Death" Python 2.7 (all platforms)

    Python 3.4 (Linux) Python 3.5 (Linux) Earlier versions support 2.6 and 3.3
  42. @hawkieowl "The Report of Twisted's Death" PyPy is close, only

    a handful of tests remain, nearly all are due to CPython assumptions
  43. @hawkieowl "The Report of Twisted's Death" Twisted and Tornado fit

    in this ecosystem, if only as competitors
  44. @hawkieowl "The Report of Twisted's Death" See: "Building Protocol Libraries

    the Right Way" by Cory Benfield @ PyCon US "Thinking in Coroutines" by Lukasz Langa @ PyCon US