The Report of Twisted's Death; or Twisted & Tornado in the Asyncio Age (EuroPython 2016)

The Report of Twisted’s Death …or why Twisted and Tornado
are relevant in the asyncio age

Hello, I’m Amber Brown (HawkOwl)

@hawkieowl atleastfornow.net

I live in Perth, Western Australia

@hawkieowl "The Report of Twisted's Death" Core Developer Release Manager
Ported 40KLoC+ to Python 3

(image by isometri.cc)

@hawkieowl "The Report of Twisted's Death" Binary release management across
3 distros Ported Autobahn|Python (Tx) and Crossbar.io to Python 3 Web API/REST integration in CB

Glyph Lefkowitz Credit where credit is due… Russell Keith-Magee

@hawkieowl "The Report of Twisted's Death" https://goo.gl/89FqCM "The Report Of
Our Death" by Glyph

So you want to do some I/O…

@hawkieowl "The Report of Twisted's Death" Synchronous I/O frameworks (Django,
Pyramid, Flask) serve one request at a time

@hawkieowl "The Report of Twisted's Death" Deployed with runners that
run many copies using threads or processes

@hawkieowl "The Report of Twisted's Death" In Python, threads or
processes won’t help with C10K (10,000 concurrent connections)

@hawkieowl "The Report of Twisted's Death"

@hawkieowl "The Report of Twisted's Death" Threads are hard to:
use safely (without race conditions) scale with (1 thread per connection)

@hawkieowl "The Report of Twisted's Death" Thread memory overhead: 32kB
to 8MB per thread

@hawkieowl "The Report of Twisted's Death" 128kB of per-thread stack
x 10,000 threads = 1.3GiB of overhead

@hawkieowl "The Report of Twisted's Death" Python’s Global Interpreter Lock
== no parallelism

@hawkieowl "The Report of Twisted's Death" You will not do
threads properly.

@hawkieowl "The Report of Twisted's Death" Microthreads/green threads (eventlet, gevent)
are no better

@hawkieowl "The Report of Twisted's Death" https://goo.gl/7xE4d1 "Unyielding", by Glyph

Non-Threaded Asynchronous I/O

@hawkieowl "The Report of Twisted's Death" The approach of Twisted,
Tornado, asyncio, curio

@hawkieowl "The Report of Twisted's Death" Twisted was one of
the first

@hawkieowl "The Report of Twisted's Death" SVN Git CVS

@hawkieowl "The Report of Twisted's Death" asyncio was introduced much
later

@hawkieowl "The Report of Twisted's Death" Identical system call at
their core: Selector functions

@hawkieowl "The Report of Twisted's Death" select() and friends (poll,
epoll, kqueue) (or IOCP, but that's diﬀerent)

@hawkieowl "The Report of Twisted's Death" Selector functions take a
list of file descriptors (e.g. sockets, open files) and tell you what is ready for reading or writing

@hawkieowl "The Report of Twisted's Death" Selector loops can handle
thousands of open sockets and events

@hawkieowl "The Report of Twisted's Death" Twisted 16.3 PyPy 5.3
OS X 10.11

@hawkieowl "The Report of Twisted's Death" Data is channeled through
a transport to a protocol implementation (e.g. HTTP)

@hawkieowl "The Report of Twisted's Death" Sending data is queued
until the network is ready

@hawkieowl "The Report of Twisted's Death" Nothing blocks! It just
waits until the network is ready for more data to be sent, or more data has arrived

@hawkieowl "The Report of Twisted's Death" “I/O loops” or “reactors”
(after the “reactor pattern”)

@hawkieowl "The Report of Twisted's Death" Higher density per core
No threads required! Concurrency, not parallelism

@hawkieowl "The Report of Twisted's Death" Best case: high I/O
throughput, high-latency clients, low CPU processing

@hawkieowl "The Report of Twisted's Death" You’re probably waiting on
the client or the database

@hawkieowl "The Report of Twisted's Death" Some implementations come with
abstractions to help you

@hawkieowl "The Report of Twisted's Death" They provide an object
which is a standin for some future result, and a way of being notified of that result

@hawkieowl "The Report of Twisted's Death" Twisted uses a Deferred
asyncio uses a Future

@hawkieowl "The Report of Twisted's Death" d = Deferred() d.addCallback(print)
# some time later... d.callback("Hi!")

@hawkieowl "The Report of Twisted's Death" f = Future() f.add_done_callback(print)
# some time later... f.set_result("Hi!")

@hawkieowl "The Report of Twisted's Death" Deferreds run callbacks as
soon as they are able Futures schedule a callback for the next reactor loop

Why a new solution?

@hawkieowl "The Report of Twisted's Death" asynchronous I/O in Python
circa 2012 was a total mess

@hawkieowl "The Report of Twisted's Death" no gevent or eventlet
little Twisted ported most of Tornado ported

@hawkieowl "The Report of Twisted's Death" Elsewhere: node.js getting popular
async/await in .NET 4.5

@hawkieowl "The Report of Twisted's Death" Python 3 adoption was
chugging along, but there wasn't anything unique to it

Why asyncio?

@hawkieowl "The Report of Twisted's Death" A framework designed around
“coroutines” from the start

@hawkieowl "The Report of Twisted's Death" coroutines are a special
generator

@hawkieowl "The Report of Twisted's Death" Python 3.5 contains syntax
that makes Futures act like coroutines

@hawkieowl "The Report of Twisted's Death" import asyncio import datetime
async def display_date(loop): end_time = loop.time() + 5.0 while True: print(datetime.datetime.now()) if (loop.time() + 1.0) >= end_time: break await asyncio.sleep(1) loop = asyncio.get_event_loop() # Blocking call which returns when the display_date() # coroutine is done loop.run_until_complete(display_date(loop)) loop.close()

@hawkieowl "The Report of Twisted's Death" asyncio.sleep(1) returns a Future
which is awaited on in the coroutine

@hawkieowl "The Report of Twisted's Death" Repairing library API fragmentation

“It should be easy for (Python 3.3 ports of) frameworks
like Twisted, Tornado, or even gevent to either adapt the default event loop implementation to their needs using a lightweight adapter or proxy, or to replace the default event loop implementation with an adaptation of their own event loop implementation.” “Interoperability - asyncio”  https://www.python.org/dev/peps/pep-3156/

@hawkieowl "The Report of Twisted's Death" Reducing duplication

“For this interoperability to be eﬀective, the preferred direction of
adaptation in third party frameworks is to keep the default event loop and adapt it to the framework's API. Ideally all third party frameworks would give up their own event loop implementation in favor of the standard implementation.” “Interoperability - asyncio”  https://www.python.org/dev/peps/pep-3156/

So doesn’t asyncio replace Twisted?

@hawkieowl "The Report of Twisted's Death" cooperative, single-threaded multitasking primitives
for supporting asynchronous programming (Futures are like Deferreds, coroutines are like inlineCallbacks)

@hawkieowl "The Report of Twisted's Death" Same system APIs select,
poll, epoll(Linux), kqueue (BSDs), IOCP(Windows)

@hawkieowl "The Report of Twisted's Death" Protocols and transports are
directly inspired by Twisted

@hawkieowl "The Report of Twisted's Death" I/O loop is architecturally
similar to Twisted’s

@hawkieowl "The Report of Twisted's Death" Newer, “standard” API Just
there in Python 3.4+!

“Twisted is an async I/O thing, asyncio is an async
I/O thing. Therefore they are the same kind of thing. I only need one kind of thing in each category of thing. Therefore I only need one of them, and the “standard” one is probably the better one to depend on. So I guess nobody will need Twisted any more!” https://glyph.twistedmatrix.com/2014/05/the-report-of-our-death.html

asyncio is an apple Twisted is a fruit salad

@hawkieowl "The Report of Twisted's Death" 0 75000 150000 225000
300000 Twisted 16.1 asyncio 3,352 107,612 21,902 176,927 Code (lines) Comments (lines) Lines of Code (Python & C) With Tests

140000 Twisted 16.1 asyncio 2,355 54,242 8,452 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests

@hawkieowl "The Report of Twisted's Death" Twisted did a lot
of things because none of these things were otherwise available

@hawkieowl "The Report of Twisted's Death" One big package was
easier to: distribute install use

140000 Twisted 16.1 Django 1.9 25,625 54,242 74,033 74,250 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests

@hawkieowl "The Report of Twisted's Death" Asynchronous I/O primitives Tools
for doing async I/O Python utilities Protocols using all of the above

30000 Twisted 16.1  (asyncio parity) asyncio  (+ concurrent.futures) 2,712 8,434 9,143 13,722 Code (lines) Comments (lines) Lines of Code (Python & C) Without Tests

twisted.internet twisted.web.http/wsgi twisted.words.protocols.irc twisted.protocols.memcache twisted.names.resolve twisted.conch.ssh twisted.mail.smtp twisted.words.protocols.jabber twisted.python.threadpool asyncio
aiohttp irc3 aiomemcached aiodns AsyncSSH aiosmtpd aioxmpp concurrent.futures

@hawkieowl "The Report of Twisted's Death" Twisted also contains protocols
that don’t have asyncio implementations

Tornado

@hawkieowl "The Report of Twisted's Death" Asynchronous web framework in
Python by FriendFeed/Facebook

@hawkieowl "The Report of Twisted's Death" IOStream is similar to
Twisted/asyncio Transports

@hawkieowl "The Report of Twisted's Death" Protocols are not as
well defined

@hawkieowl "The Report of Twisted's Death" Implements its own I/O
loop Twisted & asyncio integration (yield Deferreds or Futures)

@hawkieowl "The Report of Twisted's Death" Ultimately, Tornado may remove
their own I/O loop

@hawkieowl "The Report of Twisted's Death" 1.0 (2009): callback-based 2.1
(2011): Generator-based 3.0 (2013): concurrent.Futures 4.3 (Nov 2015): Python 3.5 coroutines

@hawkieowl "The Report of Twisted's Death" Tornado is a great
example of interoperation

@hawkieowl "The Report of Twisted's Death" An example for Twisted?

In where Amber discovers that interoperation is hard

@hawkieowl "The Report of Twisted's Death" asyncio is similar but
not the same

@hawkieowl "The Report of Twisted's Death" My focus: async/await

@hawkieowl "The Report of Twisted's Death" Introduced in Python 3.5
Detailed in PEP-0492

@hawkieowl "The Report of Twisted's Death" async def read_data(db): data
= await db.fetch('SELECT ...') ...

@hawkieowl "The Report of Twisted's Death" await gets the result
of a coroutine coroutines are a special kind of generator

@hawkieowl "The Report of Twisted's Death" Similar to yield from,
delegates to subgenerator Asynchronous code executed in a synchronous style Yielding for other things while it waits for a new result

@hawkieowl "The Report of Twisted's Death" @inlineCallbacks def loadData(url): response
= yield makeRequest(url) return json.loads(response) Twisted has had a trampoline to turn Deferreds into a generator since 2006

@hawkieowl "The Report of Twisted's Death" Coming soon: @ensureDeferred

@hawkieowl "The Report of Twisted's Death" Coroutine -> Deferred The
coroutine can await on Deferreds!

@hawkieowl "The Report of Twisted's Death" import treq from twisted.internet.defer
import ensureDeferred from twisted.internet.task import react async def main(reactor): pages = [ "http://google.com/", "http://twistedmatrix.com", ] results = {} for page in pages: results[page] = await treq.content(await treq.get(page)) print(results) react(lambda r: ensureDeferred(main(r)))

@hawkieowl "The Report of Twisted's Death" Coming soon: asyncioreactor twistd
--reactor=asyncio

@hawkieowl "The Report of Twisted's Death" Twisted reactor on top
of asyncio

@hawkieowl "The Report of Twisted's Death" Coming soon: deferredToFuture

@hawkieowl "The Report of Twisted's Death" import asyncio import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) from twisted.internet.asyncioreactor import install install() import treq from aiohttp import web from twisted.internet import reactor, defer reactor.startRunning(True) user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1' async def handle(request): url = "http://www.google.com" + request.path_qs headers = dict(request.headers) try: headers.pop('HOST') headers.pop('CONNECTION') except: pass get = await defer.deferredToFuture(treq.get(url, headers=headers)) text = await defer.deferredToFuture(treq.content(get)) return web.Response( body=text, content_type=get.headers.getRawHeaders("content-type")[0]) app = web.Application() app.router.add_route('GET', '/{tail:.*}', handle) web.run_app(app)

@hawkieowl "The Report of Twisted's Death" text = await defer.deferredToFuture(treq.content(get))

@hawkieowl "The Report of Twisted's Death" Not reviewed/merged Requires asyncio
patches

Why Twisted is still worth using

Released often

@hawkieowl "The Report of Twisted's Death" 3+ times a year
2016 is set to have 5 releases

@hawkieowl "The Report of Twisted's Death" Time based releases, taken
oﬀ our trunk branch

@hawkieowl "The Report of Twisted's Death" Need the cutting edge?
Our trunk branch is stable!

Lots of protocols, out of the box!

@hawkieowl "The Report of Twisted's Death" HTTP/1, HTTP/2, SMTP, DNS,
IRC, NMEA, FTP, SSH2, Memcache, Finger, Telnet, SOCKS, POP3, IMAP4 Python 2+3, Python 2

@hawkieowl "The Report of Twisted's Death" Super easy to make
your own protocols!

@hawkieowl "The Report of Twisted's Death" from twisted.internet import reactor
from twisted.internet.endpoints import serverFromString from twisted.internet.protocol import Factory from twisted.protocols.basic import LineReceiver class MyProtocol(LineReceiver): def lineReceived(self, line): print("I got this line!") print(line) factory = Factory.forProtocol(MyProtocol) endpoint = serverFromString(reactor, "tcp:7000") endpoint.listen(factory) reactor.run()

@hawkieowl "The Report of Twisted's Death" HTTP/2 in 16.3!

Established library support

@hawkieowl "The Report of Twisted's Death" Example: txacme and txsni
Python interface to Let’s Encrypt Automatic certificate renewal

@hawkieowl "The Report of Twisted's Death" Example: hendrix WSGI runner
on top of Twisted Websockets, TLS, run Twisted code github.com/hendrix/hendrix

@hawkieowl "The Report of Twisted's Death" Example: Autobahn|Python Websockets for
Twisted and asyncio WS + WAMP, a RPC & PubSub framework Super fast under PyPy! I work on it :D github.com/crossbario/autobahnpython

We're a dependable base

@hawkieowl "The Report of Twisted's Death" We try not to
break your code

@hawkieowl "The Report of Twisted's Death" Deprecation cycles mean you
have a year to notice that we’re removing something

@hawkieowl "The Report of Twisted's Death" Upgrade with impunity!

@hawkieowl "The Report of Twisted's Death" Code review Automated testing
Thousands of tests

Fast, just add PyPy!

Many oﬀicially supported platforms

@hawkieowl "The Report of Twisted's Death" Oﬀicially supported means the
tests pass, and must pass before branches are merged

@hawkieowl "The Report of Twisted's Death" Ubuntu 12.04/14.04/16.04 Debian 8
CentOS 7 Fedora 22/23 Windows 7 Windows Server 2012 R2 OS X 10.10

@hawkieowl "The Report of Twisted's Death" Python 2.7 (all platforms)
Python 3.4 (Linux) Python 3.5 (Linux) Earlier versions support 2.6 and 3.3

@hawkieowl "The Report of Twisted's Death" PyPy is close, only
a handful of tests remain, nearly all are due to CPython assumptions

@hawkieowl "The Report of Twisted's Death" PyPy3 (3.3.5) is getting
worked on

@hawkieowl "The Report of Twisted's Death" Python 3.4/3.5 is coming
to Windows soon!

Competition is good!

@hawkieowl "The Report of Twisted's Death" Twisted and Tornado fit
in this ecosystem, if only as competitors

Where to from here?

@hawkieowl "The Report of Twisted's Death" Twisted calling asyncio asyncio
calling Twisted

@hawkieowl "The Report of Twisted's Death" async-sig mailing list

@hawkieowl "The Report of Twisted's Death" See: "Building Protocol Libraries
the Right Way" by Cory Benfield @ PyCon US "Thinking in Coroutines" by Lukasz Langa @ PyCon US

Questions? (pls no statements, save them for after)

The Report of Twisted's Death; or Twisted & Tor...

The Report of Twisted's Death; or Twisted & Tornado in the Asyncio Age (EuroPython 2016)

More Decks by Amber Brown (HawkOwl)

Other Decks in Programming

Featured

Transcript