Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Cooperative Multitasking with Twisted: Getting Things Done Concurrently
Search
David Reid
February 01, 2012
Programming
1
580
Cooperative Multitasking with Twisted: Getting Things Done Concurrently
David Reid
February 01, 2012
Tweet
Share
Other Decks in Programming
See All in Programming
VSCodeでのDatabricks開発もお勧めしたい/I would also recommend Databricks development with VSCode.
kazumain
0
260
SwiftUIで使いやすいToastの作り方 / How to build a Toast system which is easy to use in SwiftUI
lovee
3
150
効率化に挑戦してみたらモバイル開発が少し快適になった話
ryunakayama
0
130
SIMD Parallel Programming with the Vector API
josepaumard
0
180
Goのエラースタックトレースの歴史と今後
sonatard
9
1.5k
Ruby Function Composition
bkuhlmann
1
330
Elm Form Validation
bkuhlmann
0
510
AWS CDKコントリビュートTIPS / aws-cdk-contribution-tips
gotok365
2
200
Ruby GitHub Packages
bkuhlmann
0
630
新宿ダンジョンを可視化してみた
satoshi7190
2
260
Netty Chicago Java User Group 2024-04-17
sullis
0
180
GitHub Actionsで泣かないためにやっておきたい設定 / Recommended GHA settings to avoid crying
pinkumohikan
3
540
Featured
See All Featured
VelocityConf: Rendering Performance Case Studies
addyosmani
320
23k
The Cost Of JavaScript in 2023
addyosmani
16
3.9k
Imperfection Machines: The Place of Print at Facebook
scottboms
260
12k
The Art of Programming - Codeland 2020
erikaheidi
42
12k
Being A Developer After 40
akosma
57
580k
Teambox: Starting and Learning
jrom
128
8.4k
The Invisible Customer
myddelton
114
12k
The MySQL Ecosystem @ GitHub 2015
samlambert
243
12k
Statistics for Hackers
jakevdp
789
220k
4 Signs Your Business is Dying
shpigford
175
21k
KATA
mclloyd
15
12k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
501
140k
Transcript
Cooperative Multitasking with Twisted “Getting Things Done Concurrently” David Reid
<
[email protected]
>
Alternate Title
Alternate Title How I learned to stop worrying (about threads)
and use timed events to yield to the reactor so that synchronous tasks can cooperate with each other in a single thread.
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Things will block
Fine in moderation
So what blocks?
Non-twisted networking
Non-twisted networking urllib & httplib smtplib python-memcached
Non-twisted networking twisted.web.client twisted.mail.smtp twisted.protocols.memcache
C Extensions
C Extensions DB-APIs: psycopg2, sqlite Hard math: PyCrypto socket.gethostbyname
C Extensions twisted.enterprise deferToThread deferToThread/twisted.names
File I/O
File I/O Yeah ... pretty much all of it.
The for loop.
def do_stuff(data): for x in data: do_work(x)
def do_stuff(data): for x in data: do_work(x)
Next: Introducing Twisted in 3 slides.
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
Deferreds Data Source result or failure Deferred
Deferreds Data Source result or failure Deferred res
Deferreds Data Source result or failure Deferred err
Deferreds Data Source result or failure Deferred
Deferreds Data Source result or failure Deferred err
Deferreds Data Source result or failure Deferred res
reactor.callLater
reactor.callLater reactor.callLater(10, do_stuff)
reactor.callLater reactor.callLater(0.001, do_stuff)
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
twisted.internet.task.Cooperator
twisted.internet.task.Cooperator twisted.internet.task.coiterate
twisted.internet.task.Cooperator twisted.internet.task.coiterate twisted.internet.task.cooperate
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
• Similar to coiterate • Returns a CooperativeTask instead of
a Deferred cooperate
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
• Your code is going to block. • Iterating over
stuff is a big place where synchronous code blocks for unacceptable periods of time. • Iterating is easily broken up by timed events. • Twisted comes with APIs for making this easy. • These APIs are the building blocks for your ideal task system.
Questions (Mine, not yours.)
Why?
Why? Being able to do more stuff is sometimes more
important than doing stuff quickly. * * Making a single user’s request 10ms faster is great, but not if you can only handle 1 request at a time.
Why call later instead of next iteration?
Why call later instead of next iteration? Timed events are
more flexible and you need them anyway.
Can’t I just use deferToThread for python code?
Can’t I just use deferToThread for python code? Yes, but
the GIL hates you.
Can’t I just use deferToThread for python code? Yes, but
the GIL hates you. It’s OK, it hates me too.
Can’t I just run code in a separate processes?
Can’t I just run code in a separate processes? Yes,
look at Ampoule.
Can’t I just run code in a separate processes? Yes,
look at Ampoule. But you probably want those processes to do more than one thing too.
This isn’t appropriate for my situation at all.
This isn’t appropriate for my situation at all. So don’t
do it.
This isn’t appropriate for my situation at all. So don’t
do it. Seriously.
SERIOUSLY.
• Twisted (http://twistedmatrix.com) • Ampoule (https://launchpad.net/ampoule) • txRDQ (https://launchpad.net/txrdq) •
Mochi Media (WE’RE HIRING) • http://mochimedia.com • http://bit.ly/mochi-jobs References