Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Cooperative Multitasking with Twisted: Getting ...
Search
David Reid
February 01, 2012
Programming
1
740
Cooperative Multitasking with Twisted: Getting Things Done Concurrently
David Reid
February 01, 2012
Tweet
Share
Other Decks in Programming
See All in Programming
PostgreSQLのRow Level SecurityをPHPのORMで扱う Eloquent vs Doctrine #phpcon #track2
77web
2
390
ReadMoreTextView
fornewid
1
480
エンジニア向け採用ピッチ資料
inusan
0
160
Is Xcode slowly dying out in 2025?
uetyo
1
210
Google Agent Development Kit でLINE Botを作ってみた
ymd65536
2
200
datadog dash 2025 LLM observability for reliability and stability
ivry_presentationmaterials
0
120
5つのアンチパターンから学ぶLT設計
narihara
1
120
Select API from Kotlin Coroutine
jmatsu
1
190
LT 2025-06-30: プロダクトエンジニアの役割
yamamotok
0
540
プロダクト志向ってなんなんだろうね
righttouch
PRO
0
160
Modern Angular with Signals and Signal Store:New Rules for Your Architecture @enterJS Advanced Angular Day 2025
manfredsteyer
PRO
0
130
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
430
Featured
See All Featured
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
For a Future-Friendly Web
brad_frost
179
9.8k
How GitHub (no longer) Works
holman
314
140k
VelocityConf: Rendering Performance Case Studies
addyosmani
330
24k
Six Lessons from altMBA
skipperchong
28
3.9k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
60k
Automating Front-end Workflow
addyosmani
1370
200k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
The World Runs on Bad Software
bkeepers
PRO
69
11k
How to train your dragon (web standard)
notwaldorf
94
6.1k
Scaling GitHub
holman
459
140k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Transcript
Cooperative Multitasking with Twisted “Getting Things Done Concurrently” David Reid
<
[email protected]
>
Alternate Title
Alternate Title How I learned to stop worrying (about threads)
and use timed events to yield to the reactor so that synchronous tasks can cooperate with each other in a single thread.
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Things will block
Fine in moderation
So what blocks?
Non-twisted networking
Non-twisted networking urllib & httplib smtplib python-memcached
Non-twisted networking twisted.web.client twisted.mail.smtp twisted.protocols.memcache
C Extensions
C Extensions DB-APIs: psycopg2, sqlite Hard math: PyCrypto socket.gethostbyname
C Extensions twisted.enterprise deferToThread deferToThread/twisted.names
File I/O
File I/O Yeah ... pretty much all of it.
The for loop.
def do_stuff(data): for x in data: do_work(x)
def do_stuff(data): for x in data: do_work(x)
Next: Introducing Twisted in 3 slides.
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
Deferreds Data Source result or failure Deferred
Deferreds Data Source result or failure Deferred res
Deferreds Data Source result or failure Deferred err
Deferreds Data Source result or failure Deferred
Deferreds Data Source result or failure Deferred err
Deferreds Data Source result or failure Deferred res
reactor.callLater
reactor.callLater reactor.callLater(10, do_stuff)
reactor.callLater reactor.callLater(0.001, do_stuff)
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
twisted.internet.task.Cooperator
twisted.internet.task.Cooperator twisted.internet.task.coiterate
twisted.internet.task.Cooperator twisted.internet.task.coiterate twisted.internet.task.cooperate
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
• Similar to coiterate • Returns a CooperativeTask instead of
a Deferred cooperate
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
• Your code is going to block. • Iterating over
stuff is a big place where synchronous code blocks for unacceptable periods of time. • Iterating is easily broken up by timed events. • Twisted comes with APIs for making this easy. • These APIs are the building blocks for your ideal task system.
Questions (Mine, not yours.)
Why?
Why? Being able to do more stuff is sometimes more
important than doing stuff quickly. * * Making a single user’s request 10ms faster is great, but not if you can only handle 1 request at a time.
Why call later instead of next iteration?
Why call later instead of next iteration? Timed events are
more flexible and you need them anyway.
Can’t I just use deferToThread for python code?
Can’t I just use deferToThread for python code? Yes, but
the GIL hates you.
Can’t I just use deferToThread for python code? Yes, but
the GIL hates you. It’s OK, it hates me too.
Can’t I just run code in a separate processes?
Can’t I just run code in a separate processes? Yes,
look at Ampoule.
Can’t I just run code in a separate processes? Yes,
look at Ampoule. But you probably want those processes to do more than one thing too.
This isn’t appropriate for my situation at all.
This isn’t appropriate for my situation at all. So don’t
do it.
This isn’t appropriate for my situation at all. So don’t
do it. Seriously.
SERIOUSLY.
• Twisted (http://twistedmatrix.com) • Ampoule (https://launchpad.net/ampoule) • txRDQ (https://launchpad.net/txrdq) •
Mochi Media (WE’RE HIRING) • http://mochimedia.com • http://bit.ly/mochi-jobs References