Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Cooperative Multitasking with Twisted: Getting Things Done Concurrently
Search
David Reid
February 01, 2012
Programming
1
590
Cooperative Multitasking with Twisted: Getting Things Done Concurrently
David Reid
February 01, 2012
Tweet
Share
Other Decks in Programming
See All in Programming
freeeのエンジニアが 就活で出そうな コーディングテストを 解説してみる
freee
0
140
Scalable Customer Journey Orchestration (CJO)
lewuathe
0
480
GitLab CI/CD で C#/WPFアプリケーションのテストとインストーラーのビルド・デプロイを自動化する
hacarus
0
590
障害対応を起点としたもっといい開発と運用のサイクル作りのためにできること / Hatena Enginner Seminar #29
polamjag
0
470
Timeline エディター拡張入門
yucchiy
0
440
TypeScriptのパフォーマンス改善
yajihum
12
4.9k
哲学史とモデリング
tanakahisateru
2
380
CREってこういうこと? 体験入社 - 提案資料 - / what-is-cre-trial-employment
shinden
1
620
Docker_OSS_ホスティング入門
satokoki645
0
140
RaaP
ksss
0
140
TypeScriptコードの漸進的改善 / Progressive Improvement of TypeScript Code
medley
1
400
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
0
110
Featured
See All Featured
Documentation Writing (for coders)
carmenintech
60
4k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
7k
Being A Developer After 40
akosma
67
580k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
34
8.9k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
8
3.5k
Gamification - CAS2011
davidbonilla
77
4.6k
Fantastic passwords and where to find them - at NoRuKo
philnash
39
2.5k
What the flash - Photography Introduction
edds
64
11k
Optimizing for Happiness
mojombo
370
69k
The World Runs on Bad Software
bkeepers
PRO
61
6.7k
How To Stay Up To Date on Web Technology
chriscoyier
782
250k
The Art of Programming - Codeland 2020
erikaheidi
43
12k
Transcript
Cooperative Multitasking with Twisted “Getting Things Done Concurrently” David Reid
<
[email protected]
>
Alternate Title
Alternate Title How I learned to stop worrying (about threads)
and use timed events to yield to the reactor so that synchronous tasks can cooperate with each other in a single thread.
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Cooperative Multitasking
Things will block
Fine in moderation
So what blocks?
Non-twisted networking
Non-twisted networking urllib & httplib smtplib python-memcached
Non-twisted networking twisted.web.client twisted.mail.smtp twisted.protocols.memcache
C Extensions
C Extensions DB-APIs: psycopg2, sqlite Hard math: PyCrypto socket.gethostbyname
C Extensions twisted.enterprise deferToThread deferToThread/twisted.names
File I/O
File I/O Yeah ... pretty much all of it.
The for loop.
def do_stuff(data): for x in data: do_work(x)
def do_stuff(data): for x in data: do_work(x)
Next: Introducing Twisted in 3 slides.
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The
Reactor: Approximately
Deferreds Data Source result or failure Deferred
Deferreds Data Source result or failure Deferred res
Deferreds Data Source result or failure Deferred err
Deferreds Data Source result or failure Deferred
Deferreds Data Source result or failure Deferred err
Deferreds Data Source result or failure Deferred res
reactor.callLater
reactor.callLater reactor.callLater(10, do_stuff)
reactor.callLater reactor.callLater(0.001, do_stuff)
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not
data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
twisted.internet.task.Cooperator
twisted.internet.task.Cooperator twisted.internet.task.coiterate
twisted.internet.task.Cooperator twisted.internet.task.coiterate twisted.internet.task.cooperate
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return coiterate(doterator())
• Similar to coiterate • Returns a CooperativeTask instead of
a Deferred cooperate
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator())
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
def do_stuff(data): def doterator(): for x in data: do_work(x) yield
None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
• Your code is going to block. • Iterating over
stuff is a big place where synchronous code blocks for unacceptable periods of time. • Iterating is easily broken up by timed events. • Twisted comes with APIs for making this easy. • These APIs are the building blocks for your ideal task system.
Questions (Mine, not yours.)
Why?
Why? Being able to do more stuff is sometimes more
important than doing stuff quickly. * * Making a single user’s request 10ms faster is great, but not if you can only handle 1 request at a time.
Why call later instead of next iteration?
Why call later instead of next iteration? Timed events are
more flexible and you need them anyway.
Can’t I just use deferToThread for python code?
Can’t I just use deferToThread for python code? Yes, but
the GIL hates you.
Can’t I just use deferToThread for python code? Yes, but
the GIL hates you. It’s OK, it hates me too.
Can’t I just run code in a separate processes?
Can’t I just run code in a separate processes? Yes,
look at Ampoule.
Can’t I just run code in a separate processes? Yes,
look at Ampoule. But you probably want those processes to do more than one thing too.
This isn’t appropriate for my situation at all.
This isn’t appropriate for my situation at all. So don’t
do it.
This isn’t appropriate for my situation at all. So don’t
do it. Seriously.
SERIOUSLY.
• Twisted (http://twistedmatrix.com) • Ampoule (https://launchpad.net/ampoule) • txRDQ (https://launchpad.net/txrdq) •
Mochi Media (WE’RE HIRING) • http://mochimedia.com • http://bit.ly/mochi-jobs References