Cooperative Multitasking with Twisted: Getting Things Done Concurrently

Transcript

1. Cooperative Multitasking with Twisted “Getting Things Done Concurrently” David Reid <dreid@dreid.org>
2. Alternate Title
3. Alternate Title How I learned to stop worrying (about threads) and use timed events to yield to the reactor so that synchronous tasks can cooperate with each other in a single thread.
4. Cooperative Multitasking
5. Cooperative Multitasking
6. Cooperative Multitasking
7. Cooperative Multitasking
8. Cooperative Multitasking
9. Cooperative Multitasking
10. Things will block
11. Fine in moderation
12. So what blocks?
13. Non-twisted networking
14. Non-twisted networking urllib & httplib smtplib python-memcached
15. Non-twisted networking twisted.web.client twisted.mail.smtp twisted.protocols.memcache
16. C Extensions
17. C Extensions DB-APIs: psycopg2, sqlite Hard math: PyCrypto socket.gethostbyname
18. C Extensions twisted.enterprise deferToThread deferToThread/twisted.names
19. File I/O
20. File I/O Yeah ... pretty much all of it.
21. The for loop.
22. def do_stuff(data): for x in data: do_work(x)
23. def do_stuff(data): for x in data: do_work(x)
24. Next: Introducing Twisted in 3 slides.
25. while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The Reactor: Approximately
26. while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The Reactor: Approximately
27. while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The Reactor: Approximately
28. while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The Reactor: Approximately
29. Deferreds Data Source result or failure Deferred
30. Deferreds Data Source result or failure Deferred res
31. Deferreds Data Source result or failure Deferred err
32. Deferreds Data Source result or failure Deferred
33. Deferreds Data Source result or failure Deferred err
34. Deferreds Data Source result or failure Deferred res
35. reactor.callLater
36. reactor.callLater reactor.callLater(10, do_stuff)
37. reactor.callLater reactor.callLater(0.001, do_stuff)
38. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
39. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
40. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
41. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
42. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
43. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
44. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
45. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d
46. twisted.internet.task.Cooperator
47. twisted.internet.task.Cooperator twisted.internet.task.coiterate
48. twisted.internet.task.Cooperator twisted.internet.task.coiterate twisted.internet.task.cooperate
49. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return coiterate(doterator())
50. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return coiterate(doterator())
51. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return coiterate(doterator())
52. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return coiterate(doterator())
53. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return coiterate(doterator())
54. • Similar to coiterate • Returns a CooperativeTask instead of a Deferred cooperate
55. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator())
56. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator())
57. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator())
58. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator())
59. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
60. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
61. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
62. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
63. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
64. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
65. def do_stuff(data): def doterator(): for x in data: do_work(x) yield None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()
66. • Your code is going to block. • Iterating over stuff is a big place where synchronous code blocks for unacceptable periods of time. • Iterating is easily broken up by timed events. • Twisted comes with APIs for making this easy. • These APIs are the building blocks for your ideal task system.
67. Questions (Mine, not yours.)
68. Why?
69. Why? Being able to do more stuff is sometimes more important than doing stuff quickly. * * Making a single user’s request 10ms faster is great, but not if you can only handle 1 request at a time.
70. Why call later instead of next iteration?
71. Why call later instead of next iteration? Timed events are more flexible and you need them anyway.
72. Can’t I just use deferToThread for python code?
73. Can’t I just use deferToThread for python code? Yes, but the GIL hates you.
74. Can’t I just use deferToThread for python code? Yes, but the GIL hates you. It’s OK, it hates me too.
75. Can’t I just run code in a separate processes?
76. Can’t I just run code in a separate processes? Yes, look at Ampoule.
77. Can’t I just run code in a separate processes? Yes, look at Ampoule. But you probably want those processes to do more than one thing too.
78. This isn’t appropriate for my situation at all.
79. This isn’t appropriate for my situation at all. So don’t do it.
80. This isn’t appropriate for my situation at all. So don’t do it. Seriously.
81. SERIOUSLY.
82. • Twisted (http://twistedmatrix.com) • Ampoule (https://launchpad.net/ampoule) • txRDQ (https://launchpad.net/txrdq) • Mochi Media (WE’RE HIRING) • http://mochimedia.com • http://bit.ly/mochi-jobs References