Cooperative Multitasking with Twisted: Getting Things Done Concurrently

Transcript

1. Cooperative Multitasking with Twisted “Getting Things Done Concurrently” David Reid

   <[email protected]>

2. Alternate Title

3. Alternate Title How I learned to stop worrying (about threads)

   and use timed events to yield to the reactor so that synchronous tasks can cooperate with each other in a single thread.

4. Cooperative Multitasking

5. Cooperative Multitasking

6. Cooperative Multitasking

7. Cooperative Multitasking

8. Cooperative Multitasking

9. Cooperative Multitasking

10. Things will block

11. Fine in moderation

12. So what blocks?

13. Non-twisted networking

14. Non-twisted networking urllib & httplib smtplib python-memcached

15. Non-twisted networking twisted.web.client twisted.mail.smtp twisted.protocols.memcache

16. C Extensions

17. C Extensions DB-APIs: psycopg2, sqlite Hard math: PyCrypto socket.gethostbyname

18. C Extensions twisted.enterprise deferToThread deferToThread/twisted.names

19. File I/O

20. File I/O Yeah ... pretty much all of it.

21. The for loop.

22. def do_stuff(data): for x in data: do_work(x)

23. def do_stuff(data): for x in data: do_work(x)

24. Next: Introducing Twisted in 3 slides.

25. while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The

    Reactor: Approximately

26. while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The

    Reactor: Approximately

27. while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The

    Reactor: Approximately

28. while true: stuff = wait_for_stuff(timeout) if stuff: check_reads_and_writes() run_timed_events() The

    Reactor: Approximately

29. Deferreds Data Source result or failure Deferred

30. Deferreds Data Source result or failure Deferred res

31. Deferreds Data Source result or failure Deferred err

32. Deferreds Data Source result or failure Deferred

33. Deferreds Data Source result or failure Deferred err

34. Deferreds Data Source result or failure Deferred res

35. reactor.callLater

36. reactor.callLater reactor.callLater(10, do_stuff)

37. reactor.callLater reactor.callLater(0.001, do_stuff)

38. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not

    data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d

39. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not

    data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d

40. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not

    data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d

41. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not

    data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d

42. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not

    data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d

43. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not

    data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d

44. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not

    data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d

45. def do_stuff(data): d = Deferred() def do_later(): do_work(data.pop()) if not

    data: d.callback(True) else: reactor.callLater(0.1, do_later) reactor.callLater(0.1, do_later) return d

46. twisted.internet.task.Cooperator

47. twisted.internet.task.Cooperator twisted.internet.task.coiterate

48. twisted.internet.task.Cooperator twisted.internet.task.coiterate twisted.internet.task.cooperate

49. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return coiterate(doterator())

50. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return coiterate(doterator())

51. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return coiterate(doterator())

52. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return coiterate(doterator())

53. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return coiterate(doterator())

54. • Similar to coiterate • Returns a CooperativeTask instead of

    a Deferred cooperate

55. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator())

56. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator())

57. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator())

58. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator())

59. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()

60. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()

61. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()

62. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()

63. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()

64. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()

65. def do_stuff(data): def doterator(): for x in data: do_work(x) yield

    None return cooperate(doterator()) task = do_stuff(someData) task_d = task.whenDone() task.pause() task.resume() task.stop()

66. • Your code is going to block. • Iterating over

    stuff is a big place where synchronous code blocks for unacceptable periods of time. • Iterating is easily broken up by timed events. • Twisted comes with APIs for making this easy. • These APIs are the building blocks for your ideal task system.

67. Questions (Mine, not yours.)

68. Why?

69. Why? Being able to do more stuff is sometimes more

    important than doing stuff quickly. * * Making a single user’s request 10ms faster is great, but not if you can only handle 1 request at a time.

70. Why call later instead of next iteration?

71. Why call later instead of next iteration? Timed events are

    more flexible and you need them anyway.

72. Can’t I just use deferToThread for python code?

73. Can’t I just use deferToThread for python code? Yes, but

    the GIL hates you.

74. Can’t I just use deferToThread for python code? Yes, but

    the GIL hates you. It’s OK, it hates me too.

75. Can’t I just run code in a separate processes?

76. Can’t I just run code in a separate processes? Yes,

    look at Ampoule.

77. Can’t I just run code in a separate processes? Yes,

    look at Ampoule. But you probably want those processes to do more than one thing too.

78. This isn’t appropriate for my situation at all.

79. This isn’t appropriate for my situation at all. So don’t

    do it.

80. This isn’t appropriate for my situation at all. So don’t

    do it. Seriously.

81. SERIOUSLY.

82. • Twisted (http://twistedmatrix.com) • Ampoule (https://launchpad.net/ampoule) • txRDQ (https://launchpad.net/txrdq) •

    Mochi Media (WE’RE HIRING) • http://mochimedia.com • http://bit.ly/mochi-jobs References