Modern Python Concurrency

Transcript

1. Modern Python concurrency toward futures and asyncio Andrew Montalenti Doug

   Turnbull Python Charlottesville Meetup September 2, 2014

2. @softwaredoug @amontalenti

3. Doug’s part

4. What’s Concurrency? • Our system’s ability to do more than

   one thing at once I.E.: or simultaneous: web requests database transactions requests to drive requests to databases requests web services user input

5. • run your app with less • simplify • save

   green: $ From 100 web requests per second per server... ...to 10,000 web requests per second per server Getting it right is a Big Deal(R)

6. Concurrency -- How? • Nature of the work? • My

   OS just does this, right? • Is Python good at this?

7. Nature of the work? CPU Bound -- “calculating digits of

   pi” IO Bound Work -- “take request, query db, send response”

8. What stops us from doing work? • contention -- fighting

   over a resource (like a lock) • blocking -- stopping execution to wait (stops me, lets everyone else go)

9. My OS just does this right? • processes: sandboxed memory

   space • threads: runs within process, shares memory with other threads • These get me where I need to be?

10. Is Python good at this? • It’s half good GIL!!!!

    Who is GIL? Global Interpreter Lock • One thread runs in Python interpreter at once • Threads tend to keep GIL until done or IO But… • This slows us down. Contention! • To teh codez!

11. Python CPU Bound Threads from queue import Queue from threading

    import Thread inQ = Queue() outQ = Queue() def worker(): while True: l = inQ.get() sumL = sum(l) outQ.put( sumL ) numWorkers=10 ts = [Thread(target=worker) for i in xrange(numWorkers)] for t in ts: t.start() Get work to do CPU Bound Work Work output Main thread carries on Create Threads

12. Python IO Worker Threads ... inQ = Queue() outQ =

    Queue() def worker(): while True: url = inQ.get() resp = requests.get(url) outQ.put( (url, resp.status_code, resp.text) ) numWorkers=10 ts = [Thread(target=worker) for i in xrange(numWorkers)] ... Get work to do Blocking IO Work output

13. CPU Bound Threads… we like? from queue import Queue from

    threading import Thread inQ = Queue() outQ = Queue() def worker(): while True: l = inQ.get() sumL = sum(l) outQ.put( sumL ) numWorkers=10 ts = [Thread(target=worker) for i in xrange(numWorkers)] ... ☹ CPU Bound Work doesn’t release GIL so... ☹ … no gain from more threads/cores ☹ … only more contention ☺ Code straight-forward ☹ Contention Points?

14. IO Worker Threads.. we like? … inQ = Queue() outQ

    = Queue() def worker(): while True: url = inQ.get() resp = requests.get(url) outQ.put( (url, resp.status_code, resp.text) ) numWorkers=10 ts = [Thread(target=worker) for i in xrange(numWorkers)] ... ☹ … how many to start? pool? ☹ Contention Points? ☺ Gives up GIL ☹ One blocking IO operation per thread so... ☺Code straight-forward

15. Improve upon CPU bound? from multiprocessing import Process, Queue def

    worker(inQ, outQ): while True: l = inQ.get() sumL = sum(l) outQ.put( sumL ) inQ = Queue() outQ = Queue() p = Process(target=worker, args=(inQ, outQ)) p.start() Using Processes (and an interprocess Queue) Same code

16. CPU Bound Processes… we like? def worker(inQ, outQ): while True:

    l = inQ.get() sumL = sum(l) outQ.put( sumL ) numWorkers=10 ts = [Process(target=worker, args=(inQ, outQ)) for i in xrange(numWorkers)] ... ☺Not sharing GIL ☺ No GIL: More Processes means concurrent work ☺ Max out all your cores! ☺ Code straight-forward ☹ Contention? ☺(less scary -- process abstractions) We LIKE!

17. Processes Rule, Threads Drool Threads Processes Light? Yes ☺ Almost

    As Light ☺ Danger? High -- mutable, shared state. deadlocks. ☹ Lower, stricter communication ☺ Communication Primitives? Mutexes/locks, atomic CPU instructions, thread-safe data structures ☺ OS abstractions, pipes, sockets, shared memory, etc ☺ If they crash... whole program crashes ☹ only process crashes ☺ Control? Extremely High ☺ Moderate, through abstractions GIL? Yes ☹ No ☺

18. Processes Rule, Threads Drool • Processes: safer choice, sandboxed, coarse-grain

    control • Threads: dangerous choice, very fine-grain control/interaction

19. Improve on IO bound work? def worker(): while True: handle_ui_input()

    handle_io() def worker1(): while True: resp = handle_io1() update_shared_state(resp) def worker2(): while True: resp = handle_io2() update_shared_state(resp) Blocking Contention -- need to lock and update IO Bound work often looks like:

20. Other OS primitives help? def event_loop(): while True: whichIsReady =

    select(ui, io1, io2) if whichIsReady == io1: resp = handle_io1(req) if whichIsReady == ui ... Simultaneously block on multiple IO operations No longer need to lock shared state (in one thread)

21. Non-blocking IO… we like? def event_loop(): schedIo = [...] while

    True: whichIsReady = select(schedIo) whichIsReady.callback() httpReq = HttpReq(‘http://odu.edu’) def callWhenDone(): print httpReq.data httpReq.fetch(callback=callWhenDone) ... ☹ harder to read (Javascript anyone?) ☹ extra “IO” scheduler/event loop ☺concurrent IO not bound to number of threads

22. Non-blocking IO improvements httpReq = HttpReq(‘http://odu.edu’) def callWhenDone(): print “Fetched

    and stored imgs!” def storeInDb(httpPromise) dbPromise = db.store(httpPromise) return dbPromise promise = httpReq.fetch() imgPromise = parseImgTags(promise) dbPromise = storedInDb(imgPromise) dbPromise.onDone(callback=callWhenDone) Promise/Deferred/Futures Handle to future work Handle to pending IO Chainable with other operations Can still use callbacks

23. Non-blocking IO improvements httpReq = HttpReq(‘http://odu.edu’) def storeInDb(httpPromise) dbPromise =

    storeInDb(httpPromise) return dbPromise promise = httpReq.fetch() promise.whenComplete(callWhenDone) imgPromise = parseImgTags(promise) dbPromise = storedInDb(imgPromise) myCoroutine.yieldUntil(dbPromise) print “Fetched and stored IMG tags!” Coroutines/cooperative multitasking. I own this thread until I say I’ m done Looks like a blocking call, but in reality yields back to event loop ☺ Readability of blocking IO ☺ Performance of non-blocking async IO

24. Example: Greenlet/Gevent Greenlet: • coroutine library, greenlet decides when to

    give up control Gevent: • monkey-patch a big event loop into Python, replacing core blocking IO bits

25. Gevent is magic class Fetcher(object): def __init__(self, fetchUrl): self.fetchUrl =

    fetchUrl self.resp = None def fetch(self): self.resp = requests.get(self.fetchUrl) def fetchMultiple(urls): fetchers = [Fetcher(url) for url in urls] handles = [] for fetcher in fetchers: handles.append(gevent.spawn(fetcher.fetch)) gevent.joinall(handles) Blocking? Depends on your definition of “blocking” Spawn a Gevent worker that calls “fetch” Wait till all done

26. Other Solutions (aka future lightning talk fodder) Twisted CPython C

    Modules (scipy, your module) Cython (compile to Python -> C) Jython/IronPython (JIT to JVM or .NET CLI) GPUs (CUDA, etc) cluster frameworks (discussed later)

27. Andrew’s part

28. Python GIL Why it’s there What it messes up, concurrency-wise

    Failed efforts to remove GIL pypy and pypy-stm

29. Python concurrency menu CPU-bound work IO-bound work single-node multiprocessing ProcessPoolExecutor

    Twisted (Network) Tornado (HTTP) ThreadPoolExecutor (Generic) gevent (Sockets) asyncore (Sockets) asyncio (Generic) multi-node rq celery IPython.parallel scrapyd? (HTTP)

30. I love processes. (you should, too)

31. Choose your own message broker Pure Python tasks Pure Python

    worker infrastructure Advanced message patterns Choose your own message broker Mix Python + Java or other languages Java worker infrastructure Advanced message patterns More complex operationally High availability & linear scalability “Lambda Architecture” Options for multi-node concurrency start here upgrade here Redis as message broker Pure Python tasks Pure Python worker infrastructure Simple message patterns

32. Python concurrency in clusters • mrjob: Hadoop Streaming (batch) •

    streamparse: Apache Storm (real-time) • parallelize through Python process model • mixed workloads ◦ CPU- and IO-bound • mixed concurrency models are possible ◦ threads within Storm Bolts ◦ process pools within Hadoop Tasks

33. My instinct: threads = bugs

34. None

35. But, sometimes necessary - BatchingBolt - IO-bound Drivers

36. What is asyncore? • stdlib-included async sockets (like libev) •

    in stdlib since 2000! Comment from the source code in 2000: There are only two ways to have a program on a single processor do "more than one thing at a time". Multi-threaded programming is the simplest and most popular way to do it, but there is another very different technique, that lets you have nearly all the advantages of multi-threading, without actually using multiple threads. it's really only practical if your program is largely I/O bound. If your program is CPU bound, then pre- emptive scheduled threads are probably what you really need. Network servers are rarely CPU-bound, however.

37. Python async networking comparison to nginx / Node.JS Twisted Tornado

    gevent / gthreads all using their own “reactor” / “event loop”

38. What is concurrent.futures? • PEP-3148 • new unified API for

    concurrency in Python • in stdlib in Python 3.2+ • backport in 2.7 (pip install futures) • API design like Java Executor Framework • a Future abstraction as a return value • an Executor abstraction for running things

39. asyncio history • PEP-3153: Async IO Support • PEP-3156: Async

    IO Support “Rebooted” • GvR’s pet project from 2012-2014 • Original implementation called tulip • Released in Python 3.4 as asyncio • PyCon 2013 keynote by GvR focused on it • PEP-380 (yield from) utilized by it

40. asyncio primitives • a loop that starts and stops •

    callback scheduling ◦ now ◦ time in in the future ◦ repeated / periodic • associate callbacks with file I/O states • offer pluggable I/O multiplexing mechanism ◦ select() ◦ poll(), epoll(), others

41. asyncio “ooooh ahhhh” moments • introduces @coroutine decorator • uses

    yield from to simplify callback hell • one event loop to rule them all ◦ Twisted and Tornado and gevent in same app! • offers an asyncio.Future ◦ asyncio.Future quacks-like futures.Future ◦ asyncio.wrap_future is an adapter ◦ asyncio.Task is subclass of Future

42. @dabeaz on generators

43. asyncio coroutines code explain result = yield from future suspend

    until future done, then return result result = yield from coroutine suspend until coroutine returns a result return expression return a result to another coroutine raise exception raise an exception to another coroutine

44. @dabeaz on Future and Task unity asyncio and an event

    loop with scheduler threading with a thread pool and executor

45. What is the gain of “yield from”? (1) So, why

    don’t I like green threads? In a simple program using stackless or gevent, it’s easy enough to say, ‘This is a call that goes to the scheduler -- it uses read() or send() or something. I know that’s a blocking call, I’ll be careful…. I don’t need explicit locking because between points A or B, I just need to make sure I don’t make any other calls to the scheduler.’ However, as code gets longer, it becomes hard to keep track. Sooner or later… - Guido van Rossum

46. What is the gain of “yield from”? (2) Just trust

    me; this problem happened to me at a young and impressionable age, and I’ve never forgotten it. - Guido van Rossum

47. The Future() is now() - Tulip project update (Jan 2014)

    by @guidovanrossum - Unyielding (Feb 2014) by @glyph - Generators, The Final Frontier (Apr 2014) by @dabeaz

48. Concurrency in the real world • Cassandra driver shows different

    models • pip install cassandra-driver • asyncore and libev event loops preferred • twisted and gevent also provided • performance benchmarking is dramatic (ranges from 2k to 18k write ops per sec)

49. yield from CassandraDemo() • spin up a Cassandra node •

    execute / benchmark naive sync writes • switch to batched futures • switch to callback chaining • try different event loops • switch to pypy • discuss how asyncio could clean this up

50. bonus round: asyncio crawler • if there’s time, show GvR’s

    example crawler • asyncio, @coroutine, and yield from in a readable Python 3.4 program • crawls 1,300 xkcd.com pages in 7 seconds