Concurrency with Multiprocessing in Python

Slides from my lightning talk on concurrency in Python using the Multiprocessing module.

Transcript

1. Concurrency with Multiprocessing in Python PhillyPug & Philly.rb RedSnake Meeting February 8th, 2011 Gavin M. Roy myYearbook.com
2. green threads in threading
3. from threading import Thread import time class MyThread(Thread): def run(self): print "%s running" % self.name time.sleep(5) print "%s done" % self.name for x in xrange(0,10): thread = MyThread() thread.start() thread.join(0)
4. gmr-0x04:pika gmr$ python threads.py MyThread-1 running MyThread-2 running MyThread-3 running MyThread-4 running MyThread-5 running MyThread-6 running MyThread-7 running MyThread-8 running MyThread-9 running MyThread-10 running MyThread-1 done MyThread-2 done MyThread-3 done MyThread-4 done MyThread-5 done MyThread-6 done MyThread-7 done MyThread-8 done MyThread-9 done MyThread-10 done
5. threading • Locks • Reentrant Locks • Conditions • Semaphores • Events • Timers Reentrant knows who own the locks and the recursion level Additional classes like Queue.Queue
7. This GIL From David Beeazley’s GIL Visualization http://www.dabeaz.com/GIL/gilvis/fourthread.html
8. enter multiprocessing
9. multiprocessing module • All the things threading has • Exchanging Objects • Queues and Pipes • Shared State • Pipes and Queues • Pools • Connections • Managers • SyncManager • Logging TCP Server coordinating shared objects Process Sync
10. from multiprocessing import Process import time class MyThread(Process): def run(self): print "%s running" % self.name time.sleep(5) print "%s done" % self.name for x in xrange(0,10): thread = MyThread() thread.start() thread.join(0)
11. gmr-0x04:pika gmr$ python processes.py Thread-1 running Thread-2 running Thread-3 running Thread-4 running Thread-5 running Thread-6 running Thread-7 running Thread-8 running Thread-9 running Thread-10 running Thread-1 done Thread-2 done Thread-3 done Thread-4 done Thread-5 done Thread-6 done Thread-7 done Thread-8 done Thread-9 done Thread-10 done
12. Photo By Susan NYC: http://www.flickr.com/photos/en321/33868864/ multiprocessing.reduction
13. # Process #1 from multiprocessing.reduction import reduce_handle import socket # Create a socket sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) # Do stuff here, client or server wise # Create the pickled socket handle handle = reduce_handle(sock.fileno) # Process #2 from multiprocessing.reduction import rebuild_handle import socket # In other process fd = rebuild_handle(handle) sock = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM) # Now I can read and write from the socket too