Case Studies in Case Studies in Running Python in Parallel Running Python in Parallel Tim McNamara (@timClicks) Tim McNamara (@timClicks) Kiwi Pycon 2012 Kiwi Pycon 2012 Content released under the Creative Commons Content released under the Creative Commons Attribution 3.0 New Zealand licence. Attribution 3.0 New Zealand licence. Original code released under MIT/X11 licence. Original code released under MIT/X11 licence. 

Objectives Objectives 1) introduce people to distributed 1) introduce people to distributed programming programming 2) peek inside Python's standard library 2) peek inside Python's standard library 3) build community around distributed 3) build community around distributed computing in NZ computing in NZ 

Outline Outline 1) introduce a problem 1) introduce a problem 2) implement several solutions 2) implement several solutions 3) consider trade offs between different 3) consider trade offs between different 

Caveat Caveat I will be using the terms parallel, I will be using the terms parallel, distributed and concurrent fairly distributed and concurrent fairly interchangeably. This is wrong. interchangeably. This is wrong. 

Why? Why? A tiny selections reasons... A tiny selections reasons... 

before before 

after after 

before before t t 

after after t t Key point: less time Key point: less time 

before before error 

after after Key point: isolation, leading to fault tolerance Key point: isolation, leading to fault tolerance error 

Doing more than one thing Doing more than one thing on the same computer on the same computer 

Where to begin? Where to begin? 

Let Let’ ’s work through a use case together s work through a use case together 

Facial recognition Facial recognition 

Problem Statement: Problem Statement: 

We are worried about We are worried about government survelliance; government survelliance; how many faces from the how many faces from the WWW is it possible to WWW is it possible to scan with $100? scan with $100? 

We are worried about We are worried about government survelliance; government survelliance; how many faces from the how many faces from the WWW is it possible to WWW is it possible to scan with $100? scan with $100? 

Original Version Original Version 

def people_in_pages(url): page = get(url) for img_url in extract_img_urls(page): img = get(url) for face in detect_faces(img): yield face 

def people_in_pages(url): page = get(url) for img_url in extract_img_urls(page): img = get(url) for face in detect_faces(img): yield face Where could we do two things at the same time? 

def people_in_pages(url): page = get(url) for img_url in extract_img_urls(page): img = get(url) for face in detect_faces(img): yield face Which portions of the code are I/O bound? 

def people_in_pages(url): page = get(url) for img_url in extract_img_urls(page): img = get(url) for face in detect_faces(img): yield face Which portions of the code are CPU bound? 

def people_in_pages(url): page = get(url) for img_url in extract_img_urls(page): img = get(url) for face in detect_faces(img): yield face Each loop can be done independently 

Working Implementation Working Implementation 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Download a page 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Extract URLs with images in them using a good enough regular expression pattern 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Matches h, t, p, s, /, word characters and dots (Redundancy added for readability) 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) That finishes in a useful file extension. 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) From the page contents. 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) While ignoring cases. 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) “cascade” is OpenCV terminology. 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) You can think of it as a model. 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Allocate a memory buffer. 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) The results of re.findall look like ('http://img.com/hi.jpg', 'jpg') 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Download the image, ignoring any problems 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Unless I have caused the problem. 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Write image to a named temporary file 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) (This approach was easier than deserialising the incoming string, decompressing the data and coverting it to an array) 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Set file extension correctly 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) This will enable OpenCV to load it properly later. 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Prevent automatic file deletion when file is closed. 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Load an image using the file's name. 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Editor's Note: This is wrong 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Do facial recognition 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Yield (x, y, height, width) 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Delete the file 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Delete the file 

How does that massive function relate to the much smaller one we saw earlier? 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) def get(url) 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) def extract_img_urls(page) 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) def extract_img_urls(page) 

import re from tempfile import NamedTemporaryFile as NTF import cv, requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) def detect_faces(image) 

Introducing Introducing multiprocessing.Pool multiprocessing.Pool 

Introducing Introducing multiprocessing.Pool multiprocessing.Pool from from multiprocessing multiprocessing import import Pool, cpu_count Pool, cpu_count P = Pool(cpu_count()-1) P = Pool(cpu_count()-1) def people_in_pages(url): page = get(url) img_urls = extract_img_urls(page) imgs = P.map(get, img_urls) faces = P.map(detect_faces, imgs) return faces 

By the way... By the way... In Python, using threading for I/O bound In Python, using threading for I/O bound problems will be just fine. problems will be just fine. That is if you are interfacing from the outside That is if you are interfacing from the outside world, audio, serial, network, disc, et cetera, world, audio, serial, network, disc, et cetera, then threads are easier. then threads are easier. Multiprocessing is a much a better way to Multiprocessing is a much a better way to distribute work across CPUs. distribute work across CPUs. 

Applying this to webscale Applying this to webscale 

from bottle import abort, request, route, run @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % people_in_pages(url) @route('/') def index(): page = ''' Hunt ''' return page run('localhost', 8000) 

from bottle import abort, request, route, run @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % people_in_pages(url) @route('/') def index(): page = ''' Hunt ''' return page run('localhost', 8000) From previous slide 

from bottle import abort, request, route, run @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % people_in_pages(url) @route('/') def index(): page = ''' Hunt ''' return page run('localhost', 8000) 

Spreading across machines Spreading across machines 

Options in Python Options in Python 

Disco Celery Roll your own iPython 

Disco Celery *-RPC *MQ Roll your own iPython 

Disco Celery MPI *-RPC *MQ Roll your own PBS LSF WinHPC SGE SSH iPython 

XML-RPC XML-RPC 

You may hate the verbosity of XML You may hate the verbosity of XML 

but the spec is XML-RPC solid but the spec is XML-RPC solid 

and an implementation exists and an implementation exists in the standard library. in the standard library. 

XML-RPC XML-RPC try: try: from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCServer SimpleXMLRPCServer except except ImportError: ImportError: from from xmlrpc.server xmlrpc.server import import SimpleXMLRPCServer SimpleXMLRPCServer server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", , 22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever() 

XML-RPC XML-RPC server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", , 22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever() Less scary without the boilerplate 

XML-RPC XML-RPC server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", , 22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever() For spec compliance 

XML-RPC XML-RPC server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", , 22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever() One thing to note, our function with yield will raise an exception. 

XML-RPC XML-RPC server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", , 22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever() You can leave the registered function unnamed, defaults to fn.__name__ 

XML-RPC XML-RPC try: try: from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCServer SimpleXMLRPCServer except except ImportError: ImportError: from from xmlrpc.server xmlrpc.server import import SimpleXMLRPCServer SimpleXMLRPCServer server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", , 22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever() 

How do we connect this to our website? How do we connect this to our website? 

from bottle import abort, request, route, run @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % people_in_pages(url) @route('/') def index(): page = ''' Hunt ''' return page run('localhost', 8000) Original 

try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... New 

try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... Connect to the XML-RPC server. 

try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... Use XML-RPC server. 

try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... Remember, we renamed the function 

try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... 

try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... Has this helped? 

try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... No 

try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... We're still only using one core 

try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... Performance is now slower... 

try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... ...everything is serialised as XML. 

How can we use the standard How can we use the standard library to create something more useful? library to create something more useful? 

The SocketServer/socketserver module The SocketServer/socketserver module provides a ThreadedTCPServer... provides a ThreadedTCPServer... 

… … but that wont help with but that wont help with CPU-bound problems. CPU-bound problems. 

Let's use adapt the Let's use adapt the SimpleXMLRPCServer SimpleXMLRPCServer code for our own use. code for our own use. 

We'll fork an OS process We'll fork an OS process for each call. for each call. 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCDispatcher): """Simple XML-RPC server. """Simple XML-RPC server. Simple XML-RPC server that allows functions and a single instance Simple XML-RPC server that allows functions and a single instance to be installed to handle requests. The default implementation to be installed to handle requests. The default implementation attempts to dispatch XML-RPC calls to the functions or instance attempts to dispatch XML-RPC calls to the functions or instance installed in the server. Override the _dispatch method inhereted installed in the server. Override the _dispatch method inhereted from SimpleXMLRPCDispatcher to change this behavior. from SimpleXMLRPCDispatcher to change this behavior. """ """ allow_reuse_address = allow_reuse_address = True True # Warning: this is for debugging purposes only! Never set this to True in # Warning: this is for debugging purposes only! Never set this to True in # production code, as will be sending out sensitive information (exception # production code, as will be sending out sensitive information (exception # and stack trace details) when exceptions are raised inside # and stack trace details) when exceptions are raised inside # SimpleXMLRPCRequestHandler.do_POST # SimpleXMLRPCRequestHandler.do_POST _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) # [Bug #1222790] If possible, set close-on-exec flag; if a # [Bug #1222790] If possible, set close-on-exec flag; if a # method spawns a subprocess, the subprocess shouldn't have # method spawns a subprocess, the subprocess shouldn't have # the listening socket open. # the listening socket open. if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCDispatcher): """Simple XML-RPC server. """Simple XML-RPC server. Simple XML-RPC server that allows functions and a single instance Simple XML-RPC server that allows functions and a single instance to be installed to handle requests. The default implementation to be installed to handle requests. The default implementation attempts to dispatch XML-RPC calls to the functions or instance attempts to dispatch XML-RPC calls to the functions or instance installed in the server. Override the _dispatch method inhereted installed in the server. Override the _dispatch method inhereted from SimpleXMLRPCDispatcher to change this behavior. from SimpleXMLRPCDispatcher to change this behavior. """ """ allow_reuse_address = allow_reuse_address = True True # Warning: this is for debugging purposes only! Never set this to True in # Warning: this is for debugging purposes only! Never set this to True in # production code, as will be sending out sensitive information (exception # production code, as will be sending out sensitive information (exception # and stack trace details) when exceptions are raised inside # and stack trace details) when exceptions are raised inside # SimpleXMLRPCRequestHandler.do_POST # SimpleXMLRPCRequestHandler.do_POST _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) # [Bug #1222790] If possible, set close-on-exec flag; if a # [Bug #1222790] If possible, set close-on-exec flag; if a # method spawns a subprocess, the subprocess shouldn't have # method spawns a subprocess, the subprocess shouldn't have # the listening socket open. # the listening socket open. if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) http://hg.python.org/cpython/file/2.7/Lib/SimpleXMLRPCServer.py#l569 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCDispatcher): """Simple XML-RPC server. """Simple XML-RPC server. Simple XML-RPC server that allows functions and a single instance Simple XML-RPC server that allows functions and a single instance to be installed to handle requests. The default implementation to be installed to handle requests. The default implementation attempts to dispatch XML-RPC calls to the functions or instance attempts to dispatch XML-RPC calls to the functions or instance installed in the server. Override the _dispatch method inhereted installed in the server. Override the _dispatch method inhereted from SimpleXMLRPCDispatcher to change this behavior. from SimpleXMLRPCDispatcher to change this behavior. """ """ allow_reuse_address = allow_reuse_address = True True # Warning: this is for debugging purposes only! Never set this to True in # Warning: this is for debugging purposes only! Never set this to True in # production code, as will be sending out sensitive information (exception # production code, as will be sending out sensitive information (exception # and stack trace details) when exceptions are raised inside # and stack trace details) when exceptions are raised inside # SimpleXMLRPCRequestHandler.do_POST # SimpleXMLRPCRequestHandler.do_POST _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) # [Bug #1222790] If possible, set close-on-exec flag; if a # [Bug #1222790] If possible, set close-on-exec flag; if a # method spawns a subprocess, the subprocess shouldn't have # method spawns a subprocess, the subprocess shouldn't have # the listening socket open. # the listening socket open. if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Urgh 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCDispatcher): """Simple XML-RPC server. """Simple XML-RPC server. Simple XML-RPC server that allows functions and a single instance Simple XML-RPC server that allows functions and a single instance to be installed to handle requests. The default implementation to be installed to handle requests. The default implementation attempts to dispatch XML-RPC calls to the functions or instance attempts to dispatch XML-RPC calls to the functions or instance installed in the server. Override the _dispatch method inhereted installed in the server. Override the _dispatch method inhereted from SimpleXMLRPCDispatcher to change this behavior. from SimpleXMLRPCDispatcher to change this behavior. """ """ allow_reuse_address = allow_reuse_address = True True # Warning: this is for debugging purposes only! Never set this to True in # Warning: this is for debugging purposes only! Never set this to True in # production code, as will be sending out sensitive information (exception # production code, as will be sending out sensitive information (exception # and stack trace details) when exceptions are raised inside # and stack trace details) when exceptions are raised inside # SimpleXMLRPCRequestHandler.do_POST # SimpleXMLRPCRequestHandler.do_POST _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) # [Bug #1222790] If possible, set close-on-exec flag; if a # [Bug #1222790] If possible, set close-on-exec flag; if a # method spawns a subprocess, the subprocess shouldn't have # method spawns a subprocess, the subprocess shouldn't have # the listening socket open. # the listening socket open. if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Let's simplify 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Important things to note 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Subclassing from SocketServer mixin... 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) ...and dispatcher. 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Perhaps we can just use SocketServer.ForkingTCPServer? 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Watch out for hard coded values! 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) This thing appears to be a workaround to a bug. 

class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Let's leave it alone. 

Our implementation: Our implementation: 

import import fcntl fcntl # needed for workaround, now hidden # needed for workaround, now hidden from from SocketServer SocketServer import import ForkingTCPServer as Server ForkingTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ... 

import import fcntl fcntl # needed for workaround, now hidden # needed for workaround, now hidden from from SocketServer SocketServer import import ForkingTCPServer as Server ForkingTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ... Here is the magic. 

What about gevent? What about gevent? 

gevent is great gevent is great for problems bound by I/O for problems bound by I/O 

import import gevent gevent gevent.patch_all() gevent.patch_all() import import fcntl fcntl # needed for workaround, now hidden # needed for workaround, now hidden from from SocketServer SocketServer import import ThreadedTCPServer as Server ThreadedTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ... 

import import gevent gevent gevent.patch_all() gevent.patch_all() import import fcntl fcntl # needed for workaround, now hidden # needed for workaround, now hidden from from SocketServer SocketServer import import ThreadedTCPServer as Server ThreadedTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ... Can you spot the differences? 

import import gevent gevent gevent.patch_all() gevent.patch_all() import import fcntl fcntl # needed for workaround, now hidden # needed for workaround, now hidden from from SocketServer SocketServer import import ThreadedTCPServer as Server ThreadedTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ... gevent monkeypatches socket... 

import import gevent gevent gevent.patch_all() gevent.patch_all() import import fcntl fcntl # needed for workaround, now hidden # needed for workaround, now hidden from from SocketServer SocketServer import import ThreadedTCPServer as Server ThreadedTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ... ...meaning you can use “threads”. 

So, what's Celery? So, what's Celery? 

A task queue that is even easier to use. A task queue that is even easier to use. 

As a bonus, it can support both As a bonus, it can support both gevent for I/O intensive tasks & gevent for I/O intensive tasks & multiprocessing for CPU intensive tasks multiprocessing for CPU intensive tasks (at the same time). (at the same time). 

As a bonus, it can support both As a bonus, it can support both gevent for I/O intensive tasks & gevent for I/O intensive tasks & multiprocessing for CPU intensive tasks multiprocessing for CPU intensive tasks (at the same time). (at the same time). You can use OS threads for I/O tasks, but you will get a soft max of 200 instead of over 1000. 

Celery Celery import import celery celery @celery.task @celery.task def people_in_pages(url): ... 

iPython iPython 

iPython looks just like a shell, iPython looks just like a shell, but it also has very stable support for but it also has very stable support for distributed programming. distributed programming. 

Controller ... Engine 0 ... An IPython Cluster An IPython Cluster 

Controller ... Engine 0 ... An IPython Cluster An IPython Cluster Controllers are Controllers are your interface to your interface to one or more one or more engines engines 

Controller ... Engine 0 ... An IPython Cluster An IPython Cluster Engines could be Engines could be local processes, local processes, remote remote processes via processes via SSH or batch SSH or batch jobs. jobs. 

Controller ... Engine 0 ... An IPython Cluster An IPython Cluster Schedulers live Schedulers live here, but are here, but are hidden from you. hidden from you. 

$ ipython profile create --parallel profile_name $ ipython profile create --parallel profile_name $ ipcluster start –n=4 --profile=profile_name $ ipcluster start –n=4 --profile=profile_name $ ipython notebook --profile=profile_name $ ipython notebook --profile=profile_name 

In [1]: In [1]: from from IPython.parallel IPython.parallel import import Client Client In [2]: In [2]: rc = Client(profile= rc = Client(profile='profile_name' 'profile_name') ) In [3]: In [3]: rc.ids rc.ids Out[3]: Out[3]: [0, 1, 2, 3] [0, 1, 2, 3] In [4]: In [4]: view = rc[:] view = rc[:] In [n]: In [n]: view.apply_async(people_in_page, url) view.apply_async(people_in_page, url) 

Some homework Some homework ● To do effective distributed programming, you need To do effective distributed programming, you need to think about (at least): to think about (at least): ● data locality data locality ● ease of configuration / maintainence ease of configuration / maintainence ● decomposition decomposition ● load balancing load balancing ● message size message size ● efficient serialisation efficient serialisation ● send references, rather than data through message queues send references, rather than data through message queues ● ease of debugging ease of debugging 

Thanks! Thanks!