Case Studies of Python in Parallel

Transcript

1. Case Studies in Case Studies in Running Python in Parallel

   Running Python in Parallel Tim McNamara (@timClicks) Tim McNamara (@timClicks) Kiwi Pycon 2012 Kiwi Pycon 2012 Content released under the Creative Commons Content released under the Creative Commons Attribution 3.0 New Zealand licence. Attribution 3.0 New Zealand licence. Original code released under MIT/X11 licence. Original code released under MIT/X11 licence.

2. Objectives Objectives 1) introduce people to distributed 1) introduce people

   to distributed programming programming 2) peek inside Python's standard library 2) peek inside Python's standard library 3) build community around distributed 3) build community around distributed computing in NZ computing in NZ

3. Outline Outline 1) introduce a problem 1) introduce a problem

   2) implement several solutions 2) implement several solutions 3) consider trade offs between different 3) consider trade offs between different

4. Caveat Caveat I will be using the terms parallel, I

   will be using the terms parallel, distributed and concurrent fairly distributed and concurrent fairly interchangeably. This is wrong. interchangeably. This is wrong.

5. Why? Why? A tiny selections reasons... A tiny selections reasons...

6. before before

7. after after

8. before before t t

9. after after t t Key point: less time Key point:

   less time

10. before before error

11. after after Key point: isolation, leading to fault tolerance Key

    point: isolation, leading to fault tolerance error

12. Doing more than one thing Doing more than one thing

    on the same computer on the same computer

13. Where to begin? Where to begin?

14. Let Let’ ’s work through a use case together s

    work through a use case together

15. Facial recognition Facial recognition

16. Problem Statement: Problem Statement:

17. We are worried about We are worried about government survelliance;

    government survelliance; how many faces from the how many faces from the WWW is it possible to WWW is it possible to scan with $100? scan with $100?

18. We are worried about We are worried about government survelliance;

    government survelliance; how many faces from the how many faces from the WWW is it possible to WWW is it possible to scan with $100? scan with $100?

19. Original Version Original Version

20. def people_in_pages(url): page = get(url) for img_url in extract_img_urls(page): img

    = get(url) for face in detect_faces(img): yield face

21. def people_in_pages(url): page = get(url) for img_url in extract_img_urls(page): img

    = get(url) for face in detect_faces(img): yield face Where could we do two things at the same time?

22. def people_in_pages(url): page = get(url) for img_url in extract_img_urls(page): img

    = get(url) for face in detect_faces(img): yield face Which portions of the code are I/O bound?

23. def people_in_pages(url): page = get(url) for img_url in extract_img_urls(page): img

    = get(url) for face in detect_faces(img): yield face Which portions of the code are CPU bound?

24. def people_in_pages(url): page = get(url) for img_url in extract_img_urls(page): img

    = get(url) for face in detect_faces(img): yield face Each loop can be done independently

25. Working Implementation Working Implementation

26. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name)

27. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Download a page

28. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Extract URLs with images in them using a good enough regular expression pattern

29. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Matches h, t, p, s, /, word characters and dots (Redundancy added for readability)

30. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) That finishes in a useful file extension.

31. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) From the page contents.

32. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) While ignoring cases.

33. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) “cascade” is OpenCV terminology.

34. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) You can think of it as a model.

35. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Allocate a memory buffer.

36. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) The results of re.findall look like ('http://img.com/hi.jpg', 'jpg')

37. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Download the image, ignoring any problems

38. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Unless I have caused the problem.

39. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Write image to a named temporary file

40. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) (This approach was easier than deserialising the incoming string, decompressing the data and coverting it to an array)

41. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Set file extension correctly

42. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) This will enable OpenCV to load it properly later.

43. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Prevent automatic file deletion when file is closed.

44. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Load an image using the file's name.

45. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Editor's Note: This is wrong

46. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Do facial recognition

47. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Yield (x, y, height, width)

48. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Delete the file

49. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) Delete the file

50. How does that massive function relate to the much smaller

    one we saw earlier?

51. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name)

52. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) def get(url)

53. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) def extract_img_urls(page)

54. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) def extract_img_urls(page)

55. import re from tempfile import NamedTemporaryFile as NTF import cv,

    requests def people_in_pages(url): page = requests.get(url).content finds = re.findall('([https://\w\.]+(jpg|png))', page, re.IGNORECASE) cascade = cv.Load('/home/tim/Libs/haarcascade_frontalface_alt.xml') store = cv.CreateMemStorage() for url, extn in finds: try: image = requests.get(url).content except KeyboardInterrupt: raise except: # can't handle relative URLs :( continue # kludge to make things easy for OpenCV with NTF(suffix='.'+extn, delete=False) as f: f.write(image) image = cv.LoadImageM(f.name, iscolor=True) faces = cv.HaarDetectObjects(image, cascade, store) for face in faces: coordinates, neighbours = face yield coordinates f.unlink(f.name) def detect_faces(image)

56. Introducing Introducing multiprocessing.Pool multiprocessing.Pool

57. Introducing Introducing multiprocessing.Pool multiprocessing.Pool from from multiprocessing multiprocessing import import

    Pool, cpu_count Pool, cpu_count P = Pool(cpu_count()-1) P = Pool(cpu_count()-1) def people_in_pages(url): page = get(url) img_urls = extract_img_urls(page) imgs = P.map(get, img_urls) faces = P.map(detect_faces, imgs) return faces

58. By the way... By the way... In Python, using threading

    for I/O bound In Python, using threading for I/O bound problems will be just fine. problems will be just fine. That is if you are interfacing from the outside That is if you are interfacing from the outside world, audio, serial, network, disc, et cetera, world, audio, serial, network, disc, et cetera, then threads are easier. then threads are easier. Multiprocessing is a much a better way to Multiprocessing is a much a better way to distribute work across CPUs. distribute work across CPUs.

59. Applying this to webscale Applying this to webscale

60. from bottle import abort, request, route, run @route('/faces') def faces():

    url = request.query['url'] if not url: abort(400) return '%s' % people_in_pages(url) @route('/') def index(): page = '''<html><head></head><body> <form method='GET' action='/faces'> <input name="url" type="text"></input> <button type="submit">Hunt</button> </body></html> ''' return page run('localhost', 8000)

61. from bottle import abort, request, route, run @route('/faces') def faces():

    url = request.query['url'] if not url: abort(400) return '%s' % people_in_pages(url) @route('/') def index(): page = '''<html><head></head><body> <form method='GET' action='/faces'> <input name="url" type="text"></input> <button type="submit">Hunt</button> </body></html> ''' return page run('localhost', 8000) From previous slide

62. from bottle import abort, request, route, run @route('/faces') def faces():

    url = request.query['url'] if not url: abort(400) return '%s' % people_in_pages(url) @route('/') def index(): page = '''<html><head></head><body> <form method='GET' action='/faces'> <input name="url" type="text"></input> <button type="submit">Hunt</button> </body></html> ''' return page run('localhost', 8000)

63. Spreading across machines Spreading across machines

64. Options in Python Options in Python

65. Disco Celery Roll your own iPython

66. Disco Celery *-RPC *MQ Roll your own iPython

67. Disco Celery MPI *-RPC *MQ Roll your own PBS LSF

    WinHPC SGE SSH iPython

68. XML-RPC XML-RPC

69. You may hate the verbosity of XML You may hate

    the verbosity of XML

70. but the spec is XML-RPC solid but the spec is

    XML-RPC solid

71. and an implementation exists and an implementation exists in the

    standard library. in the standard library.

72. XML-RPC XML-RPC try: try: from from SimpleXMLRPCServer SimpleXMLRPCServer import import

    SimpleXMLRPCServer SimpleXMLRPCServer except except ImportError: ImportError: from from xmlrpc.server xmlrpc.server import import SimpleXMLRPCServer SimpleXMLRPCServer server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", , 22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever()

73. XML-RPC XML-RPC server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", ,

    22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever() Less scary without the boilerplate

74. XML-RPC XML-RPC server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", ,

    22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever() For spec compliance

75. XML-RPC XML-RPC server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", ,

    22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever() One thing to note, our function with yield will raise an exception.

76. XML-RPC XML-RPC server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", ,

    22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever() You can leave the registered function unnamed, defaults to fn.__name__

77. XML-RPC XML-RPC try: try: from from SimpleXMLRPCServer SimpleXMLRPCServer import import

    SimpleXMLRPCServer SimpleXMLRPCServer except except ImportError: ImportError: from from xmlrpc.server xmlrpc.server import import SimpleXMLRPCServer SimpleXMLRPCServer server = SimpleXMLRPCServer(( server = SimpleXMLRPCServer(("localhost" "localhost", , 22022 22022)) )) server.register_introspection_functions() server.register_introspection_functions() server.register_function(people_in_pages, server.register_function(people_in_pages, 'people' 'people') ) server.serve_forever() server.serve_forever()

78. How do we connect this to our website? How do

    we connect this to our website?

79. from bottle import abort, request, route, run @route('/faces') def faces():

    url = request.query['url'] if not url: abort(400) return '%s' % people_in_pages(url) @route('/') def index(): page = '''<html><head></head><body> <form method='GET' action='/faces'> <input name="url" type="text"></input> <button type="submit">Hunt</button> </body></html> ''' return page run('localhost', 8000) Original

80. try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import

    ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... New

81. try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import

    ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... Connect to the XML-RPC server.

82. try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import

    ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... Use XML-RPC server.

83. try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import

    ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... Remember, we renamed the function

84. try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import

    ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ...

85. try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import

    ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... Has this helped?

86. try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import

    ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... No

87. try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import

    ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... We're still only using one core

88. try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import

    ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... Performance is now slower...

89. try: import xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import

    ServerProxy from bottle import abort, request, route, run rpc = ServerProxy('http://localhost:22022 22022') @route('/faces') def faces(): url = request.query['url'] if not url: abort(400) return '%s' % rpc.people(url) ... ...everything is serialised as XML.

90. How can we use the standard How can we use

    the standard library to create something more useful? library to create something more useful?

91. The SocketServer/socketserver module The SocketServer/socketserver module provides a ThreadedTCPServer... provides

    a ThreadedTCPServer...

92. … … but that wont help with but that wont

    help with CPU-bound problems. CPU-bound problems.

93. Let's use adapt the Let's use adapt the SimpleXMLRPCServer SimpleXMLRPCServer

    code for our own use. code for our own use.

94. We'll fork an OS process We'll fork an OS process

    for each call. for each call.

95. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCDispatcher): """Simple XML-RPC server. """Simple

    XML-RPC server. Simple XML-RPC server that allows functions and a single instance Simple XML-RPC server that allows functions and a single instance to be installed to handle requests. The default implementation to be installed to handle requests. The default implementation attempts to dispatch XML-RPC calls to the functions or instance attempts to dispatch XML-RPC calls to the functions or instance installed in the server. Override the _dispatch method inhereted installed in the server. Override the _dispatch method inhereted from SimpleXMLRPCDispatcher to change this behavior. from SimpleXMLRPCDispatcher to change this behavior. """ """ allow_reuse_address = allow_reuse_address = True True # Warning: this is for debugging purposes only! Never set this to True in # Warning: this is for debugging purposes only! Never set this to True in # production code, as will be sending out sensitive information (exception # production code, as will be sending out sensitive information (exception # and stack trace details) when exceptions are raised inside # and stack trace details) when exceptions are raised inside # SimpleXMLRPCRequestHandler.do_POST # SimpleXMLRPCRequestHandler.do_POST _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) # [Bug #1222790] If possible, set close-on-exec flag; if a # [Bug #1222790] If possible, set close-on-exec flag; if a # method spawns a subprocess, the subprocess shouldn't have # method spawns a subprocess, the subprocess shouldn't have # the listening socket open. # the listening socket open. if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags)

96. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCDispatcher): """Simple XML-RPC server. """Simple

    XML-RPC server. Simple XML-RPC server that allows functions and a single instance Simple XML-RPC server that allows functions and a single instance to be installed to handle requests. The default implementation to be installed to handle requests. The default implementation attempts to dispatch XML-RPC calls to the functions or instance attempts to dispatch XML-RPC calls to the functions or instance installed in the server. Override the _dispatch method inhereted installed in the server. Override the _dispatch method inhereted from SimpleXMLRPCDispatcher to change this behavior. from SimpleXMLRPCDispatcher to change this behavior. """ """ allow_reuse_address = allow_reuse_address = True True # Warning: this is for debugging purposes only! Never set this to True in # Warning: this is for debugging purposes only! Never set this to True in # production code, as will be sending out sensitive information (exception # production code, as will be sending out sensitive information (exception # and stack trace details) when exceptions are raised inside # and stack trace details) when exceptions are raised inside # SimpleXMLRPCRequestHandler.do_POST # SimpleXMLRPCRequestHandler.do_POST _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) # [Bug #1222790] If possible, set close-on-exec flag; if a # [Bug #1222790] If possible, set close-on-exec flag; if a # method spawns a subprocess, the subprocess shouldn't have # method spawns a subprocess, the subprocess shouldn't have # the listening socket open. # the listening socket open. if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) http://hg.python.org/cpython/file/2.7/Lib/SimpleXMLRPCServer.py#l569

97. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCDispatcher): """Simple XML-RPC server. """Simple

    XML-RPC server. Simple XML-RPC server that allows functions and a single instance Simple XML-RPC server that allows functions and a single instance to be installed to handle requests. The default implementation to be installed to handle requests. The default implementation attempts to dispatch XML-RPC calls to the functions or instance attempts to dispatch XML-RPC calls to the functions or instance installed in the server. Override the _dispatch method inhereted installed in the server. Override the _dispatch method inhereted from SimpleXMLRPCDispatcher to change this behavior. from SimpleXMLRPCDispatcher to change this behavior. """ """ allow_reuse_address = allow_reuse_address = True True # Warning: this is for debugging purposes only! Never set this to True in # Warning: this is for debugging purposes only! Never set this to True in # production code, as will be sending out sensitive information (exception # production code, as will be sending out sensitive information (exception # and stack trace details) when exceptions are raised inside # and stack trace details) when exceptions are raised inside # SimpleXMLRPCRequestHandler.do_POST # SimpleXMLRPCRequestHandler.do_POST _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) # [Bug #1222790] If possible, set close-on-exec flag; if a # [Bug #1222790] If possible, set close-on-exec flag; if a # method spawns a subprocess, the subprocess shouldn't have # method spawns a subprocess, the subprocess shouldn't have # the listening socket open. # the listening socket open. if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Urgh

98. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCDispatcher): """Simple XML-RPC server. """Simple

    XML-RPC server. Simple XML-RPC server that allows functions and a single instance Simple XML-RPC server that allows functions and a single instance to be installed to handle requests. The default implementation to be installed to handle requests. The default implementation attempts to dispatch XML-RPC calls to the functions or instance attempts to dispatch XML-RPC calls to the functions or instance installed in the server. Override the _dispatch method inhereted installed in the server. Override the _dispatch method inhereted from SimpleXMLRPCDispatcher to change this behavior. from SimpleXMLRPCDispatcher to change this behavior. """ """ allow_reuse_address = allow_reuse_address = True True # Warning: this is for debugging purposes only! Never set this to True in # Warning: this is for debugging purposes only! Never set this to True in # production code, as will be sending out sensitive information (exception # production code, as will be sending out sensitive information (exception # and stack trace details) when exceptions are raised inside # and stack trace details) when exceptions are raised inside # SimpleXMLRPCRequestHandler.do_POST # SimpleXMLRPCRequestHandler.do_POST _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) # [Bug #1222790] If possible, set close-on-exec flag; if a # [Bug #1222790] If possible, set close-on-exec flag; if a # method spawns a subprocess, the subprocess shouldn't have # method spawns a subprocess, the subprocess shouldn't have # the listening socket open. # the listening socket open. if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Let's simplify

99. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address =

    True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags)

100. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address =

     True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Important things to note

101. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address =

     True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Subclassing from SocketServer mixin...

102. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address =

     True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) ...and dispatcher.

103. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address =

     True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Perhaps we can just use SocketServer.ForkingTCPServer?

104. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address =

     True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Watch out for hard coded values!

105. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address =

     True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) This thing appears to be a workaround to a bug.

106. class class SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): SimpleXMLRPCServer(SocketServer.TCPServer, SimpleXMLRPCDispatcher): allow_reuse_address = allow_reuse_address =

     True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, (self, addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) SocketServer.TCPServer.__init__(self, addr, SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) requestHandler, bind_and_activate) if if fcntl fcntl is not is not None and hasattr(fcntl, 'FD_CLOEXEC'): None and hasattr(fcntl, 'FD_CLOEXEC'): flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC flags |= fcntl.FD_CLOEXEC fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags) Let's leave it alone.

107. Our implementation: Our implementation:

108. import import fcntl fcntl # needed for workaround, now hidden

     # needed for workaround, now hidden from from SocketServer SocketServer import import ForkingTCPServer as Server ForkingTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ...

109. import import fcntl fcntl # needed for workaround, now hidden

     # needed for workaround, now hidden from from SocketServer SocketServer import import ForkingTCPServer as Server ForkingTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ... Here is the magic.

110. What about gevent? What about gevent?

111. gevent is great gevent is great for problems bound by

     I/O for problems bound by I/O

112. import import gevent gevent gevent.patch_all() gevent.patch_all() import import fcntl fcntl

     # needed for workaround, now hidden # needed for workaround, now hidden from from SocketServer SocketServer import import ThreadedTCPServer as Server ThreadedTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ...

113. import import gevent gevent gevent.patch_all() gevent.patch_all() import import fcntl fcntl

     # needed for workaround, now hidden # needed for workaround, now hidden from from SocketServer SocketServer import import ThreadedTCPServer as Server ThreadedTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ... Can you spot the differences?

114. import import gevent gevent gevent.patch_all() gevent.patch_all() import import fcntl fcntl

     # needed for workaround, now hidden # needed for workaround, now hidden from from SocketServer SocketServer import import ThreadedTCPServer as Server ThreadedTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ... gevent monkeypatches socket...

115. import import gevent gevent gevent.patch_all() gevent.patch_all() import import fcntl fcntl

     # needed for workaround, now hidden # needed for workaround, now hidden from from SocketServer SocketServer import import ThreadedTCPServer as Server ThreadedTCPServer as Server from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCDispatcher as Dispatcher SimpleXMLRPCDispatcher as Dispatcher from from SimpleXMLRPCServer SimpleXMLRPCServer import import SimpleXMLRPCRequestHandler as Handler SimpleXMLRPCRequestHandler as Handler class class ForkingXMLRPCServer(Server, Dispatcher): ForkingXMLRPCServer(Server, Dispatcher): allow_reuse_address = allow_reuse_address = True True _send_traceback_header = _send_traceback_header = False False def def __init__ __init__(self, addr, request_handler=Handler, (self, addr, request_handler=Handler, logRequests=True, allow_none=False, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): encoding=None, bind_and_activate=True): self.logRequests = logRequests self.logRequests = logRequests Dispatcher.__init__(self, allow_none, encoding) Dispatcher.__init__(self, allow_none, encoding) Server.__init__(self, addr, request_handler, bind_and_activate) Server.__init__(self, addr, request_handler, bind_and_activate) if if fcntl fcntl is not is not None None and and hasattr(fcntl, 'FD_CLOEXEC'): hasattr(fcntl, 'FD_CLOEXEC'): ... ... ...meaning you can use “threads”.

116. So, what's Celery? So, what's Celery?

117. A task queue that is even easier to use. A

     task queue that is even easier to use.

118. As a bonus, it can support both As a bonus,

     it can support both gevent for I/O intensive tasks & gevent for I/O intensive tasks & multiprocessing for CPU intensive tasks multiprocessing for CPU intensive tasks (at the same time). (at the same time).

119. As a bonus, it can support both As a bonus,

     it can support both gevent for I/O intensive tasks & gevent for I/O intensive tasks & multiprocessing for CPU intensive tasks multiprocessing for CPU intensive tasks (at the same time). (at the same time). You can use OS threads for I/O tasks, but you will get a soft max of 200 instead of over 1000.

120. Celery Celery import import celery celery @celery.task @celery.task def people_in_pages(url):

     ...

121. iPython iPython

122. iPython looks just like a shell, iPython looks just like

     a shell, but it also has very stable support for but it also has very stable support for distributed programming. distributed programming.

123. Controller ... Engine 0 ... An IPython Cluster An IPython

     Cluster

124. Controller ... Engine 0 ... An IPython Cluster An IPython

     Cluster Controllers are Controllers are your interface to your interface to one or more one or more engines engines

125. Controller ... Engine 0 ... An IPython Cluster An IPython

     Cluster Engines could be Engines could be local processes, local processes, remote remote processes via processes via SSH or batch SSH or batch jobs. jobs.

126. Controller ... Engine 0 ... An IPython Cluster An IPython

     Cluster Schedulers live Schedulers live here, but are here, but are hidden from you. hidden from you.

127. $ ipython profile create --parallel profile_name $ ipython profile create

     --parallel profile_name $ ipcluster start –n=4 --profile=profile_name $ ipcluster start –n=4 --profile=profile_name $ ipython notebook --profile=profile_name $ ipython notebook --profile=profile_name

128. In [1]: In [1]: from from IPython.parallel IPython.parallel import import

     Client Client In [2]: In [2]: rc = Client(profile= rc = Client(profile='profile_name' 'profile_name') ) In [3]: In [3]: rc.ids rc.ids Out[3]: Out[3]: [0, 1, 2, 3] [0, 1, 2, 3] In [4]: In [4]: view = rc[:] view = rc[:] In [n]: In [n]: view.apply_async(people_in_page, url) view.apply_async(people_in_page, url)

129. Some homework Some homework • To do effective distributed programming,

     you need To do effective distributed programming, you need to think about (at least): to think about (at least): • data locality data locality • ease of configuration / maintainence ease of configuration / maintainence • decomposition decomposition • load balancing load balancing • message size message size • efficient serialisation efficient serialisation • send references, rather than data through message queues send references, rather than data through message queues • ease of debugging ease of debugging

130. Thanks! Thanks!