Parallelism in Python - Kiwi PyCon X

Python has a terrible reputation when it comes to its parallel processing capabilities.

Ignoring the standard arguments about its threads and the GIL (which are mostly valid), the real problem with parallelism in Python isn't a technical one, but a pedagogical one. The common tutorials surrounding Threading and Multiprocessing in Python, while generally excellent, are pretty “heavy.” They start in the intense stuff and stop before they get to the really good, day-to-day useful parts.

The talk will highlight the impact of Global Interpreter Lock on multithreaded programs and include code snippets to compare single and multithreaded programs using a real-life example.

Then, we move on to the concept of multiprocessing and the address its benefits over multithreading and mention some examples for it.

The talk will end with an application of multiprocessing in the area of Computer Vision and how it speeds up the completion of a task by 82%.

Moreover, this talk aims to inspire both novices and experts to look more into this subject and create something that could possibly be of benefit to the entire Python community around the world.

Transcript

1. Parallelism in Python Rounak Vyas
2. About Me • Final year CS student • Using python for the last 3 years. • Student Researcher at Next Tech Lab
3. • Python has enjoyed a decade of usage in industry and academia. • Popular abstractions to scientific computing, AI/ML, etc. • Yet, has a bad rep for its parallel processing capabilities. Overview
4. Is multi-threading a scam in Python?
5. How the interpreter works Python uses reference counting for memory management. The reference count variable needs protection from race conditions. Source: Real Python
6. This count variable can be kept safe by adding locks to all data structures. But, adding a lock to each object means multiple locks resulting in deadlocks/dec in performance. How the interpreter works
7. Global Interpreter Lock (GIL) • A mutex (or a lock). • Allows only one thread to hold the control of the Python interpreter.
8. Impact of GIL on multi-threaded programs Single thread ~ 3.60 secs Source: Real Python
9. Multiple threads ~ 3.66 secs (Overhead) Impact of GIL on multi-threaded programs Source: Real Python
10. When is GIL not a problem? I/O Bound Tasks: Everything that blocks the current thread while not consuming much CPU.
11. I/O Bound Tasks: Source: David Beazely slides
12. When it is a problem? CPU Bound Tasks: Tasks that mostly consume CPU time, like heavy computations or moving lots of data around in-memory (sorting, shuffling)
13. Basically,
14. So, is it possible to inject parallelism in CPU bound tasks?
15. Each Python process gets its own Python interpreter and memory space so the GIL won’t be a problem. Introducing: Multi-Processing
16. Introducing: Multi-Processing
17. Thumbnailing thousands of images. A common CPU bound task for someone working on vision, image processing, etc. Real Life Example
18. Real Life Example ~27.9 seconds to process 6000 Images
19. Real Life Example ~5.6 seconds!
20. How Map works?
21. Bonus…..
22. Injecting parallelism in IO bound programs
23. Retrieving multiple web pages using urlopen(). Real Life Example
24. Real Life Example
25. Real Life Example: 15 URLs
26. “There’s more than one way to do it” Feel free to reach out!
27. rounakvyas@outlook.com Thank You ! itsron717 itsron143 https://rounakvyas.me