Pamela McA'Nulty - Things I Wish They Told Me About The Multiprocessing Module in Python 3

If you haven't tried multiprocessing or you are trying to move beyond multiprocessing.map(), you will likely find that using Python's multiprocessing module can get quite intricate and convoluted. This talk focuses on a few techniques (starting, shutting down, data flow, blocking, etc) that will maximize multiprocessing’s efficiency, while also helping you through the complex issues related to coordinating startup and especially shutdown of your multiprocess app.

https://us.pycon.org/2019/schedule/presentation/203/

Transcript

1. @pmcanulty01 Thi s I Wi The d Abo M ti c in Pamela McA’Nulty 1
2. @pmcanulty01 Who Am I? Senior Developer at Bass Player in 2 Pamela AT cloudcity.io Twitter: @pmcanulty01
3. @pmcanulty01 Framing the problem “Some people, when confronted with a problem, think : ‘I know, I’ll use multithreading’. Nothhw tpe yawrve o oblems.” (Eiríkr Åsheim, 2012) 3
4. @pmcanulty01 A Taxonomy of Python Processing • __main__: One Process, Only. • Threads: The Dreaded GIL! • Asyncio: Shuffle the I/O • multiprocessing.Pool: Chew through bites of data. • 3rd Party: Maybe when you’ve gotten bigger. • multiprocessing.Process: The subject of this talk. 4
5. @pmcanulty01 When to use Multiprocessing? • CPU (not I/O) constrained • Data protection from other processes • Connection or operational integrity (e.g. Database transactions) • Library limitations 5
6. @pmcanulty01 Complex Multiprocessing Example 6
7. @pmcanulty01 Complex Multiprocessing Example 7 Main Process
8. @pmcanulty01 Complex Multiprocessing Example 8 Status Subprocess
9. @pmcanulty01 Complex Multiprocessing Example 9 Observation Subprocess
10. @pmcanulty01 Complex Multiprocessing Example 10 Send Subprocess
11. @pmcanulty01 Complex Multiprocessing Example 11 Listen Subprocess
12. @pmcanulty01 Example: IoT HVAC Monitoring Device • Main Process • System Status Sub-process • HVAC Observation Sub-process • Send Sub-process • Listen Sub-Process 12
13. @pmcanulty01 Tip #1: Don’t Share, Pass Messages • Don’t use shared data, don’t use locks, use Messages • Use multiprocessing.Queues • Each queue handles one type of message • Each Process should read from at most one queue. • Refactor later to use other queuing systems 13
14. @pmcanulty01 Tip #1: Don’t Share, Pass Messages send_q = multiprocessing.Queue() event_q = multiprocessing.Queue(5) # ... event_q.put(“FOO”) # … in another subprocess ... event = event_q.get(block=True, timeout=timeout) # … closing a queue ... queue.close() queue.join_thread() 14
15. @pmcanulty01 Tip #1: Don’t Share, Pass Messages send_q = multiprocessing.Queue() event_q = multiprocessing.Queue(5) # ... event_q.put(“FOO”) # … in another subprocess ... event = event_q.get(block=True, timeout=timeout) # … closing a queue ... queue.close() queue.join_thread() 15
16. @pmcanulty01 Tip #1: Don’t Share, Pass Messages send_q = multiprocessing.Queue() event_q = multiprocessing.Queue(5) # ... event_q.put(“FOO”) # … in another subprocess ... event = event_q.get(block=True, timeout=timeout) # … closing a queue ... queue.close() queue.join_thread() 16
17. @pmcanulty01 Tip #2: Always clean up after yourself • Notify processes to shutdown using “END” messages and a single shutdown_event • All Processes: notice shutdown and then clean up after themselves • Main Process: Cleans up Subprocesses and Queues 17
18. @pmcanulty01 Tip #2: Always clean up after yourself Notice Shutdowns: while not shutdown_event.is_set(): try: item = work_queue.get(block=True, timeout=timeout) except queue.Empty: continue if item == "END": break # …. 18
19. @pmcanulty01 Tip #2: Always clean up after yourself Main Process tells subprocesses to shut down def stop_procs(self): self.shutdown_event.set() ... 19
20. @pmcanulty01 Tip #2: Always clean up after yourself Main Process cleans up subprocesses end_time = time.time() + self.STOP_WAIT_SECS for proc in self.procs: join_secs = max( 0.0, min(end_time - time.time(), STOP_WAIT_SECS) ) proc.proc.join(join_secs) 20
21. @pmcanulty01 Tip #2: Always clean up after yourself Main Process cleans up subprocesses ... while self.procs: proc = self.procs.pop() if proc.proc.is_alive(): proc.terminate() else: exitcode = proc.proc.exitcode 21
22. @pmcanulty01 Tip #2: Always clean up after yourself Main Process cleans up queues item = work_queue.get(block=False) while item: try: Item = work_queue.get(block=False) except Empty: Break work_queue.close() work_queue.join_thread() 22
23. @pmcanulty01 Tip #2: Always clean up after yourself Main Process cleans up queues item = work_queue.get(block=False) while item: try: Item = work_queue.get(block=False) except Empty: Break work_queue.close() work_queue.join_thread() 23
24. @pmcanulty01 Tip #3: Obey all Signals • Every* process, needs to handle both TERM (kill) and INT (Ctl-C). • Set the shutdown_event the first two times • Third time raise • Maybe change system settings 24
25. @pmcanulty01 Tip #3: Obey all Signals MAX_TERMINATE_CALLED = 3 class SignalObject: def __init__(self, shutdown_event): self.terminate_called = 0 self.shutdown_event = shutdown_event def default_signal_handler(sig, exc_class, sig_num, curr_stack_frame): sig.terminate_called += 1 sig.shutdown_event.set() if sig.terminate_called >= MAX_TERMINATE_CALLED: raise exc_class() 25
26. @pmcanulty01 Tip #4: Don’t ever wait forever • No process should get stuck • Loops must terminate • Blocking calls need timeout • Timeouts based on how long you can wait 26
27. @pmcanulty01 Tip #4: Don’t ever wait forever Getting from Queues: while not shutdown_event.is_set(): try: item = work_queue.get(block=True, timeout=0.05) except queue.Empty: continue ... 27
28. @pmcanulty01 Tip #4: Don’t ever wait forever Polling Sockets: self.socket.settimeout(0.1) self.socket.listen(1) … while not shutdown_event.is_set(): try: (clientsocket, address) = self.socket.accept() except socket.timeout: continue 28
29. @pmcanulty01 Tip # 5 : Report, and log all the things • Use Python logging module. • Use single time relative to application start. • Include Name of process • Must log: Errors, Exception Tracebacks, and Warnings • Should log: Start, Stop, Events • In DEBUG mode: log a lot. Yeah, log even more. 29
30. @pmcanulty01 Tip # 5 : Report, and log all the things Logging function: start_time = time.monotonic() def _logger(name, level, msg, exc_info=None): elapsed = time.monotonic() - start_time hours = int(elapsed // 60) seconds = elapsed - (hours * 60) logging.log(level, f'{hours:3}:{seconds:06.3f} {name:20} {msg}', exc_info=exc_info) 30
31. @pmcanulty01 Tip # 5 : Report, and log all the things Logging Output Sample, Startup : DEBUG:root: 0:00.008 SEND Worker Proc.__init__ starting : SEND DEBUG:root: 0:00.013 SEND Worker Entering QueueProcWorker.init_args : (<... object at ...>,) DEBUG:root: 0:00.013 SEND Worker Entering init_signals DEBUG:root: 0:00.014 SEND Worker Entering QueueProcWorker.main_loop DEBUG:root: 0:00.014 SEND Worker Proc.__init__ starting : SEND got True DEBUG:root: 0:00.015 LISTEN Worker Proc.__init__ starting : LISTEN DEBUG:root: 0:00.019 LISTEN Worker Entering init_signals DEBUG:root: 0:00.022 LISTEN Worker Entering main_loop DEBUG:root: 0:00.022 LISTEN Worker Proc.__init__ starting : LISTEN got True DEBUG:root: 0:00.035 OBSERVATION Worker Proc.__init__ starting : OBSERVATION DEBUG:root: 0:00.039 OBSERVATION Worker Entering init_signals DEBUG:root: 0:00.040 OBSERVATION Worker Entering startup DEBUG:root: 0:00.042 OBSERVATION Worker Entering TimerProcWorker.main_loop DEBUG:root: 0:00.043 OBSERVATION Worker Proc.__init__ starting : OBSERVATION got True 31
32. @pmcanulty01 Tip # 5 : Report, and log all the things Logging Output Sample, Shutdown : ^C INFO:root: 0:03.128 OBSERVATION Worker Normal Shutdown INFO:root: 0:03.128 STATUS Worker Normal Shutdown DEBUG:root: 0:03.130 OBSERVATION Worker Entering shutdown DEBUG:root: 0:03.130 STATUS Worker Entering shutdown ERROR:root: 0:03.131 MAIN Unknown Event: OBSERVATION - SHUTDOWN : Normal DEBUG:root: 0:03.132 SEND Worker QueueProcWorker.main_loop received 'END' message INFO:root: 0:03.133 SEND Worker Normal Shutdown INFO:root: 0:04.025 LISTEN Worker Normal Shutdown DEBUG:root: 0:04.028 MAIN Process OBSERVATION ended with exitcode 0 DEBUG:root: 0:04.028 MAIN Process STATUS ended with exitcode 0 DEBUG:root: 0:04.028 MAIN Process LISTEN ended with exitcode 0 DEBUG:root: 0:04.028 MAIN Process SEND ended with exitcode 0 32
33. @pmcanulty01 Conclusion 1. Don’t Share, Pass Messages 2. Always clean up after yourself 3. Handle TERM and INT signals 4. Don’t ever wait forever 5. Report, and log all the things 33
34. @pmcanulty01 Resources My recent blog post on the topic : https://tinyurl.com/yy5n3cmf “Programming guidelines” section from the Python docs. https://tinyurl.com/yxg852g2 My work-in-progress mptools library https://github.com/PamelaM/mptools Email: pamela AT cloudcity.io Twitter: @pmcanulty01 34