Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Pamela McA'Nulty - Things I Wish They Told Me About The Multiprocessing Module in Python 3

Pamela McA'Nulty - Things I Wish They Told Me About The Multiprocessing Module in Python 3

If you haven't tried multiprocessing or you are trying to move beyond multiprocessing.map(), you will likely find that using Python's multiprocessing module can get quite intricate and convoluted. This talk focuses on a few techniques (starting, shutting down, data flow, blocking, etc) that will maximize multiprocessing’s efficiency, while also helping you through the complex issues related to coordinating startup and especially shutdown of your multiprocess app.

https://us.pycon.org/2019/schedule/presentation/203/

53b37e14a09c5a718a39fda61fe1b8e5?s=128

PyCon 2019

May 04, 2019
Tweet

Transcript

  1. @pmcanulty01 Thi s I Wi The d Abo M ti

    c in Pamela McA’Nulty 1
  2. @pmcanulty01 Who Am I? Senior Developer at Bass Player in

    2 Pamela AT cloudcity.io Twitter: @pmcanulty01
  3. @pmcanulty01 Framing the problem “Some people, when confronted with a

    problem, think : ‘I know, I’ll use multithreading’. Nothhw tpe yawrve o oblems.” (Eiríkr Åsheim, 2012) 3
  4. @pmcanulty01 A Taxonomy of Python Processing • __main__: One Process,

    Only. • Threads: The Dreaded GIL! • Asyncio: Shuffle the I/O • multiprocessing.Pool: Chew through bites of data. • 3rd Party: Maybe when you’ve gotten bigger. • multiprocessing.Process: The subject of this talk. 4
  5. @pmcanulty01 When to use Multiprocessing? • CPU (not I/O) constrained

    • Data protection from other processes • Connection or operational integrity (e.g. Database transactions) • Library limitations 5
  6. @pmcanulty01 Complex Multiprocessing Example 6

  7. @pmcanulty01 Complex Multiprocessing Example 7 Main Process

  8. @pmcanulty01 Complex Multiprocessing Example 8 Status Subprocess

  9. @pmcanulty01 Complex Multiprocessing Example 9 Observation Subprocess

  10. @pmcanulty01 Complex Multiprocessing Example 10 Send Subprocess

  11. @pmcanulty01 Complex Multiprocessing Example 11 Listen Subprocess

  12. @pmcanulty01 Example: IoT HVAC Monitoring Device • Main Process •

    System Status Sub-process • HVAC Observation Sub-process • Send Sub-process • Listen Sub-Process 12
  13. @pmcanulty01 Tip #1: Don’t Share, Pass Messages • Don’t use

    shared data, don’t use locks, use Messages • Use multiprocessing.Queues • Each queue handles one type of message • Each Process should read from at most one queue. • Refactor later to use other queuing systems 13
  14. @pmcanulty01 Tip #1: Don’t Share, Pass Messages send_q = multiprocessing.Queue()

    event_q = multiprocessing.Queue(5) # ... event_q.put(“FOO”) # … in another subprocess ... event = event_q.get(block=True, timeout=timeout) # … closing a queue ... queue.close() queue.join_thread() 14
  15. @pmcanulty01 Tip #1: Don’t Share, Pass Messages send_q = multiprocessing.Queue()

    event_q = multiprocessing.Queue(5) # ... event_q.put(“FOO”) # … in another subprocess ... event = event_q.get(block=True, timeout=timeout) # … closing a queue ... queue.close() queue.join_thread() 15
  16. @pmcanulty01 Tip #1: Don’t Share, Pass Messages send_q = multiprocessing.Queue()

    event_q = multiprocessing.Queue(5) # ... event_q.put(“FOO”) # … in another subprocess ... event = event_q.get(block=True, timeout=timeout) # … closing a queue ... queue.close() queue.join_thread() 16
  17. @pmcanulty01 Tip #2: Always clean up after yourself • Notify

    processes to shutdown using “END” messages and a single shutdown_event • All Processes: notice shutdown and then clean up after themselves • Main Process: Cleans up Subprocesses and Queues 17
  18. @pmcanulty01 Tip #2: Always clean up after yourself Notice Shutdowns:

    while not shutdown_event.is_set(): try: item = work_queue.get(block=True, timeout=timeout) except queue.Empty: continue if item == "END": break # …. 18
  19. @pmcanulty01 Tip #2: Always clean up after yourself Main Process

    tells subprocesses to shut down def stop_procs(self): self.shutdown_event.set() ... 19
  20. @pmcanulty01 Tip #2: Always clean up after yourself Main Process

    cleans up subprocesses end_time = time.time() + self.STOP_WAIT_SECS for proc in self.procs: join_secs = max( 0.0, min(end_time - time.time(), STOP_WAIT_SECS) ) proc.proc.join(join_secs) 20
  21. @pmcanulty01 Tip #2: Always clean up after yourself Main Process

    cleans up subprocesses ... while self.procs: proc = self.procs.pop() if proc.proc.is_alive(): proc.terminate() else: exitcode = proc.proc.exitcode 21
  22. @pmcanulty01 Tip #2: Always clean up after yourself Main Process

    cleans up queues item = work_queue.get(block=False) while item: try: Item = work_queue.get(block=False) except Empty: Break work_queue.close() work_queue.join_thread() 22
  23. @pmcanulty01 Tip #2: Always clean up after yourself Main Process

    cleans up queues item = work_queue.get(block=False) while item: try: Item = work_queue.get(block=False) except Empty: Break work_queue.close() work_queue.join_thread() 23
  24. @pmcanulty01 Tip #3: Obey all Signals • Every* process, needs

    to handle both TERM (kill) and INT (Ctl-C). • Set the shutdown_event the first two times • Third time raise • Maybe change system settings 24
  25. @pmcanulty01 Tip #3: Obey all Signals MAX_TERMINATE_CALLED = 3 class

    SignalObject: def __init__(self, shutdown_event): self.terminate_called = 0 self.shutdown_event = shutdown_event def default_signal_handler(sig, exc_class, sig_num, curr_stack_frame): sig.terminate_called += 1 sig.shutdown_event.set() if sig.terminate_called >= MAX_TERMINATE_CALLED: raise exc_class() 25
  26. @pmcanulty01 Tip #4: Don’t ever wait forever • No process

    should get stuck • Loops must terminate • Blocking calls need timeout • Timeouts based on how long you can wait 26
  27. @pmcanulty01 Tip #4: Don’t ever wait forever Getting from Queues:

    while not shutdown_event.is_set(): try: item = work_queue.get(block=True, timeout=0.05) except queue.Empty: continue ... 27
  28. @pmcanulty01 Tip #4: Don’t ever wait forever Polling Sockets: self.socket.settimeout(0.1)

    self.socket.listen(1) … while not shutdown_event.is_set(): try: (clientsocket, address) = self.socket.accept() except socket.timeout: continue 28
  29. @pmcanulty01 Tip # 5 : Report, and log all the

    things • Use Python logging module. • Use single time relative to application start. • Include Name of process • Must log: Errors, Exception Tracebacks, and Warnings • Should log: Start, Stop, Events • In DEBUG mode: log a lot. Yeah, log even more. 29
  30. @pmcanulty01 Tip # 5 : Report, and log all the

    things Logging function: start_time = time.monotonic() def _logger(name, level, msg, exc_info=None): elapsed = time.monotonic() - start_time hours = int(elapsed // 60) seconds = elapsed - (hours * 60) logging.log(level, f'{hours:3}:{seconds:06.3f} {name:20} {msg}', exc_info=exc_info) 30
  31. @pmcanulty01 Tip # 5 : Report, and log all the

    things Logging Output Sample, Startup : DEBUG:root: 0:00.008 SEND Worker Proc.__init__ starting : SEND DEBUG:root: 0:00.013 SEND Worker Entering QueueProcWorker.init_args : (<... object at ...>,) DEBUG:root: 0:00.013 SEND Worker Entering init_signals DEBUG:root: 0:00.014 SEND Worker Entering QueueProcWorker.main_loop DEBUG:root: 0:00.014 SEND Worker Proc.__init__ starting : SEND got True DEBUG:root: 0:00.015 LISTEN Worker Proc.__init__ starting : LISTEN DEBUG:root: 0:00.019 LISTEN Worker Entering init_signals DEBUG:root: 0:00.022 LISTEN Worker Entering main_loop DEBUG:root: 0:00.022 LISTEN Worker Proc.__init__ starting : LISTEN got True DEBUG:root: 0:00.035 OBSERVATION Worker Proc.__init__ starting : OBSERVATION DEBUG:root: 0:00.039 OBSERVATION Worker Entering init_signals DEBUG:root: 0:00.040 OBSERVATION Worker Entering startup DEBUG:root: 0:00.042 OBSERVATION Worker Entering TimerProcWorker.main_loop DEBUG:root: 0:00.043 OBSERVATION Worker Proc.__init__ starting : OBSERVATION got True 31
  32. @pmcanulty01 Tip # 5 : Report, and log all the

    things Logging Output Sample, Shutdown : ^C INFO:root: 0:03.128 OBSERVATION Worker Normal Shutdown INFO:root: 0:03.128 STATUS Worker Normal Shutdown DEBUG:root: 0:03.130 OBSERVATION Worker Entering shutdown DEBUG:root: 0:03.130 STATUS Worker Entering shutdown ERROR:root: 0:03.131 MAIN Unknown Event: OBSERVATION - SHUTDOWN : Normal DEBUG:root: 0:03.132 SEND Worker QueueProcWorker.main_loop received 'END' message INFO:root: 0:03.133 SEND Worker Normal Shutdown INFO:root: 0:04.025 LISTEN Worker Normal Shutdown DEBUG:root: 0:04.028 MAIN Process OBSERVATION ended with exitcode 0 DEBUG:root: 0:04.028 MAIN Process STATUS ended with exitcode 0 DEBUG:root: 0:04.028 MAIN Process LISTEN ended with exitcode 0 DEBUG:root: 0:04.028 MAIN Process SEND ended with exitcode 0 32
  33. @pmcanulty01 Conclusion 1. Don’t Share, Pass Messages 2. Always clean

    up after yourself 3. Handle TERM and INT signals 4. Don’t ever wait forever 5. Report, and log all the things 33
  34. @pmcanulty01 Resources My recent blog post on the topic :

    https://tinyurl.com/yy5n3cmf “Programming guidelines” section from the Python docs. https://tinyurl.com/yxg852g2 My work-in-progress mptools library https://github.com/PamelaM/mptools Email: pamela AT cloudcity.io Twitter: @pmcanulty01 34