Pamela McA'Nulty - Things I Wish They Told Me About The Multiprocessing Module in Python 3

If you haven't tried multiprocessing or you are trying to move beyond multiprocessing.map(), you will likely find that using Python's multiprocessing module can get quite intricate and convoluted. This talk focuses on a few techniques (starting, shutting down, data flow, blocking, etc) that will maximize multiprocessing’s efficiency, while also helping you through the complex issues related to coordinating startup and especially shutdown of your multiprocess app.

https://us.pycon.org/2019/schedule/presentation/203/

Transcript

1. @pmcanulty01
   Thi s I Wi
   The d
   Abo M ti c in
   Pamela McA’Nulty
   1
   

   View Slide

2. @pmcanulty01
   Who Am I?
   Senior Developer at
   Bass Player in
   2
   Pamela AT cloudcity.io
   Twitter: @pmcanulty01
   

   View Slide

3. @pmcanulty01
   Framing the problem
   “Some people, when confronted with a problem,
   think : ‘I know, I’ll use multithreading’. Nothhw tpe
   yawrve o oblems.” (Eiríkr Åsheim, 2012)
   3
   

   View Slide

4. @pmcanulty01
   A Taxonomy of Python Processing
   ● __main__: One Process, Only.
   ● Threads: The Dreaded GIL!
   ● Asyncio: Shuffle the I/O
   ● multiprocessing.Pool: Chew through bites of data.
   ● 3rd Party: Maybe when you’ve gotten bigger.
   ● multiprocessing.Process: The subject of this talk.
   4
   

   View Slide

5. @pmcanulty01
   When to use Multiprocessing?
   ● CPU (not I/O) constrained
   ● Data protection from other processes
   ● Connection or operational integrity (e.g. Database
   transactions)
   ● Library limitations
   5
   

   View Slide

6. @pmcanulty01
   Complex Multiprocessing Example
   6
   

   View Slide

7. @pmcanulty01
   Complex Multiprocessing Example
   7
   Main Process
   

   View Slide

8. @pmcanulty01
   Complex Multiprocessing Example
   8
   Status Subprocess
   

   View Slide

9. @pmcanulty01
   Complex Multiprocessing Example
   9
   Observation
   Subprocess
   

   View Slide

10. @pmcanulty01
    Complex Multiprocessing Example
    10
    Send Subprocess
    

    View Slide

11. @pmcanulty01
    Complex Multiprocessing Example
    11
    Listen Subprocess
    

    View Slide

12. @pmcanulty01
    Example: IoT HVAC Monitoring Device
    ● Main Process
    ● System Status Sub-process
    ● HVAC Observation Sub-process
    ● Send Sub-process
    ● Listen Sub-Process
    12
    

    View Slide

13. @pmcanulty01
    Tip #1: Don’t Share, Pass Messages
    ● Don’t use shared data, don’t use locks, use
    Messages
    ● Use multiprocessing.Queues
    ● Each queue handles one type of message
    ● Each Process should read from at most one queue.
    ● Refactor later to use other queuing systems
    13
    

    View Slide

14. @pmcanulty01
    Tip #1: Don’t Share, Pass Messages
    send_q = multiprocessing.Queue()
    event_q = multiprocessing.Queue(5)
    # ...
    event_q.put(“FOO”)
    # … in another subprocess ...
    event = event_q.get(block=True, timeout=timeout)
    # … closing a queue ...
    queue.close()
    queue.join_thread()
    14
    

    View Slide

15. @pmcanulty01
    Tip #1: Don’t Share, Pass Messages
    send_q = multiprocessing.Queue()
    event_q = multiprocessing.Queue(5)
    # ...
    event_q.put(“FOO”)
    # … in another subprocess ...
    event = event_q.get(block=True, timeout=timeout)
    # … closing a queue ...
    queue.close()
    queue.join_thread()
    15
    

    View Slide

16. @pmcanulty01
    Tip #1: Don’t Share, Pass Messages
    send_q = multiprocessing.Queue()
    event_q = multiprocessing.Queue(5)
    # ...
    event_q.put(“FOO”)
    # … in another subprocess ...
    event = event_q.get(block=True, timeout=timeout)
    # … closing a queue ...
    queue.close()
    queue.join_thread()
    16
    

    View Slide

17. @pmcanulty01
    Tip #2: Always clean up after yourself
    ● Notify processes to shutdown using “END” messages
    and a single shutdown_event
    ● All Processes: notice shutdown and then clean up
    after themselves
    ● Main Process: Cleans up Subprocesses and Queues
    17
    

    View Slide

18. @pmcanulty01
    Tip #2: Always clean up after yourself
    Notice Shutdowns:
    while not shutdown_event.is_set():
    try:
    item = work_queue.get(block=True, timeout=timeout)
    except queue.Empty:
    continue
    if item == "END":
    break
    # ….
    18
    

    View Slide

19. @pmcanulty01
    Tip #2: Always clean up after yourself
    Main Process tells subprocesses to shut down
    def stop_procs(self):
    self.shutdown_event.set()
    ...
    19
    

    View Slide

20. @pmcanulty01
    Tip #2: Always clean up after yourself
    Main Process cleans up subprocesses
    end_time = time.time() + self.STOP_WAIT_SECS
    for proc in self.procs:
    join_secs = max(
    0.0,
    min(end_time - time.time(), STOP_WAIT_SECS)
    )
    proc.proc.join(join_secs)
    20
    

    View Slide

21. @pmcanulty01
    Tip #2: Always clean up after yourself
    Main Process cleans up subprocesses
    ...
    while self.procs:
    proc = self.procs.pop()
    if proc.proc.is_alive():
    proc.terminate()
    else:
    exitcode = proc.proc.exitcode
    21
    

    View Slide

22. @pmcanulty01
    Tip #2: Always clean up after yourself
    Main Process cleans up queues
    item = work_queue.get(block=False)
    while item:
    try:
    Item = work_queue.get(block=False)
    except Empty:
    Break
    work_queue.close()
    work_queue.join_thread()
    22
    

    View Slide

23. @pmcanulty01
    Tip #2: Always clean up after yourself
    Main Process cleans up queues
    item = work_queue.get(block=False)
    while item:
    try:
    Item = work_queue.get(block=False)
    except Empty:
    Break
    work_queue.close()
    work_queue.join_thread()
    23
    

    View Slide

24. @pmcanulty01
    Tip #3: Obey all Signals
    ● Every* process, needs to handle both TERM (kill)
    and INT (Ctl-C).
    ● Set the shutdown_event the first two times
    ● Third time raise
    ● Maybe change system settings
    24
    

    View Slide

25. @pmcanulty01
    Tip #3: Obey all Signals
    MAX_TERMINATE_CALLED = 3
    class SignalObject:
    def __init__(self, shutdown_event):
    self.terminate_called = 0
    self.shutdown_event = shutdown_event
    def default_signal_handler(sig, exc_class, sig_num,
    curr_stack_frame):
    sig.terminate_called += 1
    sig.shutdown_event.set()
    if sig.terminate_called >= MAX_TERMINATE_CALLED:
    raise exc_class()
    25
    

    View Slide

26. @pmcanulty01
    Tip #4: Don’t ever wait forever
    ● No process should get stuck
    ● Loops must terminate
    ● Blocking calls need timeout
    ● Timeouts based on how long you can wait
    26
    

    View Slide

27. @pmcanulty01
    Tip #4: Don’t ever wait forever
    Getting from Queues:
    while not shutdown_event.is_set():
    try:
    item = work_queue.get(block=True, timeout=0.05)
    except queue.Empty:
    continue
    ...
    27
    

    View Slide

28. @pmcanulty01
    Tip #4: Don’t ever wait forever
    Polling Sockets:
    self.socket.settimeout(0.1)
    self.socket.listen(1)
    …
    while not shutdown_event.is_set():
    try:
    (clientsocket, address) = self.socket.accept()
    except socket.timeout:
    continue
    28
    

    View Slide

29. @pmcanulty01
    Tip # 5 : Report, and log all the things
    ● Use Python logging module.
    ● Use single time relative to application start.
    ● Include Name of process
    ● Must log: Errors, Exception Tracebacks, and Warnings
    ● Should log: Start, Stop, Events
    ● In DEBUG mode: log a lot. Yeah, log even more.
    29
    

    View Slide

30. @pmcanulty01
    Tip # 5 : Report, and log all the things
    Logging function:
    start_time = time.monotonic()
    def _logger(name, level, msg, exc_info=None):
    elapsed = time.monotonic() - start_time
    hours = int(elapsed // 60)
    seconds = elapsed - (hours * 60)
    logging.log(level,
    f'{hours:3}:{seconds:06.3f} {name:20} {msg}',
    exc_info=exc_info)
    30
    

    View Slide

31. @pmcanulty01
    Tip # 5 : Report, and log all the things
    Logging Output Sample, Startup :
    DEBUG:root: 0:00.008 SEND Worker Proc.__init__ starting : SEND
    DEBUG:root: 0:00.013 SEND Worker Entering QueueProcWorker.init_args : (,)
    DEBUG:root: 0:00.013 SEND Worker Entering init_signals
    DEBUG:root: 0:00.014 SEND Worker Entering QueueProcWorker.main_loop
    DEBUG:root: 0:00.014 SEND Worker Proc.__init__ starting : SEND got True
    DEBUG:root: 0:00.015 LISTEN Worker Proc.__init__ starting : LISTEN
    DEBUG:root: 0:00.019 LISTEN Worker Entering init_signals
    DEBUG:root: 0:00.022 LISTEN Worker Entering main_loop
    DEBUG:root: 0:00.022 LISTEN Worker Proc.__init__ starting : LISTEN got True
    DEBUG:root: 0:00.035 OBSERVATION Worker Proc.__init__ starting : OBSERVATION
    DEBUG:root: 0:00.039 OBSERVATION Worker Entering init_signals
    DEBUG:root: 0:00.040 OBSERVATION Worker Entering startup
    DEBUG:root: 0:00.042 OBSERVATION Worker Entering TimerProcWorker.main_loop
    DEBUG:root: 0:00.043 OBSERVATION Worker Proc.__init__ starting : OBSERVATION got True
    31
    

    View Slide

32. @pmcanulty01
    Tip # 5 : Report, and log all the things
    Logging Output Sample, Shutdown :
    ^C
    INFO:root: 0:03.128 OBSERVATION Worker Normal Shutdown
    INFO:root: 0:03.128 STATUS Worker Normal Shutdown
    DEBUG:root: 0:03.130 OBSERVATION Worker Entering shutdown
    DEBUG:root: 0:03.130 STATUS Worker Entering shutdown
    ERROR:root: 0:03.131 MAIN Unknown Event: OBSERVATION - SHUTDOWN : Normal
    DEBUG:root: 0:03.132 SEND Worker QueueProcWorker.main_loop received 'END' message
    INFO:root: 0:03.133 SEND Worker Normal Shutdown
    INFO:root: 0:04.025 LISTEN Worker Normal Shutdown
    DEBUG:root: 0:04.028 MAIN Process OBSERVATION ended with exitcode 0
    DEBUG:root: 0:04.028 MAIN Process STATUS ended with exitcode 0
    DEBUG:root: 0:04.028 MAIN Process LISTEN ended with exitcode 0
    DEBUG:root: 0:04.028 MAIN Process SEND ended with exitcode 0
    32
    

    View Slide

33. @pmcanulty01
    Conclusion
    1. Don’t Share, Pass Messages
    2. Always clean up after yourself
    3. Handle TERM and INT signals
    4. Don’t ever wait forever
    5. Report, and log all the things
    33
    

    View Slide

34. @pmcanulty01
    Resources
    My recent blog post on the topic :
    https://tinyurl.com/yy5n3cmf
    “Programming guidelines” section from the Python docs.
    https://tinyurl.com/yxg852g2
    My work-in-progress mptools library
    https://github.com/PamelaM/mptools
    Email: pamela AT cloudcity.io Twitter: @pmcanulty01
    34
    

    View Slide