Slide 1

Slide 1 text

@pmcanulty01 Thi s I Wi The d Abo M ti c in Pamela McA’Nulty 1

Slide 2

Slide 2 text

@pmcanulty01 Who Am I? Senior Developer at Bass Player in 2 Pamela AT cloudcity.io Twitter: @pmcanulty01

Slide 3

Slide 3 text

@pmcanulty01 Framing the problem “Some people, when confronted with a problem, think : ‘I know, I’ll use multithreading’. Nothhw tpe yawrve o oblems.” (Eiríkr Åsheim, 2012) 3

Slide 4

Slide 4 text

@pmcanulty01 A Taxonomy of Python Processing ● __main__: One Process, Only. ● Threads: The Dreaded GIL! ● Asyncio: Shuffle the I/O ● multiprocessing.Pool: Chew through bites of data. ● 3rd Party: Maybe when you’ve gotten bigger. ● multiprocessing.Process: The subject of this talk. 4

Slide 5

Slide 5 text

@pmcanulty01 When to use Multiprocessing? ● CPU (not I/O) constrained ● Data protection from other processes ● Connection or operational integrity (e.g. Database transactions) ● Library limitations 5

Slide 6

Slide 6 text

@pmcanulty01 Complex Multiprocessing Example 6

Slide 7

Slide 7 text

@pmcanulty01 Complex Multiprocessing Example 7 Main Process

Slide 8

Slide 8 text

@pmcanulty01 Complex Multiprocessing Example 8 Status Subprocess

Slide 9

Slide 9 text

@pmcanulty01 Complex Multiprocessing Example 9 Observation Subprocess

Slide 10

Slide 10 text

@pmcanulty01 Complex Multiprocessing Example 10 Send Subprocess

Slide 11

Slide 11 text

@pmcanulty01 Complex Multiprocessing Example 11 Listen Subprocess

Slide 12

Slide 12 text

@pmcanulty01 Example: IoT HVAC Monitoring Device ● Main Process ● System Status Sub-process ● HVAC Observation Sub-process ● Send Sub-process ● Listen Sub-Process 12

Slide 13

Slide 13 text

@pmcanulty01 Tip #1: Don’t Share, Pass Messages ● Don’t use shared data, don’t use locks, use Messages ● Use multiprocessing.Queues ● Each queue handles one type of message ● Each Process should read from at most one queue. ● Refactor later to use other queuing systems 13

Slide 14

Slide 14 text

@pmcanulty01 Tip #1: Don’t Share, Pass Messages send_q = multiprocessing.Queue() event_q = multiprocessing.Queue(5) # ... event_q.put(“FOO”) # … in another subprocess ... event = event_q.get(block=True, timeout=timeout) # … closing a queue ... queue.close() queue.join_thread() 14

Slide 15

Slide 15 text

@pmcanulty01 Tip #1: Don’t Share, Pass Messages send_q = multiprocessing.Queue() event_q = multiprocessing.Queue(5) # ... event_q.put(“FOO”) # … in another subprocess ... event = event_q.get(block=True, timeout=timeout) # … closing a queue ... queue.close() queue.join_thread() 15

Slide 16

Slide 16 text

@pmcanulty01 Tip #1: Don’t Share, Pass Messages send_q = multiprocessing.Queue() event_q = multiprocessing.Queue(5) # ... event_q.put(“FOO”) # … in another subprocess ... event = event_q.get(block=True, timeout=timeout) # … closing a queue ... queue.close() queue.join_thread() 16

Slide 17

Slide 17 text

@pmcanulty01 Tip #2: Always clean up after yourself ● Notify processes to shutdown using “END” messages and a single shutdown_event ● All Processes: notice shutdown and then clean up after themselves ● Main Process: Cleans up Subprocesses and Queues 17

Slide 18

Slide 18 text

@pmcanulty01 Tip #2: Always clean up after yourself Notice Shutdowns: while not shutdown_event.is_set(): try: item = work_queue.get(block=True, timeout=timeout) except queue.Empty: continue if item == "END": break # …. 18

Slide 19

Slide 19 text

@pmcanulty01 Tip #2: Always clean up after yourself Main Process tells subprocesses to shut down def stop_procs(self): self.shutdown_event.set() ... 19

Slide 20

Slide 20 text

@pmcanulty01 Tip #2: Always clean up after yourself Main Process cleans up subprocesses end_time = time.time() + self.STOP_WAIT_SECS for proc in self.procs: join_secs = max( 0.0, min(end_time - time.time(), STOP_WAIT_SECS) ) proc.proc.join(join_secs) 20

Slide 21

Slide 21 text

@pmcanulty01 Tip #2: Always clean up after yourself Main Process cleans up subprocesses ... while self.procs: proc = self.procs.pop() if proc.proc.is_alive(): proc.terminate() else: exitcode = proc.proc.exitcode 21

Slide 22

Slide 22 text

@pmcanulty01 Tip #2: Always clean up after yourself Main Process cleans up queues item = work_queue.get(block=False) while item: try: Item = work_queue.get(block=False) except Empty: Break work_queue.close() work_queue.join_thread() 22

Slide 23

Slide 23 text

@pmcanulty01 Tip #2: Always clean up after yourself Main Process cleans up queues item = work_queue.get(block=False) while item: try: Item = work_queue.get(block=False) except Empty: Break work_queue.close() work_queue.join_thread() 23

Slide 24

Slide 24 text

@pmcanulty01 Tip #3: Obey all Signals ● Every* process, needs to handle both TERM (kill) and INT (Ctl-C). ● Set the shutdown_event the first two times ● Third time raise ● Maybe change system settings 24

Slide 25

Slide 25 text

@pmcanulty01 Tip #3: Obey all Signals MAX_TERMINATE_CALLED = 3 class SignalObject: def __init__(self, shutdown_event): self.terminate_called = 0 self.shutdown_event = shutdown_event def default_signal_handler(sig, exc_class, sig_num, curr_stack_frame): sig.terminate_called += 1 sig.shutdown_event.set() if sig.terminate_called >= MAX_TERMINATE_CALLED: raise exc_class() 25

Slide 26

Slide 26 text

@pmcanulty01 Tip #4: Don’t ever wait forever ● No process should get stuck ● Loops must terminate ● Blocking calls need timeout ● Timeouts based on how long you can wait 26

Slide 27

Slide 27 text

@pmcanulty01 Tip #4: Don’t ever wait forever Getting from Queues: while not shutdown_event.is_set(): try: item = work_queue.get(block=True, timeout=0.05) except queue.Empty: continue ... 27

Slide 28

Slide 28 text

@pmcanulty01 Tip #4: Don’t ever wait forever Polling Sockets: self.socket.settimeout(0.1) self.socket.listen(1) … while not shutdown_event.is_set(): try: (clientsocket, address) = self.socket.accept() except socket.timeout: continue 28

Slide 29

Slide 29 text

@pmcanulty01 Tip # 5 : Report, and log all the things ● Use Python logging module. ● Use single time relative to application start. ● Include Name of process ● Must log: Errors, Exception Tracebacks, and Warnings ● Should log: Start, Stop, Events ● In DEBUG mode: log a lot. Yeah, log even more. 29

Slide 30

Slide 30 text

@pmcanulty01 Tip # 5 : Report, and log all the things Logging function: start_time = time.monotonic() def _logger(name, level, msg, exc_info=None): elapsed = time.monotonic() - start_time hours = int(elapsed // 60) seconds = elapsed - (hours * 60) logging.log(level, f'{hours:3}:{seconds:06.3f} {name:20} {msg}', exc_info=exc_info) 30

Slide 31

Slide 31 text

@pmcanulty01 Tip # 5 : Report, and log all the things Logging Output Sample, Startup : DEBUG:root: 0:00.008 SEND Worker Proc.__init__ starting : SEND DEBUG:root: 0:00.013 SEND Worker Entering QueueProcWorker.init_args : (<... object at ...>,) DEBUG:root: 0:00.013 SEND Worker Entering init_signals DEBUG:root: 0:00.014 SEND Worker Entering QueueProcWorker.main_loop DEBUG:root: 0:00.014 SEND Worker Proc.__init__ starting : SEND got True DEBUG:root: 0:00.015 LISTEN Worker Proc.__init__ starting : LISTEN DEBUG:root: 0:00.019 LISTEN Worker Entering init_signals DEBUG:root: 0:00.022 LISTEN Worker Entering main_loop DEBUG:root: 0:00.022 LISTEN Worker Proc.__init__ starting : LISTEN got True DEBUG:root: 0:00.035 OBSERVATION Worker Proc.__init__ starting : OBSERVATION DEBUG:root: 0:00.039 OBSERVATION Worker Entering init_signals DEBUG:root: 0:00.040 OBSERVATION Worker Entering startup DEBUG:root: 0:00.042 OBSERVATION Worker Entering TimerProcWorker.main_loop DEBUG:root: 0:00.043 OBSERVATION Worker Proc.__init__ starting : OBSERVATION got True 31

Slide 32

Slide 32 text

@pmcanulty01 Tip # 5 : Report, and log all the things Logging Output Sample, Shutdown : ^C INFO:root: 0:03.128 OBSERVATION Worker Normal Shutdown INFO:root: 0:03.128 STATUS Worker Normal Shutdown DEBUG:root: 0:03.130 OBSERVATION Worker Entering shutdown DEBUG:root: 0:03.130 STATUS Worker Entering shutdown ERROR:root: 0:03.131 MAIN Unknown Event: OBSERVATION - SHUTDOWN : Normal DEBUG:root: 0:03.132 SEND Worker QueueProcWorker.main_loop received 'END' message INFO:root: 0:03.133 SEND Worker Normal Shutdown INFO:root: 0:04.025 LISTEN Worker Normal Shutdown DEBUG:root: 0:04.028 MAIN Process OBSERVATION ended with exitcode 0 DEBUG:root: 0:04.028 MAIN Process STATUS ended with exitcode 0 DEBUG:root: 0:04.028 MAIN Process LISTEN ended with exitcode 0 DEBUG:root: 0:04.028 MAIN Process SEND ended with exitcode 0 32

Slide 33

Slide 33 text

@pmcanulty01 Conclusion 1. Don’t Share, Pass Messages 2. Always clean up after yourself 3. Handle TERM and INT signals 4. Don’t ever wait forever 5. Report, and log all the things 33

Slide 34

Slide 34 text

@pmcanulty01 Resources My recent blog post on the topic : https://tinyurl.com/yy5n3cmf “Programming guidelines” section from the Python docs. https://tinyurl.com/yxg852g2 My work-in-progress mptools library https://github.com/PamelaM/mptools Email: pamela AT cloudcity.io Twitter: @pmcanulty01 34