Parallel Processing with Daemons in PHP - Splitting Big Problems Into Smaller Pieces

Slide 1

Slide 1 text

Lucas Mendes | Software Architect | @devsdmf PARALLEL PROCESSING   WITH DAEMONS IN PHP SPLITTING BIG PROBLEMS INTO SMALLER PIECES

Slide 2

Slide 2 text

$ whoami

Slide 3

Slide 3 text

PARALLEL PROCESSING WITH DAEMONS IN PHP AGENDA ▸ Parallel Computing ▸ Daemons ▸ A bit of PHP ▸ The Problem to Solve ▸ The Implementation ▸ Implementing a Daemon in PHP ▸ Code time

Slide 4

Slide 4 text

PARALLEL COMPUTING GETTING THINGS DONE IN HALF A TIME

Slide 5

Slide 5 text

WTF IS PARALLEL COMPUTING ?!

Slide 6

Slide 6 text

PARALLEL COMPUTING IS A TYPE OF COMPUTATION IN WHICH MANY CALCULATIONS OR THE EXECUTION OF PROCESS ARE CARRIED SIMULTANEOUSLY. Wikipedia. PARALLEL PROCESSING WITH DAEMONS IN PHP

Slide 7

Slide 7 text

PARALLEL VS DISTRIBUTED VS CONCURRENT

Slide 8

Slide 8 text

DAEMONS

Slide 9

Slide 9 text

A DAEMON IS A LONG-RUNNING BACKGROUND PROCESS THAT ANSWERS REQUESTS FOR SERVICES. Indiana University - Knowledge Base PARALLEL PROCESSING WITH DAEMONS IN PHP

Slide 10

Slide 10 text

BUT EVERY BACKGROUND PROCESS IS A DAEMON ?

Slide 11

Slide 11 text

PARALLEL PROCESSING WITH DAEMONS IN PHP A DAEMON SHOULD… ▸ Always acts as a background process ▸ Not allow direct user interaction ▸ Postﬁx the process name with the letter “d” (i.e. httpd, syslogd…) ▸ Respond to signals sent by other process or by the operating system ▸ Exit nicely ▸ Restart if needed

Slide 12

Slide 12 text

PHP

Slide 13

Slide 13 text

PARALLEL PROCESSING WITH DAEMONS IN PHP PHP OVERVIEW ▸ PHP: Hypertext Preprocessor ▸ Created in 1994 by Rasmus Lerdof ▸ Interpreted, imperative, procedural, object-oriented ▸ Dynamic type ▸ Synchronous by nature ▸ Single process and single threaded

Slide 14

Slide 14 text

THE PROBLEM

Slide 15

Slide 15 text

PARALLEL PROCESSING WITH DAEMONS IN PHP THE PROBLEM ▸ Every month Correios updates its CEP database ▸ Every month we need to update our database too ▸ There are more than 1M CEPs until now ▸ The provide this database in the CSV format ▸ The CSV ﬁle has 94.4MB of plain text

Slide 16

Slide 16 text

THE IMPLEMENTATION

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

FIRST ATTEMPT

Slide 19

Slide 19 text

PARALLEL PROCESSING WITH DAEMONS IN PHP CONVENTIONAL PHP

Slide 20

Slide 20 text

PARALLEL PROCESSING WITH DAEMONS IN PHP CONVENTIONAL PHP - PROS ▸ Easy to implement

Slide 21

Slide 21 text

PARALLEL PROCESSING WITH DAEMONS IN PHP CONVENTIONAL PHP - CONS ▸ Long process time ▸ Blocks the UI ▸ Blocks the I/O ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout can abort the process

Slide 22

Slide 22 text

IF WE MAKE IT ASYNCHRONOUS ?

Slide 23

Slide 23 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING ASYNCHRONOUS PROGRAMMING

Slide 24

Slide 24 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING ASYNC PROGRAMMING - PROS ▸ Async Implementation ▸ Don’t block the I/O

Slide 25

Slide 25 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING ASYNC PROGRAMMING - CONS ▸ Long process time ▸ Blocks the UI ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout

Slide 26

Slide 26 text

MAYBE WE COULD BREAK IT IN THREADS…

Slide 27

Slide 27 text

PARALLEL PROCESSING WITH DAEMONS IN PHP MULTI-THREADED PHP

Slide 28

Slide 28 text

PARALLEL PROCESSING WITH DAEMONS IN PHP MULTI-THREADED PHP - PROS ▸ Faster than procedural implementation ▸ Better usage of serve resources

Slide 29

Slide 29 text

PARALLEL PROCESSING WITH DAEMONS IN PHP MULTI-THREADED PHP - CONS ▸ Requires PHP 7.2+ w/ ZTS ▸ Extension dependent ▸ Blocks the UI ▸ Blocks the I/O ▸ If fails, needs user retry

Slide 30

Slide 30 text

SO THE PROBLEM IS NOT JUST THE IMPLEMENTATION…

Slide 31

Slide 31 text

BUT THE ARCHITECTURE

Slide 32

Slide 32 text

WHAT ABOUT AN ASYNC ARCHITECTURE ?

Slide 33

Slide 33 text

LETS TRY WITH A CRONTAB

Slide 34

Slide 34 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING CRONTAB

Slide 35

Slide 35 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING CRONTAB - PROS ▸ Don’t block the UI ▸ If fails, can retry in next batch

Slide 36

Slide 36 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING CRONTAB - CONS ▸ Time based ▸ Blocks the I/O ▸ Bad server resource usage ▸ Possible memory leaks

Slide 37

Slide 37 text

WE HAVE NOT TRIED WITH GEARMAN

Slide 38

Slide 38 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING GERMAN

Slide 39

Slide 39 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING GEARMAN - PROS ▸ Abstracts the architecture ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O

Slide 40

Slide 40 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING GEARMAN - CONS ▸ Not too easy to implement ▸ Needs to conﬁgure and deploy a Gearman server ▸ Extension dependent ▸ External libraries dependent (libgearman, libevent, uuid) ▸ Doesn’t support newer versions of PHP (only 5.3 to 5.6) ▸ Not currently maintained

Slide 41

Slide 41 text

LETS TRY WITH DAEMONS

Slide 42

Slide 42 text

PCNTL

Slide 43

Slide 43 text

PROCESS CONTROL SUPPORT IN PHP IMPLEMENTS THE UNIX STYLE OF PROCESS CREATION, PROGRAM EXECUTION, SIGNAL HANDLING AND PROCESS TERMINATION. PHP Official Documentation PARALLEL PROCESSING WITH DAEMONS IN PHP

Slide 44

Slide 44 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING PCNTL

Slide 45

Slide 45 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING PCNTL - PROS ▸ Easy to implement ▸ Architecture control ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O ▸ Native in PHP, needs to be compiled with, but still native

Slide 46

Slide 46 text

PARALLEL PROCESSING WITH DAEMONS IN PHP USING FCNTL - CONS ▸ Doesn’t work on Windows

Slide 47

Slide 47 text

WHO CARES?

Slide 48

Slide 48 text

IMPLEMENTING A DAEMON WITH   PCNTL

Slide 49

Slide 49 text

SIMPLE PROGRAM WITH PCNTL

Slide 50

Slide 50 text

PARALLEL PROCESSING WITH DAEMONS IN PHP SIMPLEM PROGRAM WITH PCNTL

Slide 51

Slide 51 text

ZOMBIES   VS   ORPHANS

Slide 52

Slide 52 text

PARALLEL PROCESSING WITH DAEMONS IN PHP ZOMBIES VS ORPHANS ▸ Zombies are dead ▸ Orphans are children whose parents has died

Slide 53

Slide 53 text

USING SIGNALS

Slide 54

Slide 54 text

SIGNALS ARE A LIMITED FORM OF IPC, TYPICALLY USED IN UNIX, UNIX-LIKE AND OTHER POSIX-COMPLIANT OPERATING SYSTEMS. A SIGNAL IS AN ASYNCHRONOUS NOTIFICATION SENT TO A PROCESS TO NOTIFY IT OF AN EVENT THAT OCCURRED. Wikipedia. PARALLEL PROCESSING WITH DAEMONS IN PHP

Slide 55

Slide 55 text

PARALLEL PROCESSING WITH DAEMONS IN PHP SOME SIGNALS AVAILABLE… ▸ SIGCHLD ▸ SIGHUP ▸ SIGINT ▸ SIGKILL ▸ SIGTERM ▸ So many others…

Slide 56

Slide 56 text

HANDLING SIGNALS

Slide 57

Slide 57 text

PARALLEL PROCESSING WITH DAEMONS IN PHP HANDLING SIGNALS

Slide 58

Slide 58 text

CLOSING DESCRIPTORS

Slide 59

Slide 59 text

PARALLEL PROCESSING WITH DAEMONS IN PHP CLOSING DESCRIPTORS

Slide 60

Slide 60 text

SETTING UP THE SESSION, USER AND GROUP

Slide 61

Slide 61 text

PARALLEL PROCESSING WITH DAEMONS IN PHP SETTTING UP THE SESSION, USER AND GROUP

Slide 62

Slide 62 text

LETS CODE!

Slide 63

Slide 63 text

THANK YOU! Lucas Mendes  Software Architect at Tienda Nube  about.me/devsdmf We're hiring, join the crew!   bit.ly/work-at-tiendanube