Scaling PHP with Daemons and Long-Running Processes

Slide 1

Slide 1 text

Lucas Mendes | Product Owner | @devsdmf SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES

Slide 2

Slide 2 text

$ whoami

Slide 3

Slide 3 text

DAEMONS

Slide 4

Slide 4 text

A DAEMON IS A LONG-RUNNING BACKGROUND PROCESS THAT ANSWERS REQUESTS FOR SERVICES. Indiana University - Knowledge Base SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES

Slide 5

Slide 5 text

BUT EVERY BACKGROUND PROCESS IS A DAEMON ?

Slide 6

Slide 6 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES A DAEMON SHOULD… ▸ Always acts as a background process ▸ Not allow direct user interaction ▸ Postﬁx the process name with the letter “d” (i.e. httpd, syslogd…) ▸ Respond to signals sent by other process or by the operating system ▸ Exit nicely ▸ Restart if needed

Slide 7

Slide 7 text

THE PROBLEM

Slide 8

Slide 8 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES THE PROBLEM ▸ Every month Correios updates its CEP database ▸ Every month we need to update our database too ▸ There are more than 1M CEPs until now ▸ The provide this database in the CSV format ▸ The CSV ﬁle has 94.4MB of plain text

Slide 9

Slide 9 text

THE IMPLEMENTATION

Slide 10

Slide 10 text

FIRST ATTEMPT

Slide 11

Slide 11 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CONVENTIONAL PHP

Slide 12

Slide 12 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CONVENTIONAL PHP - PROS ▸ Easy to implement

Slide 13

Slide 13 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CONVENTIONAL PHP - CONS ▸ Long process time ▸ Blocks the UI ▸ Blocks the I/O ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout can abort the process

Slide 14

Slide 14 text

IF WE MAKE IT ASYNCHRONOUS ?

Slide 15

Slide 15 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNCHRONOUS PROGRAMMING

Slide 16

Slide 16 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNC PROGRAMMING - PROS ▸ Async Implementation ▸ Don’t block the I/O

Slide 17

Slide 17 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNC PROGRAMMING - CONS ▸ Long process time ▸ Blocks the UI ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout

Slide 18

Slide 18 text

MAYBE WE COULD BREAK IT IN THREADS…

Slide 19

Slide 19 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP

Slide 20

Slide 20 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP - PROS ▸ Faster than procedural implementation ▸ Better usage of serve resources

Slide 21

Slide 21 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP - CONS ▸ Requires PHP 7.2+ w/ ZTS ▸ Extension dependent ▸ Blocks the UI ▸ Blocks the I/O ▸ If fails, needs user retry

Slide 22

Slide 22 text

SO THE PROBLEM IS NOT JUST THE IMPLEMENTATION…

Slide 23

Slide 23 text

BUT THE ARCHITECTURE

Slide 24

Slide 24 text

WHAT ABOUT AN ASYNC ARCHITECTURE ?

Slide 25

Slide 25 text

LETS TRY WITH A CRONTAB

Slide 26

Slide 26 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB

Slide 27

Slide 27 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB - PROS ▸ Don’t block the UI ▸ If fails, can retry in next batch

Slide 28

Slide 28 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB - CONS ▸ Time based ▸ Blocks the I/O ▸ Bad server resource usage ▸ Possible memory leaks

Slide 29

Slide 29 text

WE HAVE NOT TRIED WITH GEARMAN

Slide 30

Slide 30 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GERMAN

Slide 31

Slide 31 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GEARMAN - PROS ▸ Abstracts the architecture ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O

Slide 32

Slide 32 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GEARMAN - CONS ▸ Not too easy to implement ▸ Needs to conﬁgure and deploy a Gearman server ▸ Extension dependent ▸ External libraries dependent (libgearman, libevent, uuid

Slide 33

Slide 33 text

LETS TRY WITH DAEMONS

Slide 34

Slide 34 text

PCNTL

Slide 35

Slide 35 text

PROCESS CONTROL SUPPORT IN PHP IMPLEMENTS THE UNIX STYLE OF PROCESS CREATION, PROGRAM EXECUTION, SIGNAL HANDLING AND PROCESS TERMINATION. PHP Official Documentation SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES

Slide 36

Slide 36 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING PCNTL

Slide 37

Slide 37 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING PCNTL - PROS ▸ Easy to implement ▸ Architecture control ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O ▸ Native in PHP, needs to be compiled with, but still native

Slide 38

Slide 38 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING FCNTL - CONS ▸ Doesn’t work on Windows

Slide 39

Slide 39 text

WHO CARES?

Slide 40

Slide 40 text

IMPLEMENTING A DAEMON WITH   PCNTL

Slide 41

Slide 41 text

SIMPLE PROGRAM WITH PCNTL

Slide 42

Slide 42 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES SIMPLEM PROGRAM WITH PCNTL

Slide 43

Slide 43 text

ZOMBIES   VS   ORPHANS

Slide 44

Slide 44 text

PARALLEL PROCESSING WITH DAEMONS IN PHP ZOMBIES VS ORPHANS ▸ Zombies are dead ▸ Orphans are children whose parents has died

Slide 45

Slide 45 text

USING SIGNALS

Slide 46

Slide 46 text

SIGNALS ARE A LIMITED FORM OF IPC, TYPICALLY USED IN UNIX, UNIX-LIKE AND OTHER POSIX-COMPLIANT OPERATING SYSTEMS. A SIGNAL IS AN ASYNCHRONOUS NOTIFICATION SENT TO A PROCESS TO NOTIFY IT OF AN EVENT THAT OCCURRED. Wikipedia. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES

Slide 47

Slide 47 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES SOME SIGNALS AVAILABLE… ▸ SIGCHLD ▸ SIGHUP ▸ SIGINT ▸ SIGKILL ▸ SIGTERM ▸ So many others…

Slide 48

Slide 48 text

HANDLING SIGNALS

Slide 49

Slide 49 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES HANDLING SIGNALS

Slide 50

Slide 50 text

CLOSING DESCRIPTORS

Slide 51

Slide 51 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CLOSING DESCRIPTORS

Slide 52

Slide 52 text

SETTING UP THE SESSION, USER AND GROUP

Slide 53

Slide 53 text

SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES SETTTING UP THE SESSION, USER AND GROUP

Slide 54

Slide 54 text

TIME TO DEMO https://github.com/devsdmf/csv-processor

Slide 55

Slide 55 text

THANK YOU! Lucas Mendes  Product Owner at Tienda Nube  Find me: @devsdmf We're hiring, join the crew!   bit.ly/work-at-tiendanube