Scaling PHP with Daemons and Long-Running Processes

Scale a web application is not an easy task, even harder when we need to deal with lots of data processing without language features like async programming and multi-threading. In this talk, I will cover some possibilites when we need to split and improve batch processing using PHP.

Presented at PHP Conference 2018.

Transcript

1. Lucas Mendes | Product Owner | @devsdmf SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES
2. $ whoami
3. DAEMONS
4. A DAEMON IS A LONG-RUNNING BACKGROUND PROCESS THAT ANSWERS REQUESTS FOR SERVICES. Indiana University - Knowledge Base SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES
5. BUT EVERY BACKGROUND PROCESS IS A DAEMON ?
6. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES A DAEMON SHOULD… ▸ Always acts as a background process ▸ Not allow direct user interaction ▸ Postfix the process name with the letter “d” (i.e. httpd, syslogd…) ▸ Respond to signals sent by other process or by the operating system ▸ Exit nicely ▸ Restart if needed
7. THE PROBLEM
8. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES THE PROBLEM ▸ Every month Correios updates its CEP database ▸ Every month we need to update our database too ▸ There are more than 1M CEPs until now ▸ The provide this database in the CSV format ▸ The CSV file has 94.4MB of plain text
9. THE IMPLEMENTATION
10. FIRST ATTEMPT
11. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CONVENTIONAL PHP
12. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CONVENTIONAL PHP - PROS ▸ Easy to implement
13. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CONVENTIONAL PHP - CONS ▸ Long process time ▸ Blocks the UI ▸ Blocks the I/O ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout can abort the process
14. IF WE MAKE IT ASYNCHRONOUS ?
15. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNCHRONOUS PROGRAMMING
16. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNC PROGRAMMING - PROS ▸ Async Implementation ▸ Don’t block the I/O
17. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNC PROGRAMMING - CONS ▸ Long process time ▸ Blocks the UI ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout
18. MAYBE WE COULD BREAK IT IN THREADS…
19. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP
20. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP - PROS ▸ Faster than procedural implementation ▸ Better usage of serve resources
21. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP - CONS ▸ Requires PHP 7.2+ w/ ZTS ▸ Extension dependent ▸ Blocks the UI ▸ Blocks the I/O ▸ If fails, needs user retry
22. SO THE PROBLEM IS NOT JUST THE IMPLEMENTATION…
23. BUT THE ARCHITECTURE
24. WHAT ABOUT AN ASYNC ARCHITECTURE ?
25. LETS TRY WITH A CRONTAB
26. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB
27. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB - PROS ▸ Don’t block the UI ▸ If fails, can retry in next batch
28. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB - CONS ▸ Time based ▸ Blocks the I/O ▸ Bad server resource usage ▸ Possible memory leaks
29. WE HAVE NOT TRIED WITH GEARMAN
30. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GERMAN
31. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GEARMAN - PROS ▸ Abstracts the architecture ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O
32. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GEARMAN - CONS ▸ Not too easy to implement ▸ Needs to configure and deploy a Gearman server ▸ Extension dependent ▸ External libraries dependent (libgearman, libevent, uuid
33. LETS TRY WITH DAEMONS
34. PCNTL
35. PROCESS CONTROL SUPPORT IN PHP IMPLEMENTS THE UNIX STYLE OF PROCESS CREATION, PROGRAM EXECUTION, SIGNAL HANDLING AND PROCESS TERMINATION. PHP Official Documentation SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES
36. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING PCNTL
37. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING PCNTL - PROS ▸ Easy to implement ▸ Architecture control ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O ▸ Native in PHP, needs to be compiled with, but still native
38. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING FCNTL - CONS ▸ Doesn’t work on Windows
39. WHO CARES?
40. IMPLEMENTING A DAEMON WITH 
 PCNTL
41. SIMPLE PROGRAM WITH PCNTL
42. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES SIMPLEM PROGRAM WITH PCNTL
43. ZOMBIES 
 VS 
 ORPHANS
44. PARALLEL PROCESSING WITH DAEMONS IN PHP ZOMBIES VS ORPHANS ▸ Zombies are dead ▸ Orphans are children whose parents has died
45. USING SIGNALS
46. SIGNALS ARE A LIMITED FORM OF IPC, TYPICALLY USED IN UNIX, UNIX-LIKE AND OTHER POSIX-COMPLIANT OPERATING SYSTEMS. A SIGNAL IS AN ASYNCHRONOUS NOTIFICATION SENT TO A PROCESS TO NOTIFY IT OF AN EVENT THAT OCCURRED. Wikipedia. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES
47. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES SOME SIGNALS AVAILABLE… ▸ SIGCHLD ▸ SIGHUP ▸ SIGINT ▸ SIGKILL ▸ SIGTERM ▸ So many others…
48. HANDLING SIGNALS
49. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES HANDLING SIGNALS
50. CLOSING DESCRIPTORS
51. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CLOSING DESCRIPTORS
52. SETTING UP THE SESSION, USER AND GROUP
53. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES SETTTING UP THE SESSION, USER AND GROUP
54. TIME TO DEMO https://github.com/devsdmf/csv-processor
55. THANK YOU! Lucas Mendes
 Product Owner at Tienda Nube
 Find me: @devsdmf We're hiring, join the crew! 
 bit.ly/work-at-tiendanube