Parallel Processing with Daemons in PHP - Splitting Big Problems Into Smaller Pieces

Sometimes we need to process large amounts of data and using the conventional architectural style used by PHP is not a simple thing because the language is synchronous by its nature, so this talk aims to get new insights about the applications architectures giving you new ways to handle this kind of task.

Presented at Campus Party Minas Gerais 2017.

Transcript

1. Lucas Mendes | Software Architect | @devsdmf PARALLEL PROCESSING 
 WITH DAEMONS IN PHP SPLITTING BIG PROBLEMS INTO SMALLER PIECES
2. $ whoami
3. PARALLEL PROCESSING WITH DAEMONS IN PHP AGENDA ▸ Parallel Computing ▸ Daemons ▸ A bit of PHP ▸ The Problem to Solve ▸ The Implementation ▸ Implementing a Daemon in PHP ▸ Code time
4. PARALLEL COMPUTING GETTING THINGS DONE IN HALF A TIME
5. WTF IS PARALLEL COMPUTING ?!
6. PARALLEL COMPUTING IS A TYPE OF COMPUTATION IN WHICH MANY CALCULATIONS OR THE EXECUTION OF PROCESS ARE CARRIED SIMULTANEOUSLY. Wikipedia. PARALLEL PROCESSING WITH DAEMONS IN PHP
7. PARALLEL VS DISTRIBUTED VS CONCURRENT
8. DAEMONS
9. A DAEMON IS A LONG-RUNNING BACKGROUND PROCESS THAT ANSWERS REQUESTS FOR SERVICES. Indiana University - Knowledge Base PARALLEL PROCESSING WITH DAEMONS IN PHP
10. BUT EVERY BACKGROUND PROCESS IS A DAEMON ?
11. PARALLEL PROCESSING WITH DAEMONS IN PHP A DAEMON SHOULD… ▸ Always acts as a background process ▸ Not allow direct user interaction ▸ Postfix the process name with the letter “d” (i.e. httpd, syslogd…) ▸ Respond to signals sent by other process or by the operating system ▸ Exit nicely ▸ Restart if needed
12. PHP
13. PARALLEL PROCESSING WITH DAEMONS IN PHP PHP OVERVIEW ▸ PHP: Hypertext Preprocessor ▸ Created in 1994 by Rasmus Lerdof ▸ Interpreted, imperative, procedural, object-oriented ▸ Dynamic type ▸ Synchronous by nature ▸ Single process and single threaded
14. THE PROBLEM
15. PARALLEL PROCESSING WITH DAEMONS IN PHP THE PROBLEM ▸ Every month Correios updates its CEP database ▸ Every month we need to update our database too ▸ There are more than 1M CEPs until now ▸ The provide this database in the CSV format ▸ The CSV file has 94.4MB of plain text
16. THE IMPLEMENTATION
18. FIRST ATTEMPT
19. PARALLEL PROCESSING WITH DAEMONS IN PHP CONVENTIONAL PHP
20. PARALLEL PROCESSING WITH DAEMONS IN PHP CONVENTIONAL PHP - PROS ▸ Easy to implement
21. PARALLEL PROCESSING WITH DAEMONS IN PHP CONVENTIONAL PHP - CONS ▸ Long process time ▸ Blocks the UI ▸ Blocks the I/O ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout can abort the process
22. IF WE MAKE IT ASYNCHRONOUS ?
23. PARALLEL PROCESSING WITH DAEMONS IN PHP USING ASYNCHRONOUS PROGRAMMING
24. PARALLEL PROCESSING WITH DAEMONS IN PHP USING ASYNC PROGRAMMING - PROS ▸ Async Implementation ▸ Don’t block the I/O
25. PARALLEL PROCESSING WITH DAEMONS IN PHP USING ASYNC PROGRAMMING - CONS ▸ Long process time ▸ Blocks the UI ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout
26. MAYBE WE COULD BREAK IT IN THREADS…
27. PARALLEL PROCESSING WITH DAEMONS IN PHP MULTI-THREADED PHP
28. PARALLEL PROCESSING WITH DAEMONS IN PHP MULTI-THREADED PHP - PROS ▸ Faster than procedural implementation ▸ Better usage of serve resources
29. PARALLEL PROCESSING WITH DAEMONS IN PHP MULTI-THREADED PHP - CONS ▸ Requires PHP 7.2+ w/ ZTS ▸ Extension dependent ▸ Blocks the UI ▸ Blocks the I/O ▸ If fails, needs user retry
30. SO THE PROBLEM IS NOT JUST THE IMPLEMENTATION…
31. BUT THE ARCHITECTURE
32. WHAT ABOUT AN ASYNC ARCHITECTURE ?
33. LETS TRY WITH A CRONTAB
34. PARALLEL PROCESSING WITH DAEMONS IN PHP USING CRONTAB
35. PARALLEL PROCESSING WITH DAEMONS IN PHP USING CRONTAB - PROS ▸ Don’t block the UI ▸ If fails, can retry in next batch
36. PARALLEL PROCESSING WITH DAEMONS IN PHP USING CRONTAB - CONS ▸ Time based ▸ Blocks the I/O ▸ Bad server resource usage ▸ Possible memory leaks
37. WE HAVE NOT TRIED WITH GEARMAN
38. PARALLEL PROCESSING WITH DAEMONS IN PHP USING GERMAN
39. PARALLEL PROCESSING WITH DAEMONS IN PHP USING GEARMAN - PROS ▸ Abstracts the architecture ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O
40. PARALLEL PROCESSING WITH DAEMONS IN PHP USING GEARMAN - CONS ▸ Not too easy to implement ▸ Needs to configure and deploy a Gearman server ▸ Extension dependent ▸ External libraries dependent (libgearman, libevent, uuid) ▸ Doesn’t support newer versions of PHP (only 5.3 to 5.6) ▸ Not currently maintained
41. LETS TRY WITH DAEMONS
42. PCNTL
43. PROCESS CONTROL SUPPORT IN PHP IMPLEMENTS THE UNIX STYLE OF PROCESS CREATION, PROGRAM EXECUTION, SIGNAL HANDLING AND PROCESS TERMINATION. PHP Official Documentation PARALLEL PROCESSING WITH DAEMONS IN PHP
44. PARALLEL PROCESSING WITH DAEMONS IN PHP USING PCNTL
45. PARALLEL PROCESSING WITH DAEMONS IN PHP USING PCNTL - PROS ▸ Easy to implement ▸ Architecture control ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O ▸ Native in PHP, needs to be compiled with, but still native
46. PARALLEL PROCESSING WITH DAEMONS IN PHP USING FCNTL - CONS ▸ Doesn’t work on Windows
47. WHO CARES?
48. IMPLEMENTING A DAEMON WITH 
 PCNTL
49. SIMPLE PROGRAM WITH PCNTL
50. PARALLEL PROCESSING WITH DAEMONS IN PHP SIMPLEM PROGRAM WITH PCNTL
51. ZOMBIES 
 VS 
 ORPHANS
52. PARALLEL PROCESSING WITH DAEMONS IN PHP ZOMBIES VS ORPHANS ▸ Zombies are dead ▸ Orphans are children whose parents has died
53. USING SIGNALS
54. SIGNALS ARE A LIMITED FORM OF IPC, TYPICALLY USED IN UNIX, UNIX-LIKE AND OTHER POSIX-COMPLIANT OPERATING SYSTEMS. A SIGNAL IS AN ASYNCHRONOUS NOTIFICATION SENT TO A PROCESS TO NOTIFY IT OF AN EVENT THAT OCCURRED. Wikipedia. PARALLEL PROCESSING WITH DAEMONS IN PHP
55. PARALLEL PROCESSING WITH DAEMONS IN PHP SOME SIGNALS AVAILABLE… ▸ SIGCHLD ▸ SIGHUP ▸ SIGINT ▸ SIGKILL ▸ SIGTERM ▸ So many others…
56. HANDLING SIGNALS
57. PARALLEL PROCESSING WITH DAEMONS IN PHP HANDLING SIGNALS
58. CLOSING DESCRIPTORS
59. PARALLEL PROCESSING WITH DAEMONS IN PHP CLOSING DESCRIPTORS
60. SETTING UP THE SESSION, USER AND GROUP
61. PARALLEL PROCESSING WITH DAEMONS IN PHP SETTTING UP THE SESSION, USER AND GROUP
62. LETS CODE!
63. THANK YOU! Lucas Mendes
 Software Architect at Tienda Nube
 about.me/devsdmf We're hiring, join the crew! 
 bit.ly/work-at-tiendanube