Scaling PHP with Daemons and Long-Running Processes

7a736764e6d6c830c049bdc2be34ec19?s=47 Lucas Mendes
December 08, 2018

Scaling PHP with Daemons and Long-Running Processes

Scale a web application is not an easy task, even harder when we need to deal with lots of data processing without language features like async programming and multi-threading. In this talk, I will cover some possibilites when we need to split and improve batch processing using PHP.

Presented at PHP Conference 2018.

7a736764e6d6c830c049bdc2be34ec19?s=128

Lucas Mendes

December 08, 2018
Tweet

Transcript

  1. Lucas Mendes | Product Owner | @devsdmf SCALING PHP WITH

    DAEMONS AND LONG-RUNNING PROCESSES
  2. $ whoami

  3. DAEMONS

  4. A DAEMON IS A LONG-RUNNING BACKGROUND PROCESS THAT ANSWERS REQUESTS

    FOR SERVICES. Indiana University - Knowledge Base SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES
  5. BUT EVERY BACKGROUND PROCESS IS A DAEMON ?

  6. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES A DAEMON SHOULD…

    ▸ Always acts as a background process ▸ Not allow direct user interaction ▸ Postfix the process name with the letter “d” (i.e. httpd, syslogd…) ▸ Respond to signals sent by other process or by the operating system ▸ Exit nicely ▸ Restart if needed
  7. THE PROBLEM

  8. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES THE PROBLEM ▸

    Every month Correios updates its CEP database ▸ Every month we need to update our database too ▸ There are more than 1M CEPs until now ▸ The provide this database in the CSV format ▸ The CSV file has 94.4MB of plain text
  9. THE IMPLEMENTATION

  10. FIRST ATTEMPT

  11. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CONVENTIONAL PHP

  12. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CONVENTIONAL PHP -

    PROS ▸ Easy to implement
  13. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CONVENTIONAL PHP -

    CONS ▸ Long process time ▸ Blocks the UI ▸ Blocks the I/O ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout can abort the process
  14. IF WE MAKE IT ASYNCHRONOUS ?

  15. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNCHRONOUS PROGRAMMING

  16. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNC PROGRAMMING

    - PROS ▸ Async Implementation ▸ Don’t block the I/O
  17. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNC PROGRAMMING

    - CONS ▸ Long process time ▸ Blocks the UI ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout
  18. MAYBE WE COULD BREAK IT IN THREADS…

  19. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP

  20. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP -

    PROS ▸ Faster than procedural implementation ▸ Better usage of serve resources
  21. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP -

    CONS ▸ Requires PHP 7.2+ w/ ZTS ▸ Extension dependent ▸ Blocks the UI ▸ Blocks the I/O ▸ If fails, needs user retry
  22. SO THE PROBLEM IS NOT JUST THE IMPLEMENTATION…

  23. BUT THE ARCHITECTURE

  24. WHAT ABOUT AN ASYNC ARCHITECTURE ?

  25. LETS TRY WITH A CRONTAB

  26. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB

  27. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB -

    PROS ▸ Don’t block the UI ▸ If fails, can retry in next batch
  28. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB -

    CONS ▸ Time based ▸ Blocks the I/O ▸ Bad server resource usage ▸ Possible memory leaks
  29. WE HAVE NOT TRIED WITH GEARMAN

  30. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GERMAN

  31. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GEARMAN -

    PROS ▸ Abstracts the architecture ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O
  32. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GEARMAN -

    CONS ▸ Not too easy to implement ▸ Needs to configure and deploy a Gearman server ▸ Extension dependent ▸ External libraries dependent (libgearman, libevent, uuid
  33. LETS TRY WITH DAEMONS

  34. PCNTL

  35. PROCESS CONTROL SUPPORT IN PHP IMPLEMENTS THE UNIX STYLE OF

    PROCESS CREATION, PROGRAM EXECUTION, SIGNAL HANDLING AND PROCESS TERMINATION. PHP Official Documentation SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES
  36. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING PCNTL

  37. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING PCNTL -

    PROS ▸ Easy to implement ▸ Architecture control ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O ▸ Native in PHP, needs to be compiled with, but still native
  38. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING FCNTL -

    CONS ▸ Doesn’t work on Windows
  39. WHO CARES?

  40. IMPLEMENTING A DAEMON WITH 
 PCNTL

  41. SIMPLE PROGRAM WITH PCNTL

  42. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES SIMPLEM PROGRAM WITH

    PCNTL
  43. ZOMBIES 
 VS 
 ORPHANS

  44. PARALLEL PROCESSING WITH DAEMONS IN PHP ZOMBIES VS ORPHANS ▸

    Zombies are dead ▸ Orphans are children whose parents has died
  45. USING SIGNALS

  46. SIGNALS ARE A LIMITED FORM OF IPC, TYPICALLY USED IN

    UNIX, UNIX-LIKE AND OTHER POSIX-COMPLIANT OPERATING SYSTEMS. A SIGNAL IS AN ASYNCHRONOUS NOTIFICATION SENT TO A PROCESS TO NOTIFY IT OF AN EVENT THAT OCCURRED. Wikipedia. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES
  47. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES SOME SIGNALS AVAILABLE…

    ▸ SIGCHLD ▸ SIGHUP ▸ SIGINT ▸ SIGKILL ▸ SIGTERM ▸ So many others…
  48. HANDLING SIGNALS

  49. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES HANDLING SIGNALS

  50. CLOSING DESCRIPTORS

  51. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CLOSING DESCRIPTORS

  52. SETTING UP THE SESSION, USER AND GROUP

  53. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES SETTTING UP THE

    SESSION, USER AND GROUP
  54. TIME TO DEMO https://github.com/devsdmf/csv-processor

  55. THANK YOU! Lucas Mendes
 Product Owner at Tienda Nube
 Find

    me: @devsdmf We're hiring, join the crew! 
 bit.ly/work-at-tiendanube