Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling PHP with Daemons and Long-Running Processes

Lucas Mendes
December 08, 2018

Scaling PHP with Daemons and Long-Running Processes

Scale a web application is not an easy task, even harder when we need to deal with lots of data processing without language features like async programming and multi-threading. In this talk, I will cover some possibilites when we need to split and improve batch processing using PHP.

Presented at PHP Conference 2018.

Lucas Mendes

December 08, 2018
Tweet

More Decks by Lucas Mendes

Other Decks in Programming

Transcript

  1. A DAEMON IS A LONG-RUNNING BACKGROUND PROCESS THAT ANSWERS REQUESTS

    FOR SERVICES. Indiana University - Knowledge Base SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES
  2. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES A DAEMON SHOULD…

    ▸ Always acts as a background process ▸ Not allow direct user interaction ▸ Postfix the process name with the letter “d” (i.e. httpd, syslogd…) ▸ Respond to signals sent by other process or by the operating system ▸ Exit nicely ▸ Restart if needed
  3. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES THE PROBLEM ▸

    Every month Correios updates its CEP database ▸ Every month we need to update our database too ▸ There are more than 1M CEPs until now ▸ The provide this database in the CSV format ▸ The CSV file has 94.4MB of plain text
  4. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES CONVENTIONAL PHP -

    CONS ▸ Long process time ▸ Blocks the UI ▸ Blocks the I/O ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout can abort the process
  5. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNC PROGRAMMING

    - PROS ▸ Async Implementation ▸ Don’t block the I/O
  6. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING ASYNC PROGRAMMING

    - CONS ▸ Long process time ▸ Blocks the UI ▸ Possible memory leaks ▸ If fails, needs user retry ▸ Browser timeout
  7. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP -

    PROS ▸ Faster than procedural implementation ▸ Better usage of serve resources
  8. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES MULTI-THREADED PHP -

    CONS ▸ Requires PHP 7.2+ w/ ZTS ▸ Extension dependent ▸ Blocks the UI ▸ Blocks the I/O ▸ If fails, needs user retry
  9. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB -

    PROS ▸ Don’t block the UI ▸ If fails, can retry in next batch
  10. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING CRONTAB -

    CONS ▸ Time based ▸ Blocks the I/O ▸ Bad server resource usage ▸ Possible memory leaks
  11. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GEARMAN -

    PROS ▸ Abstracts the architecture ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O
  12. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING GEARMAN -

    CONS ▸ Not too easy to implement ▸ Needs to configure and deploy a Gearman server ▸ Extension dependent ▸ External libraries dependent (libgearman, libevent, uuid
  13. PROCESS CONTROL SUPPORT IN PHP IMPLEMENTS THE UNIX STYLE OF

    PROCESS CREATION, PROGRAM EXECUTION, SIGNAL HANDLING AND PROCESS TERMINATION. PHP Official Documentation SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES
  14. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES USING PCNTL -

    PROS ▸ Easy to implement ▸ Architecture control ▸ Faster than single-process implementation ▸ Don’t block the UI ▸ Don’t block the I/O ▸ Native in PHP, needs to be compiled with, but still native
  15. PARALLEL PROCESSING WITH DAEMONS IN PHP ZOMBIES VS ORPHANS ▸

    Zombies are dead ▸ Orphans are children whose parents has died
  16. SIGNALS ARE A LIMITED FORM OF IPC, TYPICALLY USED IN

    UNIX, UNIX-LIKE AND OTHER POSIX-COMPLIANT OPERATING SYSTEMS. A SIGNAL IS AN ASYNCHRONOUS NOTIFICATION SENT TO A PROCESS TO NOTIFY IT OF AN EVENT THAT OCCURRED. Wikipedia. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES
  17. SCALING PHP WITH DAEMONS AND LONG-RUNNING PROCESSES SOME SIGNALS AVAILABLE…

    ▸ SIGCHLD ▸ SIGHUP ▸ SIGINT ▸ SIGKILL ▸ SIGTERM ▸ So many others…
  18. THANK YOU! Lucas Mendes
 Product Owner at Tienda Nube
 Find

    me: @devsdmf We're hiring, join the crew! 
 bit.ly/work-at-tiendanube