Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Fork It ! Parallel Processing in PHP

Fork It ! Parallel Processing in PHP

PHPNW 2012

042ad61f65b2d5518f671efd762873bb?s=128

Nathaniel McHugh

October 07, 2012
Tweet

Transcript

  1. Parallel Processing in PHP Fork It ! Nathaniel McHugh Monday,

    8 October 12
  2. • Software Engineer Inviqa • @natmchugh • http://fishtrap.co.uk • likes

    doing useless PHP Monday, 8 October 12
  3. Monday, 8 October 12

  4. Pasta or Monday, 8 October 12

  5. 0 1 2 3 4 5 6 7 8 9

    10 1 cup 2 cups 3 cups 4 cups t (minutes) kettle hob microwave Monday, 8 October 12
  6. Monday, 8 October 12

  7. Number of Processors https://github.com/natmchugh/PHP-Num- Processors php > echo num_processors_available(); 2

    php > echo num_processors_configured(); 2 Monday, 8 October 12
  8. Forking • php has no multi threading • pcntl extension

    • *nix only • cli only really Monday, 8 October 12
  9. fork() Monday, 8 October 12

  10. fork() $ ps -a | grep ps 3394 ttys003 0:00.00

    ps -a 3395 ttys003 0:00.00 grep ps $ Monday, 8 October 12
  11. fork() $ ps -a | grep ps 3394 ttys003 0:00.00

    ps -a 3395 ttys003 0:00.00 grep ps $ $ man fork | grep returns Upon successful completion, fork() returns a value of 0 to the child process and returns the process ID of the child process to the parent Monday, 8 October 12
  12. Monday, 8 October 12

  13. Concurrency real 1m10.322s real 0m39.928s real 0m45.827s Monday, 8 October

    12
  14. Die Hard PHP Daemons <?php while (true) { $worker->doSomeWork(); }

    Monday, 8 October 12
  15. From the Beginning Monday, 8 October 12

  16. From the Beginning $ php start_with_manual.php $ Monday, 8 October

    12
  17. <?php $pid = pcntl_fork(); if ($pid == -1) { die('could

    not fork'); } else if ($pid) { // we are the parent sleep(5); pcntl_wait($status); //Protect against Zombie children } else { // we are the child sleep(5); } Monday, 8 October 12
  18. <?php $pid = pcntl_fork(); if ($pid == -1) { die('could

    not fork'); } else if ($pid) { // we are the parent sleep(5); pcntl_wait($status); //Protect against Zombie children } else { // we are the child sleep(5); } $ time php example_with_delay.php real 0m5.072s user 0m0.023s sys 0m0.021s Monday, 8 October 12
  19. • Zombies are dead • Orphans are children whose parent

    has died Zombies vs Orphans Monday, 8 October 12
  20. multiple children? <?php for ($i=0; $i < 5; $i++) {

    $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } Monday, 8 October 12
  21. multiple children? <?php for ($i=0; $i < 5; $i++) {

    $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12
  22. multiple children? <?php for ($i=0; $i < 5; $i++) {

    $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12
  23. multiple children? <?php for ($i=0; $i < 5; $i++) {

    $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12
  24. The Pattern require('deep_thought.php'); $deepThought = new DeepThought(); $children = array();

    for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if (0 === $pid) { $lifeUniverseEverthing = $deepThought->findTheAnswer(); echo "Child $i says:",$lifeUniverseEverthing,PHP_EOL; die(); } } do { $pid = pcntl_wait($status); $children = array_diff($children, array($pid)); } while (count($children) > 0); Monday, 8 October 12
  25. The Pattern require('deep_thought.php'); $deepThought = new DeepThought(); $children = array();

    for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if (0 === $pid) { $lifeUniverseEverthing = $deepThought->findTheAnswer(); echo "Child $i says:",$lifeUniverseEverthing,PHP_EOL; die(); } } do { $pid = pcntl_wait($status); $children = array_diff($children, array($pid)); } while (count($children) > 0); $ time php the_pattern.php Child 0 says:42 Child 3 says:42 Child 1 says:Child 2 says:42 42 Child 4 says:42 real 0m5.453s user 0m0.056s sys 0m0.060s $ Monday, 8 October 12
  26. Signals <?php // declare where callbacks will occur in your

    script declare(ticks = 1); function sig_handler($signo) { die("caught a siginal $signo".PHP_EOL); } $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent pcntl_wait($status); die('Parent finished'.PHP_EOL); } else { pcntl_signal(SIGINT, "sig_handler"); // we are the child while (true) { sleep(5); echo 'More',PHP_EOL; } die('Child finished'.PHP_EOL); } Monday, 8 October 12
  27. Signals <?php // declare where callbacks will occur in your

    script declare(ticks = 1); function sig_handler($signo) { die("caught a siginal $signo".PHP_EOL); } $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent pcntl_wait($status); die('Parent finished'.PHP_EOL); } else { pcntl_signal(SIGINT, "sig_handler"); // we are the child while (true) { sleep(5); echo 'More',PHP_EOL; } die('Child finished'.PHP_EOL); } $ php signals.php More More More More More More ^Ccaught a siginal 2 Monday, 8 October 12
  28. SIGCHLD <?php declare(ticks = 1); pcntl_signal(SIGCHLD, 'sig_child'); $children = array();

    for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if (0 === $pid) { die(); } } while ($children) { sleep(1); echo '.'; } function sig_child() { global $children; echo("Caught SIGCHLD".PHP_EOL); while (($pid = pcntl_wait($status, WNOHANG)) > 0) { $children = array_diff($children, array($pid)); echo "Child $pid collected".PHP_EOL; } } Monday, 8 October 12
  29. SIGCHLD <?php declare(ticks = 1); pcntl_signal(SIGCHLD, 'sig_child'); $children = array();

    for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if (0 === $pid) { die(); } } while ($children) { sleep(1); echo '.'; } function sig_child() { global $children; echo("Caught SIGCHLD".PHP_EOL); while (($pid = pcntl_wait($status, WNOHANG)) > 0) { $children = array_diff($children, array($pid)); echo "Child $pid collected".PHP_EOL; } } php sigchld.php ....Child 0 says:42 Child 2 says:42 Child 3 says:42 Child 1 says:42 Child 4 says:42 .Caught SIGCHLD Child 22877 collected Child 22876 collected Child 22874 collected .Caught SIGCHLD Child 22875 collected .Caught SIGCHLD Child 22878 collected Monday, 8 October 12
  30. Process Isolation <?php $a = 'foo'; $b = 'bar'; $pid

    = pcntl_fork(); if ($pid) { $a = 'baz'; pcntl_wait($status); } else { $b = 'bat'; die(); } var_dump($a, $b); Monday, 8 October 12
  31. Process Isolation <?php $a = 'foo'; $b = 'bar'; $pid

    = pcntl_fork(); if ($pid) { $a = 'baz'; pcntl_wait($status); } else { $b = 'bat'; die(); } var_dump($a, $b); $ php process_isolation.php string(3) "baz" string(3) "bar" $ Monday, 8 October 12
  32. Resources <?php $fh = fopen('/tmp/file', 'w+'); $pid = pcntl_fork(); if

    ($pid) { fputs($fh, 'Parent'.PHP_EOL); pcntl_wait($status); } else { fputs($fh, 'Child'.PHP_EOL); die(); } fclose($fh); readfile('/tmp/file'); Monday, 8 October 12
  33. Resources $ php resources.php Parent Child $ php resources.php Child

    Parent $ <?php $fh = fopen('/tmp/file', 'w+'); $pid = pcntl_fork(); if ($pid) { fputs($fh, 'Parent'.PHP_EOL); pcntl_wait($status); } else { fputs($fh, 'Child'.PHP_EOL); die(); } fclose($fh); readfile('/tmp/file'); Monday, 8 October 12
  34. How do you get your children to talk to you?

    Monday, 8 October 12
  35. Listening to your children • DB • files • memcached,

    apc or shared memory • sockets Monday, 8 October 12
  36. Sockets Example <?php require('deep_thought.php'); $deepThought = new DeepThought(); $forks =

    array(); $sockets = array(); for ($i=0; $i < 5; $i++) { socket_create_pair(AF_UNIX, SOCK_STREAM, 0, $sockets); $forks[] = $pid = pcntl_fork(); if ($pid) { $socketPairs[$pid] = $sockets; } else { $data = $deepThought->findTheAnswer(); writeToSocket($data, $sockets[0]); die(); } } do { $pid = pcntl_wait($status); $forks = array_diff($forks, array($pid)); $sockets = $socketPairs[$pid]; $data = readFromSocket($sockets[1]); var_dump($data); } while (count($forks) > 0); Monday, 8 October 12
  37. Next Level • Gearman • Message Queues ØMQ, RabbitMQ ..

    Monday, 8 October 12
  38. Thinking Parallel “It turns out that what was difficult, and

    almost impossible, is to take an ordinary program and automatically figure out how to use the parallel computation effectively...” Richard P. Feynman Monday, 8 October 12
  39. Input Processing Output Input Input Input Processing Processing Output Output

    Output Monday, 8 October 12
  40. • Even Distribution of work /input • Size of messages

    • Data Dependancies Monday, 8 October 12
  41. PHPLOC input: file system output: array of stats processing: tokenize

    and count occurrences Monday, 8 October 12
  42. PHPLOC array(1781) { [0] => string(51) "zf2/bin/autoload_example.php" [1] => string(52)

    "zf2/bin/ autoload_examples.php" [2] => string(53) "zf2/bin/ classmap_generator.php" [3] => string(60) "zf2/bin/ createAutoloadTestClasses.php" [4] => string(54) "zf2/bin/ pluginmap_generator.php" ... foreach ($files as $key => $file) { $directory = dirname($file); if (!isset($this->directories[$directory])) { $this->directories[$directory] = TRUE; } ... Monday, 8 October 12
  43. PHPLOC real 0m7.216s user 0m6.492s sys 0m0.240s $ time phploc

    --cores 1 zf2 $ time phploc --cores 2 zf2 real 0m5.500s user 0m6.686s sys 0m0.293s https://github.com/natmchugh/phploc Monday, 8 October 12
  44. Amdahls Law Monday, 8 October 12

  45. Embarrassingly Parallel Monday, 8 October 12

  46. Mandelbrot Set Input: Pixel co-ordinates Processing: Hard core number crunching

    Output: An image Monday, 8 October 12
  47. Dividing Work https://github.com/natmchugh/PHP-Fractals Monday, 8 October 12

  48. Thanks Joind.in: http://joind.in/7002 @natmchugh http://fishtrap.co.uk Monday, 8 October 12

  49. • Front Fork http://www.flickr.com/photos/de_atienza/ • Paint, Laptop & fork Lift

    http://consumeconsume.com/ • Haynes Manual http://www.flickr.com/photos/8490341@N04/ • Zombie Child http://www.flickr.com/photos/zenobia_joy/ • Signal http://www.flickr.com/photos/rbrwr/ • Pattern http://www.flickr.com/photos/eadaoinflynn Monday, 8 October 12