Fork It ! Parallel Processing in PHP

PHPNW 2012

Transcript

1. Parallel Processing in PHP Fork It ! Nathaniel McHugh Monday, 8 October 12
2. • Software Engineer Inviqa • @natmchugh • http://fishtrap.co.uk • likes doing useless PHP Monday, 8 October 12
3. Monday, 8 October 12
4. Pasta or Monday, 8 October 12
5. 0 1 2 3 4 5 6 7 8 9 10 1 cup 2 cups 3 cups 4 cups t (minutes) kettle hob microwave Monday, 8 October 12
6. Monday, 8 October 12
7. Number of Processors https://github.com/natmchugh/PHP-Num- Processors php > echo num_processors_available(); 2 php > echo num_processors_configured(); 2 Monday, 8 October 12
8. Forking • php has no multi threading • pcntl extension • *nix only • cli only really Monday, 8 October 12
9. fork() Monday, 8 October 12
10. fork() $ ps -a | grep ps 3394 ttys003 0:00.00 ps -a 3395 ttys003 0:00.00 grep ps $ Monday, 8 October 12
11. fork() $ ps -a | grep ps 3394 ttys003 0:00.00 ps -a 3395 ttys003 0:00.00 grep ps $ $ man fork | grep returns Upon successful completion, fork() returns a value of 0 to the child process and returns the process ID of the child process to the parent Monday, 8 October 12
12. Monday, 8 October 12
13. Concurrency real 1m10.322s real 0m39.928s real 0m45.827s Monday, 8 October 12
14. Die Hard PHP Daemons <?php while (true) { $worker->doSomeWork(); } Monday, 8 October 12
15. From the Beginning Monday, 8 October 12
16. From the Beginning $ php start_with_manual.php $ Monday, 8 October 12
17. <?php $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent sleep(5); pcntl_wait($status); //Protect against Zombie children } else { // we are the child sleep(5); } Monday, 8 October 12
18. <?php $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent sleep(5); pcntl_wait($status); //Protect against Zombie children } else { // we are the child sleep(5); } $ time php example_with_delay.php real 0m5.072s user 0m0.023s sys 0m0.021s Monday, 8 October 12
19. • Zombies are dead • Orphans are children whose parent has died Zombies vs Orphans Monday, 8 October 12
20. multiple children? <?php for ($i=0; $i < 5; $i++) { $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } Monday, 8 October 12
21. multiple children? <?php for ($i=0; $i < 5; $i++) { $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12
22. multiple children? <?php for ($i=0; $i < 5; $i++) { $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12
23. multiple children? <?php for ($i=0; $i < 5; $i++) { $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12
24. The Pattern require('deep_thought.php'); $deepThought = new DeepThought(); $children = array(); for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if (0 === $pid) { $lifeUniverseEverthing = $deepThought->findTheAnswer(); echo "Child $i says:",$lifeUniverseEverthing,PHP_EOL; die(); } } do { $pid = pcntl_wait($status); $children = array_diff($children, array($pid)); } while (count($children) > 0); Monday, 8 October 12
25. The Pattern require('deep_thought.php'); $deepThought = new DeepThought(); $children = array(); for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if (0 === $pid) { $lifeUniverseEverthing = $deepThought->findTheAnswer(); echo "Child $i says:",$lifeUniverseEverthing,PHP_EOL; die(); } } do { $pid = pcntl_wait($status); $children = array_diff($children, array($pid)); } while (count($children) > 0); $ time php the_pattern.php Child 0 says:42 Child 3 says:42 Child 1 says:Child 2 says:42 42 Child 4 says:42 real 0m5.453s user 0m0.056s sys 0m0.060s $ Monday, 8 October 12
26. Signals <?php // declare where callbacks will occur in your script declare(ticks = 1); function sig_handler($signo) { die("caught a siginal $signo".PHP_EOL); } $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent pcntl_wait($status); die('Parent finished'.PHP_EOL); } else { pcntl_signal(SIGINT, "sig_handler"); // we are the child while (true) { sleep(5); echo 'More',PHP_EOL; } die('Child finished'.PHP_EOL); } Monday, 8 October 12
27. Signals <?php // declare where callbacks will occur in your script declare(ticks = 1); function sig_handler($signo) { die("caught a siginal $signo".PHP_EOL); } $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent pcntl_wait($status); die('Parent finished'.PHP_EOL); } else { pcntl_signal(SIGINT, "sig_handler"); // we are the child while (true) { sleep(5); echo 'More',PHP_EOL; } die('Child finished'.PHP_EOL); } $ php signals.php More More More More More More ^Ccaught a siginal 2 Monday, 8 October 12
28. SIGCHLD <?php declare(ticks = 1); pcntl_signal(SIGCHLD, 'sig_child'); $children = array(); for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if (0 === $pid) { die(); } } while ($children) { sleep(1); echo '.'; } function sig_child() { global $children; echo("Caught SIGCHLD".PHP_EOL); while (($pid = pcntl_wait($status, WNOHANG)) > 0) { $children = array_diff($children, array($pid)); echo "Child $pid collected".PHP_EOL; } } Monday, 8 October 12
29. SIGCHLD <?php declare(ticks = 1); pcntl_signal(SIGCHLD, 'sig_child'); $children = array(); for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if (0 === $pid) { die(); } } while ($children) { sleep(1); echo '.'; } function sig_child() { global $children; echo("Caught SIGCHLD".PHP_EOL); while (($pid = pcntl_wait($status, WNOHANG)) > 0) { $children = array_diff($children, array($pid)); echo "Child $pid collected".PHP_EOL; } } php sigchld.php ....Child 0 says:42 Child 2 says:42 Child 3 says:42 Child 1 says:42 Child 4 says:42 .Caught SIGCHLD Child 22877 collected Child 22876 collected Child 22874 collected .Caught SIGCHLD Child 22875 collected .Caught SIGCHLD Child 22878 collected Monday, 8 October 12
30. Process Isolation <?php $a = 'foo'; $b = 'bar'; $pid = pcntl_fork(); if ($pid) { $a = 'baz'; pcntl_wait($status); } else { $b = 'bat'; die(); } var_dump($a, $b); Monday, 8 October 12
31. Process Isolation <?php $a = 'foo'; $b = 'bar'; $pid = pcntl_fork(); if ($pid) { $a = 'baz'; pcntl_wait($status); } else { $b = 'bat'; die(); } var_dump($a, $b); $ php process_isolation.php string(3) "baz" string(3) "bar" $ Monday, 8 October 12
32. Resources <?php $fh = fopen('/tmp/file', 'w+'); $pid = pcntl_fork(); if ($pid) { fputs($fh, 'Parent'.PHP_EOL); pcntl_wait($status); } else { fputs($fh, 'Child'.PHP_EOL); die(); } fclose($fh); readfile('/tmp/file'); Monday, 8 October 12
33. Resources $ php resources.php Parent Child $ php resources.php Child Parent $ <?php $fh = fopen('/tmp/file', 'w+'); $pid = pcntl_fork(); if ($pid) { fputs($fh, 'Parent'.PHP_EOL); pcntl_wait($status); } else { fputs($fh, 'Child'.PHP_EOL); die(); } fclose($fh); readfile('/tmp/file'); Monday, 8 October 12
34. How do you get your children to talk to you? Monday, 8 October 12
35. Listening to your children • DB • files • memcached, apc or shared memory • sockets Monday, 8 October 12
36. Sockets Example <?php require('deep_thought.php'); $deepThought = new DeepThought(); $forks = array(); $sockets = array(); for ($i=0; $i < 5; $i++) { socket_create_pair(AF_UNIX, SOCK_STREAM, 0, $sockets); $forks[] = $pid = pcntl_fork(); if ($pid) { $socketPairs[$pid] = $sockets; } else { $data = $deepThought->findTheAnswer(); writeToSocket($data, $sockets[0]); die(); } } do { $pid = pcntl_wait($status); $forks = array_diff($forks, array($pid)); $sockets = $socketPairs[$pid]; $data = readFromSocket($sockets[1]); var_dump($data); } while (count($forks) > 0); Monday, 8 October 12
37. Next Level • Gearman • Message Queues ØMQ, RabbitMQ .. Monday, 8 October 12
38. Thinking Parallel “It turns out that what was difficult, and almost impossible, is to take an ordinary program and automatically figure out how to use the parallel computation effectively...” Richard P. Feynman Monday, 8 October 12
39. Input Processing Output Input Input Input Processing Processing Output Output Output Monday, 8 October 12
40. • Even Distribution of work /input • Size of messages • Data Dependancies Monday, 8 October 12
41. PHPLOC input: file system output: array of stats processing: tokenize and count occurrences Monday, 8 October 12
42. PHPLOC array(1781) { [0] => string(51) "zf2/bin/autoload_example.php" [1] => string(52) "zf2/bin/ autoload_examples.php" [2] => string(53) "zf2/bin/ classmap_generator.php" [3] => string(60) "zf2/bin/ createAutoloadTestClasses.php" [4] => string(54) "zf2/bin/ pluginmap_generator.php" ... foreach ($files as $key => $file) { $directory = dirname($file); if (!isset($this->directories[$directory])) { $this->directories[$directory] = TRUE; } ... Monday, 8 October 12
43. PHPLOC real 0m7.216s user 0m6.492s sys 0m0.240s $ time phploc --cores 1 zf2 $ time phploc --cores 2 zf2 real 0m5.500s user 0m6.686s sys 0m0.293s https://github.com/natmchugh/phploc Monday, 8 October 12
44. Amdahls Law Monday, 8 October 12
45. Embarrassingly Parallel Monday, 8 October 12
46. Mandelbrot Set Input: Pixel co-ordinates Processing: Hard core number crunching Output: An image Monday, 8 October 12
47. Dividing Work https://github.com/natmchugh/PHP-Fractals Monday, 8 October 12
48. Thanks Joind.in: http://joind.in/7002 @natmchugh http://fishtrap.co.uk Monday, 8 October 12
49. • Front Fork http://www.flickr.com/photos/de_atienza/ • Paint, Laptop & fork Lift http://consumeconsume.com/ • Haynes Manual http://www.flickr.com/photos/8490341@N04/ • Zombie Child http://www.flickr.com/photos/zenobia_joy/ • Signal http://www.flickr.com/photos/rbrwr/ • Pattern http://www.flickr.com/photos/eadaoinflynn Monday, 8 October 12