Fork It ! Parallel Processing in PHP

Transcript

1. Parallel Processing in PHP Fork It ! Nathaniel McHugh Monday,

   8 October 12

2. • Software Engineer Inviqa • @natmchugh • http://fishtrap.co.uk • likes

   doing useless PHP Monday, 8 October 12

3. Monday, 8 October 12

4. Pasta or Monday, 8 October 12

5. 0 1 2 3 4 5 6 7 8 9

   10 1 cup 2 cups 3 cups 4 cups t (minutes) kettle hob microwave Monday, 8 October 12

6. Monday, 8 October 12

7. Number of Processors https://github.com/natmchugh/PHP-Num- Processors php > echo num_processors_available(); 2

   php > echo num_processors_configured(); 2 Monday, 8 October 12

8. Forking • php has no multi threading • pcntl extension

   • *nix only • cli only really Monday, 8 October 12

9. fork() Monday, 8 October 12

10. fork() $ ps -a | grep ps 3394 ttys003 0:00.00

    ps -a 3395 ttys003 0:00.00 grep ps $ Monday, 8 October 12

11. fork() $ ps -a | grep ps 3394 ttys003 0:00.00

    ps -a 3395 ttys003 0:00.00 grep ps $ $ man fork | grep returns Upon successful completion, fork() returns a value of 0 to the child process and returns the process ID of the child process to the parent Monday, 8 October 12

12. Monday, 8 October 12

13. Concurrency real 1m10.322s real 0m39.928s real 0m45.827s Monday, 8 October

    12

14. Die Hard PHP Daemons <?php while (true) { $worker->doSomeWork(); }

    Monday, 8 October 12

15. From the Beginning Monday, 8 October 12

16. From the Beginning $ php start_with_manual.php $ Monday, 8 October

    12

17. <?php $pid = pcntl_fork(); if ($pid == -1) { die('could

    not fork'); } else if ($pid) { // we are the parent sleep(5); pcntl_wait($status); //Protect against Zombie children } else { // we are the child sleep(5); } Monday, 8 October 12

18. <?php $pid = pcntl_fork(); if ($pid == -1) { die('could

    not fork'); } else if ($pid) { // we are the parent sleep(5); pcntl_wait($status); //Protect against Zombie children } else { // we are the child sleep(5); } $ time php example_with_delay.php real 0m5.072s user 0m0.023s sys 0m0.021s Monday, 8 October 12

19. • Zombies are dead • Orphans are children whose parent

    has died Zombies vs Orphans Monday, 8 October 12

20. multiple children? <?php for ($i=0; $i < 5; $i++) {

    $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } Monday, 8 October 12

21. multiple children? <?php for ($i=0; $i < 5; $i++) {

    $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12

22. multiple children? <?php for ($i=0; $i < 5; $i++) {

    $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12

23. multiple children? <?php for ($i=0; $i < 5; $i++) {

    $pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12

24. The Pattern require('deep_thought.php'); $deepThought = new DeepThought(); $children = array();

    for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if (0 === $pid) { $lifeUniverseEverthing = $deepThought->findTheAnswer(); echo "Child $i says:",$lifeUniverseEverthing,PHP_EOL; die(); } } do { $pid = pcntl_wait($status); $children = array_diff($children, array($pid)); } while (count($children) > 0); Monday, 8 October 12

25. The Pattern require('deep_thought.php'); $deepThought = new DeepThought(); $children = array();

    for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if (0 === $pid) { $lifeUniverseEverthing = $deepThought->findTheAnswer(); echo "Child $i says:",$lifeUniverseEverthing,PHP_EOL; die(); } } do { $pid = pcntl_wait($status); $children = array_diff($children, array($pid)); } while (count($children) > 0); $ time php the_pattern.php Child 0 says:42 Child 3 says:42 Child 1 says:Child 2 says:42 42 Child 4 says:42 real 0m5.453s user 0m0.056s sys 0m0.060s $ Monday, 8 October 12

26. Signals <?php // declare where callbacks will occur in your

    script declare(ticks = 1); function sig_handler($signo) { die("caught a siginal $signo".PHP_EOL); } $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent pcntl_wait($status); die('Parent finished'.PHP_EOL); } else { pcntl_signal(SIGINT, "sig_handler"); // we are the child while (true) { sleep(5); echo 'More',PHP_EOL; } die('Child finished'.PHP_EOL); } Monday, 8 October 12

27. Signals <?php // declare where callbacks will occur in your

    script declare(ticks = 1); function sig_handler($signo) { die("caught a siginal $signo".PHP_EOL); } $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent pcntl_wait($status); die('Parent finished'.PHP_EOL); } else { pcntl_signal(SIGINT, "sig_handler"); // we are the child while (true) { sleep(5); echo 'More',PHP_EOL; } die('Child finished'.PHP_EOL); } $ php signals.php More More More More More More ^Ccaught a siginal 2 Monday, 8 October 12

28. SIGCHLD <?php declare(ticks = 1); pcntl_signal(SIGCHLD, 'sig_child'); $children = array();

    for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if (0 === $pid) { die(); } } while ($children) { sleep(1); echo '.'; } function sig_child() { global $children; echo("Caught SIGCHLD".PHP_EOL); while (($pid = pcntl_wait($status, WNOHANG)) > 0) { $children = array_diff($children, array($pid)); echo "Child $pid collected".PHP_EOL; } } Monday, 8 October 12

29. SIGCHLD <?php declare(ticks = 1); pcntl_signal(SIGCHLD, 'sig_child'); $children = array();

    for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if (0 === $pid) { die(); } } while ($children) { sleep(1); echo '.'; } function sig_child() { global $children; echo("Caught SIGCHLD".PHP_EOL); while (($pid = pcntl_wait($status, WNOHANG)) > 0) { $children = array_diff($children, array($pid)); echo "Child $pid collected".PHP_EOL; } } php sigchld.php ....Child 0 says:42 Child 2 says:42 Child 3 says:42 Child 1 says:42 Child 4 says:42 .Caught SIGCHLD Child 22877 collected Child 22876 collected Child 22874 collected .Caught SIGCHLD Child 22875 collected .Caught SIGCHLD Child 22878 collected Monday, 8 October 12

30. Process Isolation <?php $a = 'foo'; $b = 'bar'; $pid

    = pcntl_fork(); if ($pid) { $a = 'baz'; pcntl_wait($status); } else { $b = 'bat'; die(); } var_dump($a, $b); Monday, 8 October 12

31. Process Isolation <?php $a = 'foo'; $b = 'bar'; $pid

    = pcntl_fork(); if ($pid) { $a = 'baz'; pcntl_wait($status); } else { $b = 'bat'; die(); } var_dump($a, $b); $ php process_isolation.php string(3) "baz" string(3) "bar" $ Monday, 8 October 12

32. Resources <?php $fh = fopen('/tmp/file', 'w+'); $pid = pcntl_fork(); if

    ($pid) { fputs($fh, 'Parent'.PHP_EOL); pcntl_wait($status); } else { fputs($fh, 'Child'.PHP_EOL); die(); } fclose($fh); readfile('/tmp/file'); Monday, 8 October 12

33. Resources $ php resources.php Parent Child $ php resources.php Child

    Parent $ <?php $fh = fopen('/tmp/file', 'w+'); $pid = pcntl_fork(); if ($pid) { fputs($fh, 'Parent'.PHP_EOL); pcntl_wait($status); } else { fputs($fh, 'Child'.PHP_EOL); die(); } fclose($fh); readfile('/tmp/file'); Monday, 8 October 12

34. How do you get your children to talk to you?

    Monday, 8 October 12

35. Listening to your children • DB • files • memcached,

    apc or shared memory • sockets Monday, 8 October 12

36. Sockets Example <?php require('deep_thought.php'); $deepThought = new DeepThought(); $forks =

    array(); $sockets = array(); for ($i=0; $i < 5; $i++) { socket_create_pair(AF_UNIX, SOCK_STREAM, 0, $sockets); $forks[] = $pid = pcntl_fork(); if ($pid) { $socketPairs[$pid] = $sockets; } else { $data = $deepThought->findTheAnswer(); writeToSocket($data, $sockets[0]); die(); } } do { $pid = pcntl_wait($status); $forks = array_diff($forks, array($pid)); $sockets = $socketPairs[$pid]; $data = readFromSocket($sockets[1]); var_dump($data); } while (count($forks) > 0); Monday, 8 October 12

37. Next Level • Gearman • Message Queues ØMQ, RabbitMQ ..

    Monday, 8 October 12

38. Thinking Parallel “It turns out that what was difficult, and

    almost impossible, is to take an ordinary program and automatically figure out how to use the parallel computation effectively...” Richard P. Feynman Monday, 8 October 12

39. Input Processing Output Input Input Input Processing Processing Output Output

    Output Monday, 8 October 12

40. • Even Distribution of work /input • Size of messages

    • Data Dependancies Monday, 8 October 12

41. PHPLOC input: file system output: array of stats processing: tokenize

    and count occurrences Monday, 8 October 12

42. PHPLOC array(1781) { [0] => string(51) "zf2/bin/autoload_example.php" [1] => string(52)

    "zf2/bin/ autoload_examples.php" [2] => string(53) "zf2/bin/ classmap_generator.php" [3] => string(60) "zf2/bin/ createAutoloadTestClasses.php" [4] => string(54) "zf2/bin/ pluginmap_generator.php" ... foreach ($files as $key => $file) { $directory = dirname($file); if (!isset($this->directories[$directory])) { $this->directories[$directory] = TRUE; } ... Monday, 8 October 12

43. PHPLOC real 0m7.216s user 0m6.492s sys 0m0.240s $ time phploc

    --cores 1 zf2 $ time phploc --cores 2 zf2 real 0m5.500s user 0m6.686s sys 0m0.293s https://github.com/natmchugh/phploc Monday, 8 October 12

44. Amdahls Law Monday, 8 October 12

45. Embarrassingly Parallel Monday, 8 October 12

46. Mandelbrot Set Input: Pixel co-ordinates Processing: Hard core number crunching

    Output: An image Monday, 8 October 12

47. Dividing Work https://github.com/natmchugh/PHP-Fractals Monday, 8 October 12

48. Thanks Joind.in: http://joind.in/7002 @natmchugh http://fishtrap.co.uk Monday, 8 October 12

49. • Front Fork http://www.flickr.com/photos/de_atienza/ • Paint, Laptop & fork Lift

    http://consumeconsume.com/ • Haynes Manual http://www.flickr.com/photos/8490341@N04/ • Zombie Child http://www.flickr.com/photos/zenobia_joy/ • Signal http://www.flickr.com/photos/rbrwr/ • Pattern http://www.flickr.com/photos/eadaoinflynn Monday, 8 October 12