Fork It ! Parallel Processing in PHP

Parallel Processing in PHP Fork It ! Nathaniel McHugh Monday,

8 October 12

• Software Engineer Inviqa • @natmchugh • http://fishtrap.co.uk • likes

doing useless PHP Monday, 8 October 12

Monday, 8 October 12

Pasta or Monday, 8 October 12

0 1 2 3 4 5 6 7 8 9

10 1 cup 2 cups 3 cups 4 cups t (minutes) kettle hob microwave Monday, 8 October 12

Monday, 8 October 12

Number of Processors https://github.com/natmchugh/PHP-Num- Processors php > echo num_processors_available(); 2

php > echo num_processors_configured(); 2 Monday, 8 October 12

Forking • php has no multi threading • pcntl extension

• *nix only • cli only really Monday, 8 October 12

fork() Monday, 8 October 12

fork() $ ps -a | grep ps 3394 ttys003 0:00.00

ps -a 3395 ttys003 0:00.00 grep ps $ Monday, 8 October 12

fork() $ ps -a | grep ps 3394 ttys003 0:00.00

ps -a 3395 ttys003 0:00.00 grep ps $ $ man fork | grep returns Upon successful completion, fork() returns a value of 0 to the child process and returns the process ID of the child process to the parent Monday, 8 October 12

Monday, 8 October 12

Concurrency real 1m10.322s real 0m39.928s real 0m45.827s Monday, 8 October

12

Die Hard PHP Daemons <?php while (true) { $worker->doSomeWork(); }

Monday, 8 October 12

From the Beginning Monday, 8 October 12

From the Beginning $ php start_with_manual.php $ Monday, 8 October

12

<?php $pid = pcntl_fork(); if ($pid == -1) { die('could

not fork'); } else if ($pid) { // we are the parent sleep(5); pcntl_wait($status); //Protect against Zombie children } else { // we are the child sleep(5); } Monday, 8 October 12

<?php $pid = pcntl_fork(); if ($pid == -1) { die('could

not fork'); } else if ($pid) { // we are the parent sleep(5); pcntl_wait($status); //Protect against Zombie children } else { // we are the child sleep(5); } $ time php example_with_delay.php real 0m5.072s user 0m0.023s sys 0m0.021s Monday, 8 October 12

• Zombies are dead • Orphans are children whose parent

has died Zombies vs Orphans Monday, 8 October 12

multiple children? <?php for ($i=0; $i < 5; $i++) {

$pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } Monday, 8 October 12

multiple children? <?php for ($i=0; $i < 5; $i++) {

$pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12

multiple children? <?php for ($i=0; $i < 5; $i++) {

$pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12

multiple children? <?php for ($i=0; $i < 5; $i++) {

$pid = pcntl_fork(); if ($pid) { pcntl_wait($status); } else { echo 'starting child ',$i,PHP_EOL; sleep(5); die(); } } $ time php multiple_children_bad.php real 0m25.441s user 0m0.057s sys 0m0.059s Monday, 8 October 12

The Pattern require('deep_thought.php'); $deepThought = new DeepThought(); $children = array();

for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if (0 === $pid) { $lifeUniverseEverthing = $deepThought->findTheAnswer(); echo "Child $i says:",$lifeUniverseEverthing,PHP_EOL; die(); } } do { $pid = pcntl_wait($status); $children = array_diff($children, array($pid)); } while (count($children) > 0); Monday, 8 October 12

The Pattern require('deep_thought.php'); $deepThought = new DeepThought(); $children = array();

for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if (0 === $pid) { $lifeUniverseEverthing = $deepThought->findTheAnswer(); echo "Child $i says:",$lifeUniverseEverthing,PHP_EOL; die(); } } do { $pid = pcntl_wait($status); $children = array_diff($children, array($pid)); } while (count($children) > 0); $ time php the_pattern.php Child 0 says:42 Child 3 says:42 Child 1 says:Child 2 says:42 42 Child 4 says:42 real 0m5.453s user 0m0.056s sys 0m0.060s $ Monday, 8 October 12

Signals <?php // declare where callbacks will occur in your

script declare(ticks = 1); function sig_handler($signo) { die("caught a siginal $signo".PHP_EOL); } $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent pcntl_wait($status); die('Parent finished'.PHP_EOL); } else { pcntl_signal(SIGINT, "sig_handler"); // we are the child while (true) { sleep(5); echo 'More',PHP_EOL; } die('Child finished'.PHP_EOL); } Monday, 8 October 12

Signals <?php // declare where callbacks will occur in your

script declare(ticks = 1); function sig_handler($signo) { die("caught a siginal $signo".PHP_EOL); } $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent pcntl_wait($status); die('Parent finished'.PHP_EOL); } else { pcntl_signal(SIGINT, "sig_handler"); // we are the child while (true) { sleep(5); echo 'More',PHP_EOL; } die('Child finished'.PHP_EOL); } $ php signals.php More More More More More More ^Ccaught a siginal 2 Monday, 8 October 12

SIGCHLD <?php declare(ticks = 1); pcntl_signal(SIGCHLD, 'sig_child'); $children = array();

for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if (0 === $pid) { die(); } } while ($children) { sleep(1); echo '.'; } function sig_child() { global $children; echo("Caught SIGCHLD".PHP_EOL); while (($pid = pcntl_wait($status, WNOHANG)) > 0) { $children = array_diff($children, array($pid)); echo "Child $pid collected".PHP_EOL; } } Monday, 8 October 12

SIGCHLD <?php declare(ticks = 1); pcntl_signal(SIGCHLD, 'sig_child'); $children = array();

for ($i=0; $i < 5; $i++) { $children[] = $pid = pcntl_fork(); if (0 === $pid) { die(); } } while ($children) { sleep(1); echo '.'; } function sig_child() { global $children; echo("Caught SIGCHLD".PHP_EOL); while (($pid = pcntl_wait($status, WNOHANG)) > 0) { $children = array_diff($children, array($pid)); echo "Child $pid collected".PHP_EOL; } } php sigchld.php ....Child 0 says:42 Child 2 says:42 Child 3 says:42 Child 1 says:42 Child 4 says:42 .Caught SIGCHLD Child 22877 collected Child 22876 collected Child 22874 collected .Caught SIGCHLD Child 22875 collected .Caught SIGCHLD Child 22878 collected Monday, 8 October 12

Process Isolation <?php $a = 'foo'; $b = 'bar'; $pid

= pcntl_fork(); if ($pid) { $a = 'baz'; pcntl_wait($status); } else { $b = 'bat'; die(); } var_dump($a, $b); Monday, 8 October 12

Process Isolation <?php $a = 'foo'; $b = 'bar'; $pid

= pcntl_fork(); if ($pid) { $a = 'baz'; pcntl_wait($status); } else { $b = 'bat'; die(); } var_dump($a, $b); $ php process_isolation.php string(3) "baz" string(3) "bar" $ Monday, 8 October 12

Resources <?php $fh = fopen('/tmp/file', 'w+'); $pid = pcntl_fork(); if

($pid) { fputs($fh, 'Parent'.PHP_EOL); pcntl_wait($status); } else { fputs($fh, 'Child'.PHP_EOL); die(); } fclose($fh); readfile('/tmp/file'); Monday, 8 October 12

Resources $ php resources.php Parent Child $ php resources.php Child

Parent $ <?php $fh = fopen('/tmp/file', 'w+'); $pid = pcntl_fork(); if ($pid) { fputs($fh, 'Parent'.PHP_EOL); pcntl_wait($status); } else { fputs($fh, 'Child'.PHP_EOL); die(); } fclose($fh); readfile('/tmp/file'); Monday, 8 October 12

How do you get your children to talk to you?

Monday, 8 October 12

Listening to your children • DB • files • memcached,

apc or shared memory • sockets Monday, 8 October 12

Sockets Example <?php require('deep_thought.php'); $deepThought = new DeepThought(); $forks =

array(); $sockets = array(); for ($i=0; $i < 5; $i++) { socket_create_pair(AF_UNIX, SOCK_STREAM, 0, $sockets); $forks[] = $pid = pcntl_fork(); if ($pid) { $socketPairs[$pid] = $sockets; } else { $data = $deepThought->findTheAnswer(); writeToSocket($data, $sockets[0]); die(); } } do { $pid = pcntl_wait($status); $forks = array_diff($forks, array($pid)); $sockets = $socketPairs[$pid]; $data = readFromSocket($sockets[1]); var_dump($data); } while (count($forks) > 0); Monday, 8 October 12

Next Level • Gearman • Message Queues ØMQ, RabbitMQ ..

Monday, 8 October 12

Thinking Parallel “It turns out that what was difficult, and

almost impossible, is to take an ordinary program and automatically figure out how to use the parallel computation effectively...” Richard P. Feynman Monday, 8 October 12

Input Processing Output Input Input Input Processing Processing Output Output

Output Monday, 8 October 12

• Even Distribution of work /input • Size of messages

• Data Dependancies Monday, 8 October 12

PHPLOC input: file system output: array of stats processing: tokenize

and count occurrences Monday, 8 October 12

PHPLOC array(1781) { [0] => string(51) "zf2/bin/autoload_example.php" [1] => string(52)

"zf2/bin/ autoload_examples.php" [2] => string(53) "zf2/bin/ classmap_generator.php" [3] => string(60) "zf2/bin/ createAutoloadTestClasses.php" [4] => string(54) "zf2/bin/ pluginmap_generator.php" ... foreach ($files as $key => $file) { $directory = dirname($file); if (!isset($this->directories[$directory])) { $this->directories[$directory] = TRUE; } ... Monday, 8 October 12

PHPLOC real 0m7.216s user 0m6.492s sys 0m0.240s $ time phploc

--cores 1 zf2 $ time phploc --cores 2 zf2 real 0m5.500s user 0m6.686s sys 0m0.293s https://github.com/natmchugh/phploc Monday, 8 October 12

Amdahls Law Monday, 8 October 12

Embarrassingly Parallel Monday, 8 October 12

Mandelbrot Set Input: Pixel co-ordinates Processing: Hard core number crunching

Output: An image Monday, 8 October 12

Dividing Work https://github.com/natmchugh/PHP-Fractals Monday, 8 October 12

Thanks Joind.in: http://joind.in/7002 @natmchugh http://fishtrap.co.uk Monday, 8 October 12

• Front Fork http://www.flickr.com/photos/de_atienza/ • Paint, Laptop & fork Lift

http://consumeconsume.com/ • Haynes Manual http://www.flickr.com/photos/8490341@N04/ • Zombie Child http://www.flickr.com/photos/zenobia_joy/ • Signal http://www.flickr.com/photos/rbrwr/ • Pattern http://www.flickr.com/photos/eadaoinflynn Monday, 8 October 12