Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Fork It ! Parallel Processing in PHP

Fork It ! Parallel Processing in PHP

PHPNW 2012

Nathaniel McHugh

October 07, 2012
Tweet

More Decks by Nathaniel McHugh

Other Decks in Technology

Transcript

  1. Parallel Processing in PHP
    Fork It !
    Nathaniel McHugh
    Monday, 8 October 12

    View Slide

  2. • Software Engineer Inviqa
    • @natmchugh
    • http://fishtrap.co.uk
    • likes doing useless PHP
    Monday, 8 October 12

    View Slide

  3. Monday, 8 October 12

    View Slide

  4. Pasta
    or
    Monday, 8 October 12

    View Slide

  5. 0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    1 cup 2 cups 3 cups 4 cups
    t (minutes)
    kettle hob microwave
    Monday, 8 October 12

    View Slide

  6. Monday, 8 October 12

    View Slide

  7. Number of Processors
    https://github.com/natmchugh/PHP-Num-
    Processors
    php > echo num_processors_available();
    2
    php > echo num_processors_configured();
    2
    Monday, 8 October 12

    View Slide

  8. Forking
    • php has no multi
    threading
    • pcntl extension
    • *nix only
    • cli only really
    Monday, 8 October 12

    View Slide

  9. fork()
    Monday, 8 October 12

    View Slide

  10. fork()
    $ ps -a | grep ps
    3394 ttys003 0:00.00 ps -a
    3395 ttys003 0:00.00 grep ps
    $
    Monday, 8 October 12

    View Slide

  11. fork()
    $ ps -a | grep ps
    3394 ttys003 0:00.00 ps -a
    3395 ttys003 0:00.00 grep ps
    $
    $ man fork | grep returns
    Upon successful completion, fork() returns a
    value of 0 to the child
    process and returns the process ID of the child
    process to the parent
    Monday, 8 October 12

    View Slide

  12. Monday, 8 October 12

    View Slide

  13. Concurrency
    real 1m10.322s
    real 0m39.928s
    real 0m45.827s
    Monday, 8 October 12

    View Slide

  14. Die Hard PHP
    Daemons
    while (true) {
    $worker->doSomeWork();
    }
    Monday, 8 October 12

    View Slide

  15. From the
    Beginning
    Monday, 8 October 12

    View Slide

  16. From the
    Beginning
    $ php start_with_manual.php
    $
    Monday, 8 October 12

    View Slide

  17. $pid = pcntl_fork();
    if ($pid == -1) {
    die('could not fork');
    } else if ($pid) {
    // we are the parent
    sleep(5);
    pcntl_wait($status); //Protect against Zombie children
    } else {
    // we are the child
    sleep(5);
    }
    Monday, 8 October 12

    View Slide

  18. $pid = pcntl_fork();
    if ($pid == -1) {
    die('could not fork');
    } else if ($pid) {
    // we are the parent
    sleep(5);
    pcntl_wait($status); //Protect against Zombie children
    } else {
    // we are the child
    sleep(5);
    }
    $ time php example_with_delay.php
    real 0m5.072s
    user 0m0.023s
    sys 0m0.021s
    Monday, 8 October 12

    View Slide

  19. • Zombies are dead
    • Orphans are children
    whose parent has died
    Zombies vs Orphans
    Monday, 8 October 12

    View Slide

  20. multiple children?
    for ($i=0; $i < 5; $i++) {
    $pid = pcntl_fork();
    if ($pid) {
    pcntl_wait($status);
    } else {
    echo 'starting child ',$i,PHP_EOL;
    sleep(5);
    die();
    }
    }
    Monday, 8 October 12

    View Slide

  21. multiple children?
    for ($i=0; $i < 5; $i++) {
    $pid = pcntl_fork();
    if ($pid) {
    pcntl_wait($status);
    } else {
    echo 'starting child ',$i,PHP_EOL;
    sleep(5);
    die();
    }
    }
    $ time php multiple_children_bad.php
    real 0m25.441s
    user 0m0.057s
    sys 0m0.059s
    Monday, 8 October 12

    View Slide

  22. multiple children?
    for ($i=0; $i < 5; $i++) {
    $pid = pcntl_fork();
    if ($pid) {
    pcntl_wait($status);
    } else {
    echo 'starting child ',$i,PHP_EOL;
    sleep(5);
    die();
    }
    }
    $ time php multiple_children_bad.php
    real 0m25.441s
    user 0m0.057s
    sys 0m0.059s
    Monday, 8 October 12

    View Slide

  23. multiple children?
    for ($i=0; $i < 5; $i++) {
    $pid = pcntl_fork();
    if ($pid) {
    pcntl_wait($status);
    } else {
    echo 'starting child ',$i,PHP_EOL;
    sleep(5);
    die();
    }
    }
    $ time php multiple_children_bad.php
    real 0m25.441s
    user 0m0.057s
    sys 0m0.059s
    Monday, 8 October 12

    View Slide

  24. The Pattern
    require('deep_thought.php');
    $deepThought = new DeepThought();
    $children = array();
    for ($i=0; $i < 5; $i++) {
    $children[] = $pid = pcntl_fork();
    if ($pid == -1) {
    die('could not fork');
    } else if (0 === $pid) {
    $lifeUniverseEverthing = $deepThought->findTheAnswer();
    echo "Child $i says:",$lifeUniverseEverthing,PHP_EOL;
    die();
    }
    }
    do {
    $pid = pcntl_wait($status);
    $children = array_diff($children, array($pid));
    } while (count($children) > 0);
    Monday, 8 October 12

    View Slide

  25. The Pattern
    require('deep_thought.php');
    $deepThought = new DeepThought();
    $children = array();
    for ($i=0; $i < 5; $i++) {
    $children[] = $pid = pcntl_fork();
    if ($pid == -1) {
    die('could not fork');
    } else if (0 === $pid) {
    $lifeUniverseEverthing = $deepThought->findTheAnswer();
    echo "Child $i says:",$lifeUniverseEverthing,PHP_EOL;
    die();
    }
    }
    do {
    $pid = pcntl_wait($status);
    $children = array_diff($children, array($pid));
    } while (count($children) > 0);
    $ time php the_pattern.php
    Child 0 says:42
    Child 3 says:42
    Child 1 says:Child 2 says:42
    42
    Child 4 says:42
    real 0m5.453s
    user 0m0.056s
    sys 0m0.060s
    $
    Monday, 8 October 12

    View Slide

  26. Signals
    // declare where callbacks will occur in your script
    declare(ticks = 1);
    function sig_handler($signo)
    {
    die("caught a siginal $signo".PHP_EOL);
    }
    $pid = pcntl_fork();
    if ($pid == -1) {
    die('could not fork');
    } else if ($pid) {
    // we are the parent
    pcntl_wait($status);
    die('Parent finished'.PHP_EOL);
    } else {
    pcntl_signal(SIGINT, "sig_handler");
    // we are the child
    while (true) {
    sleep(5);
    echo 'More',PHP_EOL;
    }
    die('Child finished'.PHP_EOL);
    }
    Monday, 8 October 12

    View Slide

  27. Signals
    // declare where callbacks will occur in your script
    declare(ticks = 1);
    function sig_handler($signo)
    {
    die("caught a siginal $signo".PHP_EOL);
    }
    $pid = pcntl_fork();
    if ($pid == -1) {
    die('could not fork');
    } else if ($pid) {
    // we are the parent
    pcntl_wait($status);
    die('Parent finished'.PHP_EOL);
    } else {
    pcntl_signal(SIGINT, "sig_handler");
    // we are the child
    while (true) {
    sleep(5);
    echo 'More',PHP_EOL;
    }
    die('Child finished'.PHP_EOL);
    }
    $ php signals.php
    More
    More
    More
    More
    More
    More
    ^Ccaught a siginal 2
    Monday, 8 October 12

    View Slide

  28. SIGCHLD
    declare(ticks = 1);
    pcntl_signal(SIGCHLD, 'sig_child');
    $children = array();
    for ($i=0; $i < 5; $i++) {
    $children[] = $pid = pcntl_fork();
    if (0 === $pid) {
    die();
    }
    }
    while ($children) {
    sleep(1);
    echo '.';
    }
    function sig_child()
    {
    global $children;
    echo("Caught SIGCHLD".PHP_EOL);
    while (($pid = pcntl_wait($status, WNOHANG)) > 0) {
    $children = array_diff($children, array($pid));
    echo "Child $pid collected".PHP_EOL;
    }
    }
    Monday, 8 October 12

    View Slide

  29. SIGCHLD
    declare(ticks = 1);
    pcntl_signal(SIGCHLD, 'sig_child');
    $children = array();
    for ($i=0; $i < 5; $i++) {
    $children[] = $pid = pcntl_fork();
    if (0 === $pid) {
    die();
    }
    }
    while ($children) {
    sleep(1);
    echo '.';
    }
    function sig_child()
    {
    global $children;
    echo("Caught SIGCHLD".PHP_EOL);
    while (($pid = pcntl_wait($status, WNOHANG)) > 0) {
    $children = array_diff($children, array($pid));
    echo "Child $pid collected".PHP_EOL;
    }
    }
    php sigchld.php
    ....Child 0 says:42
    Child 2 says:42
    Child 3 says:42
    Child 1 says:42
    Child 4 says:42
    .Caught SIGCHLD
    Child 22877 collected
    Child 22876 collected
    Child 22874 collected
    .Caught SIGCHLD
    Child 22875 collected
    .Caught SIGCHLD
    Child 22878 collected
    Monday, 8 October 12

    View Slide

  30. Process Isolation
    $a = 'foo';
    $b = 'bar';
    $pid = pcntl_fork();
    if ($pid) {
    $a = 'baz';
    pcntl_wait($status);
    } else {
    $b = 'bat';
    die();
    }
    var_dump($a, $b);
    Monday, 8 October 12

    View Slide

  31. Process Isolation
    $a = 'foo';
    $b = 'bar';
    $pid = pcntl_fork();
    if ($pid) {
    $a = 'baz';
    pcntl_wait($status);
    } else {
    $b = 'bat';
    die();
    }
    var_dump($a, $b);
    $ php process_isolation.php
    string(3) "baz"
    string(3) "bar"
    $
    Monday, 8 October 12

    View Slide

  32. Resources
    $fh = fopen('/tmp/file', 'w+');
    $pid = pcntl_fork();
    if ($pid) {
    fputs($fh, 'Parent'.PHP_EOL);
    pcntl_wait($status);
    } else {
    fputs($fh, 'Child'.PHP_EOL);
    die();
    }
    fclose($fh);
    readfile('/tmp/file');
    Monday, 8 October 12

    View Slide

  33. Resources
    $ php resources.php
    Parent
    Child
    $ php resources.php
    Child
    Parent
    $
    $fh = fopen('/tmp/file', 'w+');
    $pid = pcntl_fork();
    if ($pid) {
    fputs($fh, 'Parent'.PHP_EOL);
    pcntl_wait($status);
    } else {
    fputs($fh, 'Child'.PHP_EOL);
    die();
    }
    fclose($fh);
    readfile('/tmp/file');
    Monday, 8 October 12

    View Slide

  34. How do you get your
    children to talk to you?
    Monday, 8 October 12

    View Slide

  35. Listening to your children
    • DB
    • files
    • memcached, apc or shared memory
    • sockets
    Monday, 8 October 12

    View Slide

  36. Sockets Example
    require('deep_thought.php');
    $deepThought = new DeepThought();
    $forks = array();
    $sockets = array();
    for ($i=0; $i < 5; $i++) {
    socket_create_pair(AF_UNIX, SOCK_STREAM, 0, $sockets);
    $forks[] = $pid = pcntl_fork();
    if ($pid) {
    $socketPairs[$pid] = $sockets;
    } else {
    $data = $deepThought->findTheAnswer();
    writeToSocket($data, $sockets[0]);
    die();
    }
    }
    do {
    $pid = pcntl_wait($status);
    $forks = array_diff($forks, array($pid));
    $sockets = $socketPairs[$pid];
    $data = readFromSocket($sockets[1]);
    var_dump($data);
    } while (count($forks) > 0);
    Monday, 8 October 12

    View Slide

  37. Next Level
    • Gearman
    • Message Queues ØMQ, RabbitMQ ..
    Monday, 8 October 12

    View Slide

  38. Thinking Parallel
    “It turns out that what was
    difficult, and almost impossible, is
    to take an ordinary program and
    automatically figure out how to use
    the parallel computation
    effectively...”
    Richard P. Feynman
    Monday, 8 October 12

    View Slide

  39. Input
    Processing
    Output
    Input
    Input Input
    Processing Processing
    Output Output
    Output
    Monday, 8 October 12

    View Slide

  40. • Even Distribution of work /input
    • Size of messages
    • Data Dependancies
    Monday, 8 October 12

    View Slide

  41. PHPLOC
    input: file system
    output: array of stats
    processing: tokenize
    and count occurrences
    Monday, 8 October 12

    View Slide

  42. PHPLOC
    array(1781) {
    [0] =>
    string(51) "zf2/bin/autoload_example.php"
    [1] =>
    string(52) "zf2/bin/
    autoload_examples.php"
    [2] =>
    string(53) "zf2/bin/
    classmap_generator.php"
    [3] =>
    string(60) "zf2/bin/
    createAutoloadTestClasses.php"
    [4] =>
    string(54) "zf2/bin/
    pluginmap_generator.php"
    ...
    foreach ($files as $key => $file) {
    $directory = dirname($file);
    if (!isset($this->directories[$directory])) {
    $this->directories[$directory] = TRUE;
    }
    ...
    Monday, 8 October 12

    View Slide

  43. PHPLOC
    real 0m7.216s
    user 0m6.492s
    sys 0m0.240s
    $ time phploc --cores 1 zf2
    $ time phploc --cores 2 zf2 real 0m5.500s
    user 0m6.686s
    sys 0m0.293s
    https://github.com/natmchugh/phploc
    Monday, 8 October 12

    View Slide

  44. Amdahls Law
    Monday, 8 October 12

    View Slide

  45. Embarrassingly Parallel
    Monday, 8 October 12

    View Slide

  46. Mandelbrot Set
    Input: Pixel co-ordinates
    Processing: Hard core number crunching
    Output: An image
    Monday, 8 October 12

    View Slide

  47. Dividing Work
    https://github.com/natmchugh/PHP-Fractals
    Monday, 8 October 12

    View Slide

  48. Thanks
    Joind.in: http://joind.in/7002
    @natmchugh
    http://fishtrap.co.uk
    Monday, 8 October 12

    View Slide

  49. • Front Fork http://www.flickr.com/photos/de_atienza/
    • Paint, Laptop & fork Lift http://consumeconsume.com/
    • Haynes Manual http://www.flickr.com/photos/[email protected]/
    • Zombie Child http://www.flickr.com/photos/zenobia_joy/
    • Signal http://www.flickr.com/photos/rbrwr/
    • Pattern http://www.flickr.com/photos/eadaoinflynn
    Monday, 8 October 12

    View Slide