$30 off During Our Annual Pro Sale. View Details »

Gearman & Kohana

Gearman & Kohana

Russell Smith

April 20, 2012
Tweet

More Decks by Russell Smith

Other Decks in Technology

Transcript

  1. 15th May 2010
    Gearman (& Kohana), an introduction
    Russell Smith / UKD1 Limited / @ukd1
    Friday, 20 April 12

    View Slide

  2. What is Gearman?
    ✤ “Gearman is a system to
    farm out work to other
    machines, dispatching
    function calls to machines
    that are better suited to do
    work, to do work in parallel,
    to load balance lots of
    function calls, or to call
    functions between
    languages.”
    ✤ History:
    ✤ Originally written for LiveJournal in
    Perl
    ✤ The name is an anagram for “Manager,”
    since it dispatches jobs to be done, but
    does not do anything useful itself.
    ✤ More modern version is written in C
    Friday, 20 April 12

    View Slide

  3. Whats the point?
    ✤ Distribute and control load
    ✤ Reduce front end latency by offloading longer running tasks - not everything needs to
    be done now
    ✤ Run jobs synchronously or asynchronously, in parallel
    ✤ Run jobs on machines better suited for the work
    ✤ Run jobs cross language (queue a job using PHP, which is run by Ruby / C / ...)
    Friday, 20 April 12

    View Slide

  4. Common Applications
    ✤ Resizing images
    ✤ Encoding video
    ✤ Sending emails
    ✤ Asynchronous logging
    ✤ Migration of files / archiving
    ✤ Fetching / parsing feeds
    Friday, 20 April 12

    View Slide

  5. The Stack
    Application
    Gearman client API
    Gearman Server(s)
    Gearman worker API
    Workers
    Kohana
    PHP module
    Server
    Friday, 20 April 12

    View Slide

  6. Workers
    ✤ Workers can run on any machine, not just the machine with Gearmand.
    ✤ Running more workers means you can run more jobs at once.
    ✤ Workers can return statuses, fail, warning, complete, etc
    Friday, 20 April 12

    View Slide

  7. Jobs vs Tasks
    ✤ Tasks, do, dobackground - whats the
    difference?
    Friday, 20 April 12

    View Slide

  8. A very simple example
    # Reverse Client Code
    $client= new GearmanClient();
    $client->addServer();
    print $client->do("reverse", "Hello World!");
    # Reverse Worker Code
    $worker= new GearmanWorker();
    $worker->addServer();
    $worker->addFunction("reverse", "my_reverse_function");
    while ($worker->work());
    function my_reverse_function($job)
    {
    return strrev($job->workload());
    }
    Client Worker
    You can add as many functions
    as you wish, though each worker
    will (obviously) only run one job
    at a time
    The worker will run until it
    crashes or is killed.
    In this example, we connect to
    gearman (default is localhost)
    and request “reverse” be run.
    “do” means the job is enqueued
    and will run synchronously.
    Friday, 20 April 12

    View Slide

  9. Real world use; Sending Emails
    ✤ Sending emails is a common task, however it can be unpredictable and slow
    ✤ Sending using a Gearman worker makes it asynchronous, resulting in a faster front-end
    interface
    ✤ When sending bulk email multiple workers can increase throughput
    Friday, 20 April 12

    View Slide

  10. Installation of the Server
    ✤ Latest downloads are available from : https://code.launchpad.net/gearmand/
    +download
    ✤ ./configure
    ✤ make
    ✤ make install
    Friday, 20 April 12

    View Slide

  11. Starting Gearmand
    ✤ gearmand -d
    ✤ or, should you wish to run on a different ip / port;
    ✤ gearmand -d -L xxx.xxx.xxx.xxx -p 4730
    ✤ I would also suggest running gearmand under supervise, especially if you are using a
    checked out version.
    Friday, 20 April 12

    View Slide

  12. Installation of PHP module
    ✤ Make sure you have the PHP development tools (apt-get install php-dev)
    ✤ pecl install channel://pecl.php.net/gearman-0.7.0
    ✤ Make sure you add the module to the php.ini
    Friday, 20 April 12

    View Slide

  13. Kohana Worker
    ✤ Run as a Kohana controller
    ✤ Run seperate types of workers if you need
    ✤ Run by:
    php index.php “gearman/worker” &
    (or whatever you’ve called your route)
    ✤ Better version will register the PID somewhere for management and targeted killing
    Friday, 20 April 12

    View Slide

  14. Kohana Client
    ✤ I use a class in libraries...
    ✤ Just wrap the function parameters for the existing non-gearman code to fire a job
    Friday, 20 April 12

    View Slide

  15. Sample applicaion
    ✤ Demo
    ✤ Code will be on github tonight (http://github.com/ukd1)
    ✤ Slides will be on my blog too (http://www.ukd1.co.uk/blog/)
    ✤ Follow @ukd1
    Friday, 20 April 12

    View Slide

  16. Pitfalls with Gearman
    ✤ Memory leaks cause workers to crash
    ✤ Error handling must be robust
    ✤ If you use database connections, beware they can timeout and if kept open each worker will consume
    resources on the db server too
    ✤ During deployment you must use a method of restarting workers in a sensible manner across all your
    machines. The safest way is to stop jobs being created and kill / restart all the workers - having different
    versions of the same function accepting jobs is not a good idea. Capistrano rocks.
    ✤ Automatic restarting of workers is important - they do crash. We use deamontools, alternatives include
    launchd & the PHP based GearmanManager
    Friday, 20 April 12

    View Slide

  17. Other features not mentioned
    ✤ The complete lack of security (as yet). iptables is your friend.
    ✤ Persistent queues are available. By default jobs are lost if Gearmand crashes, using a
    persistent backend stops this. Backends include Drizzle, MySQL
    ✤ MySQL UDF (User Defined Functions) allow MySQL queries to act as Gearman clients.
    See: http://gearman.org/index.php?id=mysql_udf_readme
    Friday, 20 April 12

    View Slide

  18. Further reading
    ✤ Gearman (Danga) : http://www.danga.com/gearman/
    ✤ Gearman (C) : http://gearman.org/
    ✤ Gearman client (for PHP) : http://pecl.php.net/package/gearman
    ✤ Worker manger (in PHP) : http://github.com/brianlmoon/GearmanManager
    ✤ Deamontools : http://cr.yp.to/daemontools.html
    ✤ Launchd : http://en.wikipedia.org/wiki/Launchd
    Friday, 20 April 12

    View Slide