Gearman & Kohana

Transcript

1. 15th May 2010
   Gearman (& Kohana), an introduction
   Russell Smith / UKD1 Limited / @ukd1
   Friday, 20 April 12
   

   View Slide

2. What is Gearman?
   ✤ “Gearman is a system to
   farm out work to other
   machines, dispatching
   function calls to machines
   that are better suited to do
   work, to do work in parallel,
   to load balance lots of
   function calls, or to call
   functions between
   languages.”
   ✤ History:
   ✤ Originally written for LiveJournal in
   Perl
   ✤ The name is an anagram for “Manager,”
   since it dispatches jobs to be done, but
   does not do anything useful itself.
   ✤ More modern version is written in C
   Friday, 20 April 12
   

   View Slide

3. Whats the point?
   ✤ Distribute and control load
   ✤ Reduce front end latency by offloading longer running tasks - not everything needs to
   be done now
   ✤ Run jobs synchronously or asynchronously, in parallel
   ✤ Run jobs on machines better suited for the work
   ✤ Run jobs cross language (queue a job using PHP, which is run by Ruby / C / ...)
   Friday, 20 April 12
   

   View Slide

4. Common Applications
   ✤ Resizing images
   ✤ Encoding video
   ✤ Sending emails
   ✤ Asynchronous logging
   ✤ Migration of files / archiving
   ✤ Fetching / parsing feeds
   Friday, 20 April 12
   

   View Slide

5. The Stack
   Application
   Gearman client API
   Gearman Server(s)
   Gearman worker API
   Workers
   Kohana
   PHP module
   Server
   Friday, 20 April 12
   

   View Slide

6. Workers
   ✤ Workers can run on any machine, not just the machine with Gearmand.
   ✤ Running more workers means you can run more jobs at once.
   ✤ Workers can return statuses, fail, warning, complete, etc
   Friday, 20 April 12
   

   View Slide

7. Jobs vs Tasks
   ✤ Tasks, do, dobackground - whats the
   difference?
   Friday, 20 April 12
   

   View Slide

8. A very simple example
   # Reverse Client Code
   $client= new GearmanClient();
   $client->addServer();
   print $client->do("reverse", "Hello World!");
   # Reverse Worker Code
   $worker= new GearmanWorker();
   $worker->addServer();
   $worker->addFunction("reverse", "my_reverse_function");
   while ($worker->work());
   function my_reverse_function($job)
   {
   return strrev($job->workload());
   }
   Client Worker
   You can add as many functions
   as you wish, though each worker
   will (obviously) only run one job
   at a time
   The worker will run until it
   crashes or is killed.
   In this example, we connect to
   gearman (default is localhost)
   and request “reverse” be run.
   “do” means the job is enqueued
   and will run synchronously.
   Friday, 20 April 12
   

   View Slide

9. Real world use; Sending Emails
   ✤ Sending emails is a common task, however it can be unpredictable and slow
   ✤ Sending using a Gearman worker makes it asynchronous, resulting in a faster front-end
   interface
   ✤ When sending bulk email multiple workers can increase throughput
   Friday, 20 April 12
   

   View Slide

10. Installation of the Server
    ✤ Latest downloads are available from : https://code.launchpad.net/gearmand/
    +download
    ✤ ./configure
    ✤ make
    ✤ make install
    Friday, 20 April 12
    

    View Slide

11. Starting Gearmand
    ✤ gearmand -d
    ✤ or, should you wish to run on a different ip / port;
    ✤ gearmand -d -L xxx.xxx.xxx.xxx -p 4730
    ✤ I would also suggest running gearmand under supervise, especially if you are using a
    checked out version.
    Friday, 20 April 12
    

    View Slide

12. Installation of PHP module
    ✤ Make sure you have the PHP development tools (apt-get install php-dev)
    ✤ pecl install channel://pecl.php.net/gearman-0.7.0
    ✤ Make sure you add the module to the php.ini
    Friday, 20 April 12
    

    View Slide

13. Kohana Worker
    ✤ Run as a Kohana controller
    ✤ Run seperate types of workers if you need
    ✤ Run by:
    php index.php “gearman/worker” &
    (or whatever you’ve called your route)
    ✤ Better version will register the PID somewhere for management and targeted killing
    Friday, 20 April 12
    

    View Slide

14. Kohana Client
    ✤ I use a class in libraries...
    ✤ Just wrap the function parameters for the existing non-gearman code to fire a job
    Friday, 20 April 12
    

    View Slide

15. Sample applicaion
    ✤ Demo
    ✤ Code will be on github tonight (http://github.com/ukd1)
    ✤ Slides will be on my blog too (http://www.ukd1.co.uk/blog/)
    ✤ Follow @ukd1
    Friday, 20 April 12
    

    View Slide

16. Pitfalls with Gearman
    ✤ Memory leaks cause workers to crash
    ✤ Error handling must be robust
    ✤ If you use database connections, beware they can timeout and if kept open each worker will consume
    resources on the db server too
    ✤ During deployment you must use a method of restarting workers in a sensible manner across all your
    machines. The safest way is to stop jobs being created and kill / restart all the workers - having different
    versions of the same function accepting jobs is not a good idea. Capistrano rocks.
    ✤ Automatic restarting of workers is important - they do crash. We use deamontools, alternatives include
    launchd & the PHP based GearmanManager
    Friday, 20 April 12
    

    View Slide

17. Other features not mentioned
    ✤ The complete lack of security (as yet). iptables is your friend.
    ✤ Persistent queues are available. By default jobs are lost if Gearmand crashes, using a
    persistent backend stops this. Backends include Drizzle, MySQL
    ✤ MySQL UDF (User Defined Functions) allow MySQL queries to act as Gearman clients.
    See: http://gearman.org/index.php?id=mysql_udf_readme
    Friday, 20 April 12
    

    View Slide

18. Further reading
    ✤ Gearman (Danga) : http://www.danga.com/gearman/
    ✤ Gearman (C) : http://gearman.org/
    ✤ Gearman client (for PHP) : http://pecl.php.net/package/gearman
    ✤ Worker manger (in PHP) : http://github.com/brianlmoon/GearmanManager
    ✤ Deamontools : http://cr.yp.to/daemontools.html
    ✤ Launchd : http://en.wikipedia.org/wiki/Launchd
    Friday, 20 April 12
    

    View Slide