Gearman & Kohana

Transcript

1. 15th May 2010 Gearman (& Kohana), an introduction Russell Smith / UKD1 Limited / @ukd1 Friday, 20 April 12
2. What is Gearman? ✤ “Gearman is a system to farm out work to other machines, dispatching function calls to machines that are better suited to do work, to do work in parallel, to load balance lots of function calls, or to call functions between languages.” ✤ History: ✤ Originally written for LiveJournal in Perl ✤ The name is an anagram for “Manager,” since it dispatches jobs to be done, but does not do anything useful itself. ✤ More modern version is written in C Friday, 20 April 12
3. Whats the point? ✤ Distribute and control load ✤ Reduce front end latency by offloading longer running tasks - not everything needs to be done now ✤ Run jobs synchronously or asynchronously, in parallel ✤ Run jobs on machines better suited for the work ✤ Run jobs cross language (queue a job using PHP, which is run by Ruby / C / ...) Friday, 20 April 12
4. Common Applications ✤ Resizing images ✤ Encoding video ✤ Sending emails ✤ Asynchronous logging ✤ Migration of files / archiving ✤ Fetching / parsing feeds Friday, 20 April 12
5. The Stack Application Gearman client API Gearman Server(s) Gearman worker API Workers Kohana PHP module Server Friday, 20 April 12
6. Workers ✤ Workers can run on any machine, not just the machine with Gearmand. ✤ Running more workers means you can run more jobs at once. ✤ Workers can return statuses, fail, warning, complete, etc Friday, 20 April 12
7. Jobs vs Tasks ✤ Tasks, do, dobackground - whats the difference? Friday, 20 April 12
8. A very simple example # Reverse Client Code $client= new GearmanClient(); $client->addServer(); print $client->do("reverse", "Hello World!"); # Reverse Worker Code $worker= new GearmanWorker(); $worker->addServer(); $worker->addFunction("reverse", "my_reverse_function"); while ($worker->work()); function my_reverse_function($job) { return strrev($job->workload()); } Client Worker You can add as many functions as you wish, though each worker will (obviously) only run one job at a time The worker will run until it crashes or is killed. In this example, we connect to gearman (default is localhost) and request “reverse” be run. “do” means the job is enqueued and will run synchronously. Friday, 20 April 12
9. Real world use; Sending Emails ✤ Sending emails is a common task, however it can be unpredictable and slow ✤ Sending using a Gearman worker makes it asynchronous, resulting in a faster front-end interface ✤ When sending bulk email multiple workers can increase throughput Friday, 20 April 12
10. Installation of the Server ✤ Latest downloads are available from : https://code.launchpad.net/gearmand/ +download ✤ ./configure ✤ make ✤ make install Friday, 20 April 12
11. Starting Gearmand ✤ gearmand -d ✤ or, should you wish to run on a different ip / port; ✤ gearmand -d -L xxx.xxx.xxx.xxx -p 4730 ✤ I would also suggest running gearmand under supervise, especially if you are using a checked out version. Friday, 20 April 12
12. Installation of PHP module ✤ Make sure you have the PHP development tools (apt-get install php-dev) ✤ pecl install channel://pecl.php.net/gearman-0.7.0 ✤ Make sure you add the module to the php.ini Friday, 20 April 12
13. Kohana Worker ✤ Run as a Kohana controller ✤ Run seperate types of workers if you need ✤ Run by: php index.php “gearman/worker” & (or whatever you’ve called your route) ✤ Better version will register the PID somewhere for management and targeted killing Friday, 20 April 12
14. Kohana Client ✤ I use a class in libraries... ✤ Just wrap the function parameters for the existing non-gearman code to fire a job Friday, 20 April 12
15. Sample applicaion ✤ Demo ✤ Code will be on github tonight (http://github.com/ukd1) ✤ Slides will be on my blog too (http://www.ukd1.co.uk/blog/) ✤ Follow @ukd1 Friday, 20 April 12
16. Pitfalls with Gearman ✤ Memory leaks cause workers to crash ✤ Error handling must be robust ✤ If you use database connections, beware they can timeout and if kept open each worker will consume resources on the db server too ✤ During deployment you must use a method of restarting workers in a sensible manner across all your machines. The safest way is to stop jobs being created and kill / restart all the workers - having different versions of the same function accepting jobs is not a good idea. Capistrano rocks. ✤ Automatic restarting of workers is important - they do crash. We use deamontools, alternatives include launchd & the PHP based GearmanManager Friday, 20 April 12
17. Other features not mentioned ✤ The complete lack of security (as yet). iptables is your friend. ✤ Persistent queues are available. By default jobs are lost if Gearmand crashes, using a persistent backend stops this. Backends include Drizzle, MySQL ✤ MySQL UDF (User Defined Functions) allow MySQL queries to act as Gearman clients. See: http://gearman.org/index.php?id=mysql_udf_readme Friday, 20 April 12
18. Further reading ✤ Gearman (Danga) : http://www.danga.com/gearman/ ✤ Gearman (C) : http://gearman.org/ ✤ Gearman client (for PHP) : http://pecl.php.net/package/gearman ✤ Worker manger (in PHP) : http://github.com/brianlmoon/GearmanManager ✤ Deamontools : http://cr.yp.to/daemontools.html ✤ Launchd : http://en.wikipedia.org/wiki/Launchd Friday, 20 April 12