Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Shifting gears with Gearman

Vranac Srdjan
September 27, 2015

Shifting gears with Gearman

Everyone starts with a simple one-machine setup, running PHP(or whatever else you fancy), MySQL/PostgreSql and Apache/Nginx. Sooner or later that will not be enough, you will have more operations to do than can be done in a single request, or you will have some operations that need to be done in parallel, and you will be faced with some important architecture decisions.
This talk will show you how a distributed system built with Gearman looks like, and more importantly the strange and wonderful things you will encounter in production and how to prepare for them.

Vranac Srdjan

September 27, 2015

More Decks by Vranac Srdjan

Other Decks in Programming


  1. business owner, developer, consultant, mercenary, writing terrible code that performs

    exceptionally, wrangling elePHPants and Pythons, obsessed with process automation, interested in continuous integration and delivery, clean code, testing, best practices and distributed systems
  2. 30.000.000 Products / 12 Hours = 2.500.000 Products / 3600

    Seconds = 695 Products 600 nodes = 1.1 Sec. per product
  3. GEARMAN SETUP SERVER: • RHEL/Fedora: Run yum install gearmand •

    Debain/Ubuntu Package: Run apt-get install gearman-job-server. EXTENSION: pecl -d preferred_state=stable install -f gearman
  4. TASKS VS JOBS • Task
 A task is any request

    or communication between the client and the job server. • Jobs
 A job is something the worker does, continuously waiting on the job server to tell him when to start and with what arguments.

    Wait 2 hours for a job, then exit (upstart will restart process)
 // start the worker
 while ($worker->work() || $worker->returnCode() == GEARMAN_TIMEOUT) {
 if($worker->returnCode() == GEARMAN_TIMEOUT) {
 print “No jobs available" . PHP_EOL;
  6. RUNNING ORDER • Gearman queue is FIFO (first in, first

    out) • Multiple workers mean no reliable order of execution • Order of execution should not matter
  7. RELIABILITY • Job that is enqueued will be processed… Sometime

    • When job fails… Re-queue if needed • Queue systems are unreliable by design
  8. EVENTUAL CONSISTENCY • Data will be available across all the

    nodes… Eventually • Most projects can support it • High priority changes should be handled separately
  9. LINKS • Gearman: http://gearman.org/ • Supervisor: http://supervisord.org • Upstart: http://upstart.ubuntu.com/

    • Measure Anything, Measure Everything: http:// codeascraft.com/2011/02/15/measure-anything- measure-everything