Shifting gears with Gearman

F2d82b268a7cbccc9809c939428df64f?s=47 Vranac Srdjan
September 27, 2015

Shifting gears with Gearman

Everyone starts with a simple one-machine setup, running PHP(or whatever else you fancy), MySQL/PostgreSql and Apache/Nginx. Sooner or later that will not be enough, you will have more operations to do than can be done in a single request, or you will have some operations that need to be done in parallel, and you will be faced with some important architecture decisions.
This talk will show you how a distributed system built with Gearman looks like, and more importantly the strange and wonderful things you will encounter in production and how to prepare for them.

F2d82b268a7cbccc9809c939428df64f?s=128

Vranac Srdjan

September 27, 2015
Tweet

Transcript

  1. SHIFTING GEARS WITH GEARMAN Srdjan Vranac / code4hire / @vranac

  2. business owner, developer, consultant, mercenary, writing terrible code that performs

    exceptionally, wrangling elePHPants and Pythons, obsessed with process automation, interested in continuous integration and delivery, clean code, testing, best practices and distributed systems
  3. WARNING!

  4. STORY TIME!

  5. 30.000.000 Products

  6. 30.000.000 Products / 12 Hours = 2.500.000 Products / 3600

    Seconds = 695 Products
  7. 30.000.000 Products / 12 Hours = 2.500.000 Products / 3600

    Seconds = 695 Products 600 nodes = 1.1 Sec. per product
  8. None
  9. ASYNC PROCESSING

  10. MESSAGING

  11. QUEUES

  12. GEARMAN

  13. None
  14. GEARMAN OVERVIEW

  15. GEARMAN SETUP SERVER: • RHEL/Fedora: Run yum install gearmand •

    Debain/Ubuntu Package: Run apt-get install gearman-job-server. EXTENSION: pecl -d preferred_state=stable install -f gearman
  16. TASKS VS JOBS • Task
 A task is any request

    or communication between the client and the job server. • Jobs
 A job is something the worker does, continuously waiting on the job server to tell him when to start and with what arguments.
  17. THE MOST IMPORTANT PIECE OF CODE IN GEARMAN PROJECT //

    Wait 2 hours for a job, then exit (upstart will restart process)
 $worker->setTimeout(7200000);
 
 // start the worker
 while ($worker->work() || $worker->returnCode() == GEARMAN_TIMEOUT) {
 if($worker->returnCode() == GEARMAN_TIMEOUT) {
 print “No jobs available" . PHP_EOL;
 exit(1);
 }
 }
  18. GEARMAN DOCS

  19. DAEMONS EVERYWHERE…

  20. SUPERVISOR UPSTART

  21. THE END

  22. FUN IS ABOUT TO START

  23. RUNNING ORDER • Gearman queue is FIFO (first in, first

    out) • Multiple workers mean no reliable order of execution • Order of execution should not matter
  24. RELIABILITY • Job that is enqueued will be processed… Sometime

    • When job fails… Re-queue if needed • Queue systems are unreliable by design
  25. DATA

  26. LOCATION

  27. DE-NORMALIZATION

  28. None
  29. EVENTUAL CONSISTENCY • Data will be available across all the

    nodes… Eventually • Most projects can support it • High priority changes should be handled separately
  30. RACE CONDITIONS

  31. ATOMIC OPERATIONS

  32. WHAT IS GOING ON?

  33. THINGS FAIL ALL THE TIME

  34. DEBUGGING IS HARD

  35. MONITORING

  36. COUNT EVERYTHING

  37. WHEN DO YOU HAVE ENOUGH METRICS? N E V E

    R !
  38. ANALYSIS

  39. THANK YOU!

  40. LINKS • Gearman: http://gearman.org/ • Supervisor: http://supervisord.org • Upstart: http://upstart.ubuntu.com/

    • Measure Anything, Measure Everything: http:// codeascraft.com/2011/02/15/measure-anything- measure-everything