Shifting gears with Gearman

Everyone starts with a simple one-machine setup, running PHP(or whatever else you fancy), MySQL/PostgreSql and Apache/Nginx. Sooner or later that will not be enough, you will have more operations to do than can be done in a single request, or you will have some operations that need to be done in parallel, and you will be faced with some important architecture decisions.
This talk will show you how a distributed system built with Gearman looks like, and more importantly the strange and wonderful things you will encounter in production and how to prepare for them.

Transcript

1. SHIFTING GEARS WITH GEARMAN Srdjan Vranac / code4hire / @vranac
2. business owner, developer, consultant, mercenary, writing terrible code that performs exceptionally, wrangling elePHPants and Pythons, obsessed with process automation, interested in continuous integration and delivery, clean code, testing, best practices and distributed systems
3. WARNING!
4. STORY TIME!
5. 30.000.000 Products
6. 30.000.000 Products / 12 Hours = 2.500.000 Products / 3600 Seconds = 695 Products
7. 30.000.000 Products / 12 Hours = 2.500.000 Products / 3600 Seconds = 695 Products 600 nodes = 1.1 Sec. per product
9. ASYNC PROCESSING
10. MESSAGING
11. QUEUES
12. GEARMAN
14. GEARMAN OVERVIEW
15. GEARMAN SETUP SERVER: • RHEL/Fedora: Run yum install gearmand • Debain/Ubuntu Package: Run apt-get install gearman-job-server. EXTENSION: pecl -d preferred_state=stable install -f gearman
16. TASKS VS JOBS • Task
 A task is any request or communication between the client and the job server. • Jobs
 A job is something the worker does, continuously waiting on the job server to tell him when to start and with what arguments.
17. THE MOST IMPORTANT PIECE OF CODE IN GEARMAN PROJECT // Wait 2 hours for a job, then exit (upstart will restart process)
 $worker->setTimeout(7200000);
 
 // start the worker
 while ($worker->work() || $worker->returnCode() == GEARMAN_TIMEOUT) {
 if($worker->returnCode() == GEARMAN_TIMEOUT) {
 print “No jobs available" . PHP_EOL;
 exit(1);
 }
 }
18. GEARMAN DOCS
19. DAEMONS EVERYWHERE…
20. SUPERVISOR UPSTART
21. THE END
22. FUN IS ABOUT TO START
23. RUNNING ORDER • Gearman queue is FIFO (first in, first out) • Multiple workers mean no reliable order of execution • Order of execution should not matter
24. RELIABILITY • Job that is enqueued will be processed… Sometime • When job fails… Re-queue if needed • Queue systems are unreliable by design
25. DATA
26. LOCATION
27. DE-NORMALIZATION
29. EVENTUAL CONSISTENCY • Data will be available across all the nodes… Eventually • Most projects can support it • High priority changes should be handled separately
30. RACE CONDITIONS
31. ATOMIC OPERATIONS
32. WHAT IS GOING ON?
33. THINGS FAIL ALL THE TIME
34. DEBUGGING IS HARD
35. MONITORING
36. COUNT EVERYTHING
37. WHEN DO YOU HAVE ENOUGH METRICS? N E V E R !
38. ANALYSIS
39. THANK YOU!
40. LINKS • Gearman: http://gearman.org/ • Supervisor: http://supervisord.org • Upstart: http://upstart.ubuntu.com/ • Measure Anything, Measure Everything: http:// codeascraft.com/2011/02/15/measure-anything- measure-everything