Shifting gears with Gearman

Everyone starts with a simple one-machine setup, running PHP(or whatever else you fancy), MySQL/PostgreSql and Apache/Nginx. Sooner or later that will not be enough, you will have more operations to do than can be done in a single request, or you will have some operations that need to be done in parallel, and you will be faced with some important architecture decisions.
This talk will show you how a distributed system built with Gearman looks like, and more importantly the strange and wonderful things you will encounter in production and how to prepare for them.

Transcript

1. SHIFTING GEARS WITH
   GEARMAN
   Srdjan Vranac / code4hire / @vranac
   

   View Slide

2. business owner, developer, consultant, mercenary,
   writing terrible code that performs exceptionally,
   wrangling elePHPants and Pythons, obsessed with
   process automation, interested in continuous
   integration and delivery, clean code, testing, best
   practices and distributed systems
   

   View Slide

3. WARNING!
   

   View Slide

4. STORY TIME!
   

   View Slide

5. 30.000.000 Products
   

   View Slide

6. 30.000.000 Products
   / 12 Hours = 2.500.000 Products
   / 3600 Seconds = 695 Products
   

   View Slide

7. 30.000.000 Products
   / 12 Hours = 2.500.000 Products
   / 3600 Seconds = 695 Products
   600 nodes = 1.1 Sec. per product
   

   View Slide

8. View Slide

9. ASYNC PROCESSING
   

   View Slide

10. MESSAGING
    

    View Slide

11. QUEUES
    

    View Slide

12. GEARMAN
    

    View Slide

13. View Slide

14. GEARMAN OVERVIEW
    

    View Slide

15. GEARMAN SETUP
    SERVER:
    • RHEL/Fedora: Run yum install gearmand
    • Debain/Ubuntu Package: Run apt-get install gearman-job-server.
    EXTENSION:
    pecl -d preferred_state=stable install -f gearman
    

    View Slide

16. TASKS VS JOBS
    • Task

    A task is any request or communication between
    the client and the job server.
    • Jobs

    A job is something the worker does, continuously
    waiting on the job server to tell him when to start
    and with what arguments.
    

    View Slide

17. THE MOST IMPORTANT PIECE OF
    CODE IN GEARMAN PROJECT
    // Wait 2 hours for a job, then exit (upstart will restart process)

    $worker->setTimeout(7200000);

    

    // start the worker

    while ($worker->work() || $worker->returnCode() == GEARMAN_TIMEOUT)
    {

    if($worker->returnCode() == GEARMAN_TIMEOUT) {

    print “No jobs available" . PHP_EOL;

    exit(1);

    }

    }
    

    View Slide

18. GEARMAN DOCS
    

    View Slide

19. DAEMONS EVERYWHERE…
    

    View Slide

20. SUPERVISOR
    UPSTART
    

    View Slide

21. THE END
    

    View Slide

22. FUN IS ABOUT TO START
    

    View Slide

23. RUNNING ORDER
    • Gearman queue is FIFO (first in, first out)
    • Multiple workers mean no reliable order of execution
    • Order of execution should not matter
    

    View Slide

24. RELIABILITY
    • Job that is enqueued will be processed… Sometime
    • When job fails… Re-queue if needed
    • Queue systems are unreliable by design
    

    View Slide

25. DATA
    

    View Slide

26. LOCATION
    

    View Slide

27. DE-NORMALIZATION
    

    View Slide

28. View Slide

29. EVENTUAL CONSISTENCY
    • Data will be available across all the nodes… Eventually
    • Most projects can support it
    • High priority changes should be handled separately
    

    View Slide

30. RACE CONDITIONS
    

    View Slide

31. ATOMIC OPERATIONS
    

    View Slide

32. WHAT IS GOING ON?
    

    View Slide

33. THINGS FAIL ALL THE TIME
    

    View Slide

34. DEBUGGING IS HARD
    

    View Slide

35. MONITORING
    

    View Slide

36. COUNT EVERYTHING
    

    View Slide

37. WHEN DO YOU HAVE
    ENOUGH METRICS?
    N E V E R !
    

    View Slide

38. ANALYSIS
    

    View Slide

39. THANK YOU!
    

    View Slide

40. LINKS
    • Gearman: http://gearman.org/
    • Supervisor: http://supervisord.org
    • Upstart: http://upstart.ubuntu.com/
    • Measure Anything, Measure Everything: http://
    codeascraft.com/2011/02/15/measure-anything-
    measure-everything
    

    View Slide