$30 off During Our Annual Pro Sale. View Details »

Shifting gears with Gearman

Vranac Srdjan
September 27, 2015

Shifting gears with Gearman

Everyone starts with a simple one-machine setup, running PHP(or whatever else you fancy), MySQL/PostgreSql and Apache/Nginx. Sooner or later that will not be enough, you will have more operations to do than can be done in a single request, or you will have some operations that need to be done in parallel, and you will be faced with some important architecture decisions.
This talk will show you how a distributed system built with Gearman looks like, and more importantly the strange and wonderful things you will encounter in production and how to prepare for them.

Vranac Srdjan

September 27, 2015
Tweet

More Decks by Vranac Srdjan

Other Decks in Programming

Transcript

  1. SHIFTING GEARS WITH
    GEARMAN
    Srdjan Vranac / code4hire / @vranac

    View Slide

  2. business owner, developer, consultant, mercenary,
    writing terrible code that performs exceptionally,
    wrangling elePHPants and Pythons, obsessed with
    process automation, interested in continuous
    integration and delivery, clean code, testing, best
    practices and distributed systems

    View Slide

  3. WARNING!

    View Slide

  4. STORY TIME!

    View Slide

  5. 30.000.000 Products

    View Slide

  6. 30.000.000 Products
    / 12 Hours = 2.500.000 Products
    / 3600 Seconds = 695 Products

    View Slide

  7. 30.000.000 Products
    / 12 Hours = 2.500.000 Products
    / 3600 Seconds = 695 Products
    600 nodes = 1.1 Sec. per product

    View Slide

  8. View Slide

  9. ASYNC PROCESSING

    View Slide

  10. MESSAGING

    View Slide

  11. QUEUES

    View Slide

  12. GEARMAN

    View Slide

  13. View Slide

  14. GEARMAN OVERVIEW

    View Slide

  15. GEARMAN SETUP
    SERVER:
    • RHEL/Fedora: Run yum install gearmand
    • Debain/Ubuntu Package: Run apt-get install gearman-job-server.
    EXTENSION:
    pecl -d preferred_state=stable install -f gearman

    View Slide

  16. TASKS VS JOBS
    • Task

    A task is any request or communication between
    the client and the job server.
    • Jobs

    A job is something the worker does, continuously
    waiting on the job server to tell him when to start
    and with what arguments.

    View Slide

  17. THE MOST IMPORTANT PIECE OF
    CODE IN GEARMAN PROJECT
    // Wait 2 hours for a job, then exit (upstart will restart process)

    $worker->setTimeout(7200000);


    // start the worker

    while ($worker->work() || $worker->returnCode() == GEARMAN_TIMEOUT)
    {

    if($worker->returnCode() == GEARMAN_TIMEOUT) {

    print “No jobs available" . PHP_EOL;

    exit(1);

    }

    }

    View Slide

  18. GEARMAN DOCS

    View Slide

  19. DAEMONS EVERYWHERE…

    View Slide

  20. SUPERVISOR
    UPSTART

    View Slide

  21. THE END

    View Slide

  22. FUN IS ABOUT TO START

    View Slide

  23. RUNNING ORDER
    • Gearman queue is FIFO (first in, first out)
    • Multiple workers mean no reliable order of execution
    • Order of execution should not matter

    View Slide

  24. RELIABILITY
    • Job that is enqueued will be processed… Sometime
    • When job fails… Re-queue if needed
    • Queue systems are unreliable by design

    View Slide

  25. DATA

    View Slide

  26. LOCATION

    View Slide

  27. DE-NORMALIZATION

    View Slide

  28. View Slide

  29. EVENTUAL CONSISTENCY
    • Data will be available across all the nodes… Eventually
    • Most projects can support it
    • High priority changes should be handled separately

    View Slide

  30. RACE CONDITIONS

    View Slide

  31. ATOMIC OPERATIONS

    View Slide

  32. WHAT IS GOING ON?

    View Slide

  33. THINGS FAIL ALL THE TIME

    View Slide

  34. DEBUGGING IS HARD

    View Slide

  35. MONITORING

    View Slide

  36. COUNT EVERYTHING

    View Slide

  37. WHEN DO YOU HAVE
    ENOUGH METRICS?
    N E V E R !

    View Slide

  38. ANALYSIS

    View Slide

  39. THANK YOU!

    View Slide

  40. LINKS
    • Gearman: http://gearman.org/
    • Supervisor: http://supervisord.org
    • Upstart: http://upstart.ubuntu.com/
    • Measure Anything, Measure Everything: http://
    codeascraft.com/2011/02/15/measure-anything-
    measure-everything

    View Slide