Shifting Gears With Gearman

Everyone starts with a simple one machine setup, running PHP, MySQL and Apache(or Nginx). Sooner or later some tasks will start taking too long and your application will start to slow down or fail, or your infractstructure will not be enough.
This talk will show you how you can optimize and increase the performance of your application by distributing tasks and what problems you will encounter, and how to possibly mitigate them

Transcript

1. SHIFTING
   GEARS WITH
   GEARMAN
   Srdjan Vranac // code4hire.com // @vranac
   0
   

   View Slide

2. JUST SCRAPE IT!
   30.000.000 PRODUCTS
   / 12 HOURS = 2.500.000 PRODUCTS
   / 3600 SECONDS = 695 PRODUCTS
   600 NODES = 1.1 SEC. PER PRODUCT
   

   View Slide

3. JUST RESIZE IT!
   

   View Slide

4. ASYNCHRONOUS
   PROCESSING
   

   View Slide

5. MESSAGING AND QUEUES
   Process 4
   Process 1 Process 3
   Process 2
   COMMUNICATION BETWEEN
   PROCESSESS
   

   View Slide

6. GEARMAN
   MANAGER
   

   View Slide

7. GEARMAN
   Gearman Job
   Server
   Worker 1
   Worker 2
   Worker 3
   Worker ...
   Web Server
   

   View Slide

8. NO RUNNING ORDER OR
   RELIABILITY
   

   View Slide

9. DATA
   Web Server
   Db
   Gearman Job
   Server
   Worker 1
   Worker 2
   Worker 3
   Worker ...
   

   View Slide

10. LOCATION
    DE-NORMALIZATION
    

    View Slide

11. CAP THEOREM
    Availability
    Partition
    Tolerance
    Consistency
    CA
    AP
    CP
    

    View Slide

12. EVENTUAL CONSISTENCY
    

    View Slide

13. ATOMIC OPERATIONS
    

    View Slide

14. WHAT IS GOING ON?
    

    View Slide

15. DEBBUGGING DISTRIBUTED
    SYSTEMS IS HARD!
    

    View Slide

16. MONITORING
    

    View Slide

17. COUNT EVERYTHING
    

    View Slide

18. UPTIME, VMSTAT, NETSTAT,
    IOSTAT, MPSTAT, TCPDUMP,
    GDB, VALGRIND, ULIMIT,
    SYSCTL ETC...
    

    View Slide

19. THE END ?
    

    View Slide

20. LINKS
    , ,
    Gearman Supervisor StatsD
    Measure Anything, Measure Everything
    

    View Slide