Shifting Gears With Gearman

Everyone starts with a simple one-machine setup, running PHP(or whatever else you fancy), MySQL/PostgreSql and Apache/Nginx. Sooner or later that will not be enough, and you will be faced with some important architecture decisions. This talk will show you how you can optimize and increase the performance of your application by distributing tasks in an simple and affordable way.

Transcript

1. Shifting Gears With
   Gearman
   Srdjan Vranac // code4hire.com // @vranac
   

   View Slide

2. business owner,
   developer
   , consultant, mercenary, writing
   terrible code that performs exceptionally, wrangling ele
   PHP
   ants
   and
   Python
   s, obsessed with
   process automation
   , interested in
   continuous integration and delivery, clean code, testing, best
   practices and distributed systems
   

   View Slide

3. Just Scrape It!
   30.000.000 Products
   / 12 Hours = 2.500.000 Products
   / 3600 Seconds = 695 Products
   600 nodes = 1.1 Sec. per product
   

   View Slide

4. Just Resize It!
   

   View Slide

5. Asynchronous Processing
   

   View Slide

6. Messaging And Queues
   Process 4
   Process 1 Process 3
   Process 2
   Communication between Processess
   

   View Slide

7. Gearman
   Manager
   

   View Slide

8. Gearman
   Gearman Job
   Server
   Worker 1
   Worker 2
   Worker 3
   Worker ...
   Web Server
   

   View Slide

9. No Running Order Or Reliability
   

   View Slide

10. Data
    Web Server
    Db
    Gearman Job
    Server
    Worker 1
    Worker 2
    Worker 3
    Worker ...
    

    View Slide

11. Location
    

    View Slide

12. De-Normalization
    

    View Slide

13. CAP Theorem
    Availability
    Partition
    Tolerance
    Consistency
    CA
    AP
    CP
    

    View Slide

14. Eventual Consistency
    

    View Slide

15. Atomic Operations
    

    View Slide

16. What Is Going On?
    

    View Slide

17. Debbugging Distributed Systems is Hard!
    

    View Slide

18. Monitoring
    

    View Slide

19. Count Everything
    

    View Slide

20. Uptime, Vmstat, Netstat, Iostat, Mpstat,
    Tcpdump, Gdb, Valgrind, Ulimit, Sysctl
    etc...
    

    View Slide

21. THE END ?
    

    View Slide

22. Links
    , ,
    Gearman Supervisor StatsD
    Measure Anything, Measure Everything
    

    View Slide