Distribute the Workload. PHP UK 2012, day 1

Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow.
By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our service by using systems such as
- load balancers
- message queues
- gearman
- daemons
and a few others, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily.

During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable pieces.

Transcript

1. Distribute the workload Helgi Þormar Þorbjörnsson PHP UK, London, 24th of Feb. 2012 Monday, 27 February 12
2. Co-founded Orchestra.io Work at EngineYard PEAR Developer From Iceland @h on Twitter Helgi Monday, 27 February 12
3. Why Distribute? Budget Efficiency Perception Monday, 27 February 12
4. Efficiency 10 small servers > 1 big Monday, 27 February 12
5. Budget Spend wisely Commodity servers Cloud Computing (EC2) Monday, 27 February 12
6. Perception Defer intensive processes Give instant feedback Users keep on browsing Monday, 27 February 12
7. Monday, 27 February 12
8. Ant Colonies Monday, 27 February 12
9. Teamwork When faced with a problem they will solve the problem as one. Monday, 27 February 12
10. Monday, 27 February 12
11. Monday, 27 February 12
12. Architect for Distribution Monday, 27 February 12
13. Characteristics Decoupling Elasticity High Availability Concurrency Monday, 27 February 12
14. Decoupling Monday, 27 February 12
15. Application DB API Cache FE Monday, 27 February 12
16. Application DB API Cache FE Cache API API Monday, 27 February 12
17. Elasticity Monday, 27 February 12
18. Cloud Computing Monday, 27 February 12
19. Load Balance Monday, 27 February 12
20. HA Proxy Nginx My Favourite Monday, 27 February 12
21. Monitoring Monday, 27 February 12
22. When do I need more servers? Monday, 27 February 12
23. Needs to be around from the start! Monday, 27 February 12
24. Keep records Monday, 27 February 12
25. Spot trends Monday, 27 February 12
26. Different types Hardware Performance Software Performance Availability Resourcing Monday, 27 February 12
27. Applications New Relic CloudKick ScoutApp Nagios Cacti Circonus Monday, 27 February 12
28. Automation Monday, 27 February 12
29. Plug into your monitoring Monday, 27 February 12
30. Bringing together Monitoring and Elastic behaviour into one beautiful whole! Monday, 27 February 12
31. Add some intelligence to add / remove servers as needed based on current information. Monday, 27 February 12
32. Just make sure it doesn’t turn into... Monday, 27 February 12
33. Skynet!! Monday, 27 February 12
34. High Availability Monday, 27 February 12
35. Get a highly available and resilient setup by following a few of those recommendations Monday, 27 February 12
36. Remember, even Google has outages Monday, 27 February 12
37. What to avoid Monday, 27 February 12
38. Local Sessions Monday, 27 February 12
39. Store sessions in DB / Memcache Solution Monday, 27 February 12
40. Local Memory Monday, 27 February 12
41. Networked Memcache Solution Monday, 27 February 12
42. Local Files Monday, 27 February 12
43. Local Uploads Monday, 27 February 12
44. Writing to /tmp Monday, 27 February 12
45. Store on S3 or a networked FS Solution Monday, 27 February 12
46. Serve up static files from CDNs Solution Monday, 27 February 12
47. Servers can vanish at any given time Monday, 27 February 12
48. Internal APIs Monday, 27 February 12
49. Application S3 GFS FS Internal Storage API Monday, 27 February 12
50. Application MySQL Mongo Cache Internal DB API Monday, 27 February 12
51. Gearman Monday, 27 February 12
52. Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Gearman Job Server gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker Code Your App Gearman Monday, 27 February 12
53. pear.php.net/ net_gearman Monday, 27 February 12
54. A Story! Monday, 27 February 12
55. Financial Software Monday, 27 February 12
56. 3000+ Clients Monday, 27 February 12
57. Each one has 5 external data sources Monday, 27 February 12
58. Each data source is a web service Monday, 27 February 12
59. Ran every 6 hours every day Monday, 27 February 12
60. Cron Gearman Job 1 1 2 3 4 5 Web Services 1 4 3 2 5 Processing Monday, 27 February 12
61. But! That wasn’t enough Monday, 27 February 12
62. Job kicked off on login Monday, 27 February 12
63. Supervisord Monday, 27 February 12
64. phpadvent.org/2009/daemonize- your-php-by-sean-coates Monday, 27 February 12
65. Queue Systems Monday, 27 February 12
66. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation Search Monday, 27 February 12
67. Common Tools Gearman Hadoop ZeroMQ RabbitMQ And many others! Monday, 27 February 12
68. Questions? Joind.in: https://joind.in/4958 @h helgi@engineyard.com Monday, 27 February 12