Tales from the Ops side - Taking Black Friday

a.k.a. Scaling Magento, this is the slide deck for a #DevTO talk performed by Hany Fahim on Monday, January 26, 2015.

1. Tales from the Ops Side Taking Black Friday (a.k.a. scaling Magento) Hany Fahim hany@vmfarms.com @iHandroid vmfarms.com
2. The Problem A recent customer that operates a large successful Magento-based e-commerce site and was preparing for a large traffic bump for the upcoming Black Friday. Magento is not easy to scale.
3. The Challenge: ! Get it to scale. No downtime.
4. 3 ways to scale 1. Scaling up (a.k.a. upgrading). 2. Scaling out (a.k.a. horizontal scaling). 3. Scaling out the hard way.
5. Scaling Up • Basically more of what you have now. • More memory, CPU, faster disks. • Not many options.
6. Scaling Out (the easy way) Step 1: Turn on all the caches.
7. Scaling Out (the easy way) Step 2: Split up services.
8. Scaling Out (the easy way) Step 3: Add more servers and load balance.
9. Scaling Out (the easy way) Step 3a: Use shared storage. Note: Magento requires it’s image cache to be in sync with all servers.
10. Scaling Out (the easy way) Not recommended for production.
11. Scaling Out (the easy way) Usually failures are catastrophic.
12. Scaling Out (the hard way) ?
13. Scaling Out (the hard way) Use replication, but avoid this craziness.
14. Scaling Out (the hard way) Using lsyncd, create a ring-sync.
15. Scaling Out (the hard way) With LUA, add failover logic to skip over failed nodes.
16. Scaling Out (the hard way) Use sticky sessions to ensure a user hits the same server.
17. Success?
18. 404 Not Found The requested URL was not found on this server. ! Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
19. Foiled by the CDN
20. CloudFront caches 404s! + 404!
21. Just need to retry 404! 200! + +
22. Just need to retry H A P R O X Y X X
23. Took 7 lines sub vcl_fetch {! if (req.url ~ "^/cache/.*") {! if (beresp.status == 404) {! return(restart);! }! }! }
24. Restart until 200! + + 404 404 200!
25. The Result? Customer experienced the biggest Black Friday weekend ever. ! No downtime. No drop in performance.
26. Hany Fahim hany@vmfarms.com @iHandroid vmfarms.com Questions? psst… we’re hiring