SpreeConf 2014 - Ninefold Presents: Spree Performance

Spree commerce is capable of sizzling performance and smooth scalability. But why does performance matter so much? Because time is money. Amazon found that every 100ms of average responsiveness improvement boosted their revenue by 1%. AOL, Google, Yahoo! and Shopzilla have found similar or greater effects. With this backdrop, Daniel Sim Lind leads us through the performance testing journey that Rails hosting platform provider Ninefold took with Spree. We'll deep dive into the methodology and tools, the analysis, using user happiness as a testing constant, the results - and what it all means for you. Includes hot-off-the-press Spree 2-2 with cacheing test results, and a cheeky Magento comparison.

Transcript

1. Performance.
2. Hi. I’m Daniel.
14. How you measure things makes a big difference.
15. But first…
16. Why should you care about performance?
17. 1. Do it for the users.
18. 2. Do it for the money.
19. So, users, what do they even want?
20. Google asked.
21. “We want more results.”
22. More results were 500ms slower. 20% drop in traffic and revenue.
23. Bad performance can ruin your UX.
24. What does good even look like?
25. Google again.
26. “Mate! We’re much quicker than that!”
27. Good. Only problem is…
28. The mean is a lie.
29. Stats 101: The Bell Curve.
30. The real world.
32. So, on to the testing…
33. 1. Set up a server. 2. Design a simulation. 3. Throw simulated users at the server. 4. Measure the latency.
34. Tools.
35. JMeter.
36. JMeter.
37. JMeter.
38. JMeter.
39. Don’t cry. There’s ruby-jmeter.
41. Here’s our tool chain: Chrome -> Developer tools provide data for the test plan. ! ruby-jmeter -> Provides ruby-based testing DSL. -> Generates JMeter XML. ! Flood.io -> Eats JMeter XML. -> Spin up a “Grid” of simulated users following the XML-defined script. -> Measures and reports response times.
42. Capture click data:
43. Our test plan:
44. Our test plan:
45. Our test plan:
46. Running the test at scale.
47. Flood.io Requires some understanding of JMeter. Scales as big as you want to pay for. Gives high-level and very granular results.
48. Flood.io
49. Apdex. t = acceptable response time satisfied = response time < t tolerating = t < response time < 4t frustrated = response time > 4t Apdex [t] = (satisfied + (tolerating/2))/total requests
50. Happiness as a constant. Apdex 0.95 [0.75] Benchmarked against Heroku. Higher threshold than we would have liked.
51. Heroku details: • WEB_CONCURRENCY=7 
 (based on a series of tests and analysis of memory usage) ! • Use 2X sized dynos to ensure more memory and CPU resources ! • Postgres database : Crane configuration
52. Test parameters: • Choose 5 configurations that range from $100 per month to $1,000 per month.
 • Run flood.io tests using a 60 second ramp up and a 300 second test duration.
 • Vary the concurrent user count to discover the maximum number of users that can be serviced while maintaining an Apdex 0.95 [0.75]
53. The results:
54. $0.00$ $100.00$ $200.00$ $300.00$ $400.00$ $500.00$ $600.00$ $700.00$ $800.00$ $900.00$ $1,000.00$ $1,100.00$ $1,200.00$ 0$ 25$ 50$ 75$ 100$ 125$ 150$ 175$ 200$ 225$ 250$ 275$ 300$ 325$ 350$ 375$ 400$ 425$ 450$ 475$ 500$ 525$ 550$ 575$ 600$ 625$ Monthly(cost(for(test(configura2on( Maximum(concurrent(users(whilst(maintaining(Apdex(of(at(least(0.95(( How(much(does(it(cost(to(delight(my(customers?( Heroku$2X$ Ninefold$ Smooth scaling.
55. $0.00$ $100.00$ $200.00$ $300.00$ $400.00$ $500.00$ $600.00$ $700.00$ $800.00$ $900.00$ $1,000.00$ $1,100.00$ $1,200.00$ $1,300.00$ $1,400.00$ $1,500.00$ $1,600.00$ $1,700.00$ $1,800.00$ $1,900.00$ $2,000.00$ $2,100.00$ $2,200.00$ $2,300.00$ $2,400.00$ $2,500.00$ 0$ 25$ 50$ 75$ 100$ 125$ 150$ 175$ 200$ 225$ 250$ 275$ 300$ 325$ 350$ 375$ 400$ 425$ 450$ 475$ 500$ 525$ 550$ 575$ 600$ 625$ 650$ 675$ 700$ 725$ 750$ 775$ 800$ Monthly(cost(for(test(configura2on( Maximum(concurrent(users(whilst(maintaining(Apdex(of(at(least(0.95(( How(much(does(it(cost(to(delight(my(customers?( Ninefold$ Heroku$PX$ Smooth pricing.
56. Consistent performance.
57. 1 in 20 clicks. Meet the 95th percentile.
61. Why does it even matter?
62. Performance and engagement.
63. Visitors experiencing the best 10% of load times viewed 3 times more pages than getting the worst 10%. http://assets.en.oreilly.com/1/event/29/The%20Secret%20Weapons%20of%20the%20AOL%20Optimization%20Team%20Presentation.pdf
64. Improved load time by 400ms. Increased traffic by 9%. http://www.slideshare.net/stoyan/yslow-20-presentation
65. Time is money.
66. 2. Do it for the money.
67. Load time: 6 seconds -> 1.2 seconds. Page views: up 25%. Revenue: up 12%. http://www.slideshare.net/timmorrow/shopzilla-performance-by-design-2433735
68. For every 100ms of load time improvement: Sales increased by 1%. https://sites.google.com/site/glinden/Home/StanfordDataMining.2006-11-28.ppt?attredirects=0
69. Where should I host?
70. With Ninefold, of course… ! {Sales Pitch}
71. 1. Consistent performance. 2. Inexpensive. a) Bare VM pricing. b) Much lower sysadmin costs. Ninefold: 
 The platform without the penalty.
72. 3. Smoothly scaling costs. 4. Easy scaling. 5. Just enough control. Ninefold: 
 The platform without the penalty.
73. Remember young grasshopper: Your focus determines your power.
74. The YAQ Program.
75. 1. Clone your store. 2. Design a simulation. 3. Run the simulation at scale. 4. Benchmark the results. 10 Spree store tests.
76. daniel@ninefold.com @danielsimlind ! https://github.com/ninefold ! ninefold.com @ninefold Please chat with us 
 and grab a t-shirt.
77. Phew. Made it. Thanks.