Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SpreeConf 2014 - Ninefold Presents: Spree Performance

SpreeConf 2014 - Ninefold Presents: Spree Performance

Spree commerce is capable of sizzling performance and smooth scalability. But why does performance matter so much? Because time is money. Amazon found that every 100ms of average responsiveness improvement boosted their revenue by 1%. AOL, Google, Yahoo! and Shopzilla have found similar or greater effects. With this backdrop, Daniel Sim Lind leads us through the performance testing journey that Rails hosting platform provider Ninefold took with Spree. We'll deep dive into the methodology and tools, the analysis, using user happiness as a testing constant, the results - and what it all means for you. Includes hot-off-the-press Spree 2-2 with cacheing test results, and a cheeky Magento comparison.

B5cb385936d4d68da2e24e5a5819932b?s=128

Daniel Sim Lind

February 26, 2014
Tweet

Other Decks in Technology

Transcript

  1. Performance.

  2. Hi. I’m Daniel.

  3. None
  4. None
  5. None
  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. How you measure things makes a big difference.

  15. But first…

  16. Why should you care about performance?

  17. 1. Do it for the users.

  18. 2. Do it for the money.

  19. So, users, what do they even want?

  20. Google asked.

  21. “We want more results.”

  22. More results were 500ms slower. 20% drop in traffic and

    revenue.
  23. Bad performance can ruin your UX.

  24. What does good even look like?

  25. Google again.

  26. “Mate! We’re much quicker than that!”

  27. Good. Only problem is…

  28. The mean is a lie.

  29. Stats 101: The Bell Curve.

  30. The real world.

  31. None
  32. So, on to the testing…

  33. 1. Set up a server. 2. Design a simulation. 3.

    Throw simulated users at the server. 4. Measure the latency.
  34. Tools.

  35. JMeter.

  36. JMeter.

  37. JMeter.

  38. JMeter.

  39. Don’t cry. There’s ruby-jmeter.

  40. None
  41. Here’s our tool chain: Chrome -> Developer tools provide data

    for the test plan. ! ruby-jmeter -> Provides ruby-based testing DSL. -> Generates JMeter XML. ! Flood.io -> Eats JMeter XML. -> Spin up a “Grid” of simulated users following the XML-defined script. -> Measures and reports response times.
  42. Capture click data:

  43. Our test plan:

  44. Our test plan:

  45. Our test plan:

  46. Running the test at scale.

  47. Flood.io Requires some understanding of JMeter. Scales as big as

    you want to pay for. Gives high-level and very granular results.
  48. Flood.io

  49. Apdex. t = acceptable response time satisfied = response time

    < t tolerating = t < response time < 4t frustrated = response time > 4t Apdex [t] = (satisfied + (tolerating/2))/total requests
  50. Happiness as a constant. Apdex 0.95 [0.75] Benchmarked against Heroku.

    Higher threshold than we would have liked.
  51. Heroku details: • WEB_CONCURRENCY=7 
 (based on a series of

    tests and analysis of memory usage) ! • Use 2X sized dynos to ensure more memory and CPU resources ! • Postgres database : Crane configuration
  52. Test parameters: • Choose 5 configurations that range from $100

    per month to $1,000 per month.
 • Run flood.io tests using a 60 second ramp up and a 300 second test duration.
 • Vary the concurrent user count to discover the maximum number of users that can be serviced while maintaining an Apdex 0.95 [0.75]
  53. The results:

  54. $0.00$ $100.00$ $200.00$ $300.00$ $400.00$ $500.00$ $600.00$ $700.00$ $800.00$ $900.00$

    $1,000.00$ $1,100.00$ $1,200.00$ 0$ 25$ 50$ 75$ 100$ 125$ 150$ 175$ 200$ 225$ 250$ 275$ 300$ 325$ 350$ 375$ 400$ 425$ 450$ 475$ 500$ 525$ 550$ 575$ 600$ 625$ Monthly(cost(for(test(configura2on( Maximum(concurrent(users(whilst(maintaining(Apdex(of(at(least(0.95(( How(much(does(it(cost(to(delight(my(customers?( Heroku$2X$ Ninefold$ Smooth scaling.
  55. $0.00$ $100.00$ $200.00$ $300.00$ $400.00$ $500.00$ $600.00$ $700.00$ $800.00$ $900.00$

    $1,000.00$ $1,100.00$ $1,200.00$ $1,300.00$ $1,400.00$ $1,500.00$ $1,600.00$ $1,700.00$ $1,800.00$ $1,900.00$ $2,000.00$ $2,100.00$ $2,200.00$ $2,300.00$ $2,400.00$ $2,500.00$ 0$ 25$ 50$ 75$ 100$ 125$ 150$ 175$ 200$ 225$ 250$ 275$ 300$ 325$ 350$ 375$ 400$ 425$ 450$ 475$ 500$ 525$ 550$ 575$ 600$ 625$ 650$ 675$ 700$ 725$ 750$ 775$ 800$ Monthly(cost(for(test(configura2on( Maximum(concurrent(users(whilst(maintaining(Apdex(of(at(least(0.95(( How(much(does(it(cost(to(delight(my(customers?( Ninefold$ Heroku$PX$ Smooth pricing.
  56. Consistent performance.

  57. 1 in 20 clicks. Meet the 95th percentile.

  58. None
  59. None
  60. None
  61. Why does it even matter?

  62. Performance and engagement.

  63. Visitors experiencing the best 10% of load times viewed 3

    times more pages than getting the worst 10%. http://assets.en.oreilly.com/1/event/29/The%20Secret%20Weapons%20of%20the%20AOL%20Optimization%20Team%20Presentation.pdf
  64. Improved load time by 400ms. Increased traffic by 9%. http://www.slideshare.net/stoyan/yslow-20-presentation

  65. Time is money.

  66. 2. Do it for the money.

  67. Load time: 6 seconds -> 1.2 seconds. Page views: up

    25%. Revenue: up 12%. http://www.slideshare.net/timmorrow/shopzilla-performance-by-design-2433735
  68. For every 100ms of load time improvement: Sales increased by

    1%. https://sites.google.com/site/glinden/Home/StanfordDataMining.2006-11-28.ppt?attredirects=0
  69. Where should I host?

  70. With Ninefold, of course… ! {Sales Pitch}

  71. 1. Consistent performance. 2. Inexpensive. a) Bare VM pricing. b)

    Much lower sysadmin costs. Ninefold: 
 The platform without the penalty.
  72. 3. Smoothly scaling costs. 4. Easy scaling. 5. Just enough

    control. Ninefold: 
 The platform without the penalty.
  73. Remember young grasshopper: Your focus determines your power.

  74. The YAQ Program.

  75. 1. Clone your store. 2. Design a simulation. 3. Run

    the simulation at scale. 4. Benchmark the results. 10 Spree store tests.
  76. daniel@ninefold.com @danielsimlind ! https://github.com/ninefold ! ninefold.com @ninefold Please chat with

    us 
 and grab a t-shirt.
  77. Phew. Made it. Thanks.