SpreeConf 2014 - Ninefold Presents: Spree Performance

Slide 1

Slide 1 text

Performance.

Slide 2

Slide 2 text

Hi. I’m Daniel.

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

How you measure things makes a big difference.

Slide 15

Slide 15 text

But ﬁrst…

Slide 16

Slide 16 text

Why should you care about performance?

Slide 17

Slide 17 text

1. Do it for the users.

Slide 18

Slide 18 text

2. Do it for the money.

Slide 19

Slide 19 text

So, users, what do they even want?

Slide 20

Slide 20 text

Google asked.

Slide 21

Slide 21 text

“We want more results.”

Slide 22

Slide 22 text

More results were 500ms slower. 20% drop in trafﬁc and revenue.

Slide 23

Slide 23 text

Bad performance can ruin your UX.

Slide 24

Slide 24 text

What does good even look like?

Slide 25

Slide 25 text

Google again.

Slide 26

Slide 26 text

“Mate! We’re much quicker than that!”

Slide 27

Slide 27 text

Good. Only problem is…

Slide 28

Slide 28 text

The mean is a lie.

Slide 29

Slide 29 text

Stats 101: The Bell Curve.

Slide 30

Slide 30 text

The real world.

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

So, on to the testing…

Slide 33

Slide 33 text

1. Set up a server. 2. Design a simulation. 3. Throw simulated users at the server. 4. Measure the latency.

Slide 34

Slide 34 text

Tools.

Slide 35

Slide 35 text

JMeter.

Slide 36

Slide 36 text

JMeter.

Slide 37

Slide 37 text

JMeter.

Slide 38

Slide 38 text

JMeter.

Slide 39

Slide 39 text

Don’t cry. There’s ruby-jmeter.

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Here’s our tool chain: Chrome -> Developer tools provide data for the test plan. ! ruby-jmeter -> Provides ruby-based testing DSL. -> Generates JMeter XML. ! Flood.io -> Eats JMeter XML. -> Spin up a “Grid” of simulated users following the XML-deﬁned script. -> Measures and reports response times.

Slide 42

Slide 42 text

Capture click data:

Slide 43

Slide 43 text

Our test plan:

Slide 44

Slide 44 text

Our test plan:

Slide 45

Slide 45 text

Our test plan:

Slide 46

Slide 46 text

Running the test at scale.

Slide 47

Slide 47 text

Flood.io Requires some understanding of JMeter. Scales as big as you want to pay for. Gives high-level and very granular results.

Slide 48

Slide 48 text

Flood.io

Slide 49

Slide 49 text

Apdex. t = acceptable response time satisﬁed = response time < t tolerating = t < response time < 4t frustrated = response time > 4t Apdex [t] = (satisﬁed + (tolerating/2))/total requests

Slide 50

Slide 50 text

Happiness as a constant. Apdex 0.95 [0.75] Benchmarked against Heroku. Higher threshold than we would have liked.

Slide 51

Slide 51 text

Heroku details: • WEB_CONCURRENCY=7   (based on a series of tests and analysis of memory usage) ! • Use 2X sized dynos to ensure more memory and CPU resources ! • Postgres database : Crane conﬁguration

Slide 52

Slide 52 text

Test parameters: • Choose 5 conﬁgurations that range from $100 per month to $1,000 per month.  • Run ﬂood.io tests using a 60 second ramp up and a 300 second test duration.  • Vary the concurrent user count to discover the maximum number of users that can be serviced while maintaining an Apdex 0.95 [0.75]

Slide 53

Slide 53 text

The results:

Slide 54

Slide 54 text

$0.00$ $100.00$ $200.00$ $300.00$ $400.00$ $500.00$ $600.00$ $700.00$ $800.00$ $900.00$ $1,000.00$ $1,100.00$ $1,200.00$ 0$ 25$ 50$ 75$ 100$ 125$ 150$ 175$ 200$ 225$ 250$ 275$ 300$ 325$ 350$ 375$ 400$ 425$ 450$ 475$ 500$ 525$ 550$ 575$ 600$ 625$ Monthly(cost(for(test(conﬁgura2on( Maximum(concurrent(users(whilst(maintaining(Apdex(of(at(least(0.95(( How(much(does(it(cost(to(delight(my(customers?( Heroku$2X$ Ninefold$ Smooth scaling.

Slide 55

Slide 55 text

$0.00$ $100.00$ $200.00$ $300.00$ $400.00$ $500.00$ $600.00$ $700.00$ $800.00$ $900.00$ $1,000.00$ $1,100.00$ $1,200.00$ $1,300.00$ $1,400.00$ $1,500.00$ $1,600.00$ $1,700.00$ $1,800.00$ $1,900.00$ $2,000.00$ $2,100.00$ $2,200.00$ $2,300.00$ $2,400.00$ $2,500.00$ 0$ 25$ 50$ 75$ 100$ 125$ 150$ 175$ 200$ 225$ 250$ 275$ 300$ 325$ 350$ 375$ 400$ 425$ 450$ 475$ 500$ 525$ 550$ 575$ 600$ 625$ 650$ 675$ 700$ 725$ 750$ 775$ 800$ Monthly(cost(for(test(conﬁgura2on( Maximum(concurrent(users(whilst(maintaining(Apdex(of(at(least(0.95(( How(much(does(it(cost(to(delight(my(customers?( Ninefold$ Heroku$PX$ Smooth pricing.

Slide 56

Slide 56 text

Consistent performance.

Slide 57

Slide 57 text

1 in 20 clicks. Meet the 95th percentile.

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

Why does it even matter?

Slide 62

Slide 62 text

Performance and engagement.

Slide 63

Slide 63 text

Visitors experiencing the best 10% of load times viewed 3 times more pages than getting the worst 10%. http://assets.en.oreilly.com/1/event/29/The%20Secret%20Weapons%20of%20the%20AOL%20Optimization%20Team%20Presentation.pdf

Slide 64

Slide 64 text

Improved load time by 400ms. Increased trafﬁc by 9%. http://www.slideshare.net/stoyan/yslow-20-presentation

Slide 65

Slide 65 text

Time is money.

Slide 66

Slide 66 text

2. Do it for the money.

Slide 67

Slide 67 text

Load time: 6 seconds -> 1.2 seconds. Page views: up 25%. Revenue: up 12%. http://www.slideshare.net/timmorrow/shopzilla-performance-by-design-2433735

Slide 68

Slide 68 text

For every 100ms of load time improvement: Sales increased by 1%. https://sites.google.com/site/glinden/Home/StanfordDataMining.2006-11-28.ppt?attredirects=0

Slide 69

Slide 69 text

Where should I host?

Slide 70

Slide 70 text

With Ninefold, of course… ! {Sales Pitch}

Slide 71

Slide 71 text

1. Consistent performance. 2. Inexpensive. a) Bare VM pricing. b) Much lower sysadmin costs. Ninefold:   The platform without the penalty.

Slide 72

Slide 72 text

3. Smoothly scaling costs. 4. Easy scaling. 5. Just enough control. Ninefold:   The platform without the penalty.

Slide 73

Slide 73 text

Remember young grasshopper: Your focus determines your power.

Slide 74

Slide 74 text

The YAQ Program.

Slide 75

Slide 75 text

1. Clone your store. 2. Design a simulation. 3. Run the simulation at scale. 4. Benchmark the results. 10 Spree store tests.

Slide 76

Slide 76 text

[email protected] @danielsimlind ! https://github.com/ninefold ! ninefold.com @ninefold Please chat with us   and grab a t-shirt.

Slide 77

Slide 77 text

Phew. Made it. Thanks.