Performance Testing Modern Apps

Slide 1

Slide 1 text

Performance Tes-ng Modern Apps Dus$n Whi*le

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Dus-n Whi8le • dus-nwhi8le.com • @dus-nwhi8le • San Francisco, California, USA • Technologist, Traveler, Pilot, Skier, Diver, Sailor, Golfer

Slide 4

Slide 4 text

Why does performance ma8er?

Slide 5

Slide 5 text

MicrosoI found that Bing searches that were 2 seconds slower resulted in a 4.3% drop in revenue per user

Slide 6

Slide 6 text

When Mozilla shaved 2.2 seconds oﬀ the landing page, Firefox downloads increased 15.4%

Slide 7

Slide 7 text

Making Barack Obama’s website 60% faster increased dona-on conversions by 14%

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Performance directly impacts the bo8om line

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

How fast is fast enough? § Performance is key to a great user experience - Under 100ms is perceived as reac$ng instantaneously - A 100ms to 300ms delay is percep$ble - 1 second is about the limit for the user's ﬂow of thought to stay uninterrupted - Users expect a site to load in 2 seconds - AIer 3 seconds, 40% will abandon your site. - 10 seconds is about the limit for keeping the user's a*en$on § Modern applica-ons spend more -me in the browser than on the server-side

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Up-me is cri-cal for enterprises and consumers

Slide 14

Slide 14 text

Treat performance as a feature!

Slide 15

Slide 15 text

Tools of the trade for performance tes-ng

Slide 16

Slide 16 text

Understand your baseline performance

Slide 17

Slide 17 text

Sta-c vs Hello World vs Applica-ons

Slide 18

Slide 18 text

Apache Bench

Slide 19

Slide 19 text

ab -c 1 -t 10 -k http://acmedemoapp.com/

Slide 20

Slide 20 text

Benchmarking acmedemoapp.com (be patient) Finished 187 requests Server Software: Apache-Coyote/1.1 Server Hostname: acmedemoapp.com Server Port: 80 Document Path: / Document Length: 5217 bytes Concurrency Level: 1 Time taken for tests: 10.039 seconds Complete requests: 187 Failed requests: 0 Keep-Alive requests: 187 Total transferred: 1021768 bytes HTML transferred: 975579 bytes Requests per second: 18.63 [#/sec] (mean) Time per request: 53.687 [ms] (mean)

Slide 21

Slide 21 text

ab -c 10 -t 10 -k http://acmedemoapp.com/

Slide 22

Slide 22 text

Benchmarking acmedemoapp.com (be patient) Finished 659 requests Server Software: Apache-Coyote/1.1 Server Hostname: acmedemoapp.com Server Port: 80 Document Path: / Document Length: 5217 bytes Concurrency Level: 10 Time taken for tests: 10.015 seconds Complete requests: 659 Failed requests: 0 Keep-Alive requests: 659 Total transferred: 3600776 bytes HTML transferred: 3438003 bytes Requests per second: 65.80 [#/sec] (mean) Time per request: 151.970 [ms] (mean) Time per request: 15.197 [ms] (mean, across all

Slide 23

Slide 23 text

Siege

Slide 24

Slide 24 text

siege -c 10 -b -t 10S http://acmedemoapp.com/

Slide 25

Slide 25 text

** SIEGE 3.0.6 ** Preparing 10 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 623 hits Availability: 100.00 % Elapsed time: 9.57 secs Data transferred: 3.10 MB Response time: 0.15 secs Transaction rate: 65.10 trans/sec Throughput: 0.32 MB/sec Concurrency: 9.91 Successful transactions: 623 Failed transactions: 0 Longest transaction: 0.30 Shortest transaction: 0.10

Slide 26

Slide 26 text

Crawl the entire app to discover all urls

Slide 27

Slide 27 text

sproxy -o ./urls.txt

Slide 28

Slide 28 text

SPROXY v1.02 listening on port 9001 ...appending HTTP requests to: ./urls.txt ...default connection timeout: 120 seconds

Slide 29

Slide 29 text

wget -r -l 0 -t 1 --spider -w 1 -e "http_proxy = http://127.0.0.1:9001" "http://acmedemoapp.com/" sort -u -o urls.txt urls.txt

Slide 30

Slide 30 text

http://acmedemoapp.com/ http://acmedemoapp.com/about http://acmedemoapp.com/cart http://acmedemoapp.com/currency/change/EUR http://acmedemoapp.com/currency/change/GBP http://acmedemoapp.com/currency/change/USD http://acmedemoapp.com/login http://acmedemoapp.com/register/ http://acmedemoapp.com/t/brand/bookmania http://acmedemoapp.com/t/category/books http://acmedemoapp.com/t/category/mugs http://acmedemoapp.com/t/category/stickers http://acmedemoapp.com/terms-of-service

Slide 31

Slide 31 text

Benchmark trafﬁc across all unique urls with siege

Slide 32

Slide 32 text

siege -v -c 50 -i -t 3M -f urls.txt -d 10

Slide 33

Slide 33 text

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Multi-Mechanize is an open source framework for performance and load testing

Slide 36

Slide 36 text

pip install multi-mechanize

Slide 37

Slide 37 text

multimech-newproject demo

Slide 38

Slide 38 text

import requests class Transaction(object): def run(self): r = requests.get(‘http://acmedemoapp.com/) r.raw.read()

Slide 39

Slide 39 text

import mechanize import time class Transaction(object): def run(self): br = mechanize.Browser() br.set_handle_robots(False) start_timer = time.time() resp = br.open(‘http://acmedemoapp.com/) resp.read() latency = time.time() - start_timer self.custom_timers['homepage'] = latency start_timer = time.time() resp = br.open(‘http://acmedemoapp.com/cart') resp.read() latency = time.time() - start_timer self.custom_timers['cart'] = latency assert (resp.code == 200)

Slide 40

Slide 40 text

multimech-run demo

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

What about when you need more than one machine?

Slide 43

Slide 43 text

Who lives in the cloud?

Slide 44

Slide 44 text

Bees with Machine Guns

Slide 45

Slide 45 text

A utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications)

Slide 46

Slide 46 text

pip install beeswithmachineguns

Slide 47

Slide 47 text

# ~/.boto [Credentials] aws_access_key_id=xxx aws_secret_access_key=xxx [Boto] ec2_region_name = us-west-2 ec2_region_endpoint = ec2.us-west-2.amazonaws.com

Slide 48

Slide 48 text

bees up -s 2 -g default -z us-west-2b -i ami-bc05898c -k appdynamics- dustinwhittle-aws-us-west-2 -l ec2-user

Slide 49

Slide 49 text

Connecting to the hive. Attempting to call up 2 bees. Waiting for bees to load their machine guns... . . . . Bee i-3828400c is ready for the attack. Bee i-3928400d is ready for the attack. The swarm has assembled 2 bees.

Slide 50

Slide 50 text

bees report

Slide 51

Slide 51 text

Read 2 bees from the roster. Bee i-3828400c: running @ 54.212.22.176 Bee i-3928400d: running @ 50.112.6.191

Slide 52

Slide 52 text

bees attack -n 1000 -c 50 -u http://acmedemoapp.com/

Slide 53

Slide 53 text

Read 2 bees from the roster. Connecting to the hive. Assembling bees. Each of 2 bees will ﬁre 500 rounds, 25 at a time. Stinging URL so it will be cached for the attack. Organizing the swarm. … Offensive complete. Complete requests: 1000 Requests per second: 306.540000 [#/sec] (mean) Time per request: 163.112000 [ms] (mean) 50% response time: 151.000000 [ms] (mean) 90% response time: 192.000000 [ms] (mean) Mission Assessment: Target crushed bee offensive. The swarm is awaiting new orders.

Slide 54

Slide 54 text

bees attack -n 100000 -c 1000 -u http://acmedemoapp.com/

Slide 55

Slide 55 text

Read 2 bees from the roster. Connecting to the hive. Assembling bees. Each of 2 bees will ﬁre 50000 rounds, 500 at a time. Stinging URL so it will be cached for the attack. Organizing the swarm. … Offensive complete. Complete requests: 100000 Requests per second: 502.420000 [#/sec] (mean) Time per request: 360.114000 [ms] (mean) 50% response time: 451.000000 [ms] (mean) 90% response time: 402.000000 [ms] (mean) Mission Assessment: Target crushed bee offensive. The swarm is awaiting new orders.

Slide 56

Slide 56 text

Read 2 bees from the roster. Connecting to the hive. Assembling bees. Each of 2 bees will ﬁre 50000 rounds, 500 at a time. Stinging URL so it will be cached for the attack. Organizing the swarm. Bee 0 is joining the swarm. Bee 1 is joining the swarm. Bee 0 is ﬁring his machine gun. Bang bang! Bee 0 lost sight of the target (connection timed out). Bee 1 lost sight of the target (connection timed out). Offensive complete. Target timed out without fully responding to 2 bees. No bees completed the mission. Apparently your bees are peace-loving hippies. The swarm is awaiting new orders.

Slide 57

Slide 57 text

bees down

Slide 58

Slide 58 text

Read 2 bees from the roster. Connecting to the hive. Calling off the swarm. Stood down 2 bees.

Slide 59

Slide 59 text

locust.io https://github.com/locustio/locust

Slide 60

Slide 60 text

pip install locustio

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

There are many tools… •Gatling.io •Wrk •Vegeta •Tsung •…

Slide 65

Slide 65 text

What about the client side?

Slide 66

Slide 66 text

In modern web applications more latency comes from the client-side than the server-side.

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

npm install psi

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

curl "https://www.googleapis.com/ pagespeedonline/v1/runPagespeed? url=http://dustinwhittle.com/&key=xxx"

Slide 74

Slide 74 text

WBench

Slide 75

Slide 75 text

gem install wbench

Slide 76

Slide 76 text

wbench http://dustinwhittle.com/

Slide 77

Slide 77 text

No content

Slide 78

Slide 78 text

Automate client-side performance tes-ng with Grunt/Gulp

Slide 79

Slide 79 text

Use Bower (for dependencies), Grunt/Gulp (for automa-on), and Yeoman (for bootstrapping)

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

How many people understand exactly how fast their site runs in produc-on?

Slide 82

Slide 82 text

Track performance in development and produc-on

Slide 83

Slide 83 text

Instrument everything = code, databases, caches, queues, third party services, and infrastructure.

Slide 84

Slide 84 text

No content

Slide 85

Slide 85 text

Chef / Sensu

Slide 86

Slide 86 text

Statsd + Graphite + Grafana

Slide 87

Slide 87 text

No content

Slide 88

Slide 88 text

Track performance of end users

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

webpagetest.org

Slide 91

Slide 91 text

No content

Slide 92

Slide 92 text

No content

Slide 93

Slide 93 text

No content

Slide 94

Slide 94 text

SiteSpeed.io

Slide 95

Slide 95 text

No content

Slide 96

Slide 96 text

No content

Slide 97

Slide 97 text

No content

Slide 98

Slide 98 text

No content

Slide 99

Slide 99 text

No content

Slide 100

Slide 100 text

No content

Slide 101

Slide 101 text

No content

Slide 102

Slide 102 text

No content

Slide 103

Slide 103 text

Load tes-ng services from the cloud

Slide 104

Slide 104 text

No content

Slide 105

Slide 105 text

No content

Slide 106

Slide 106 text

No content

Slide 107

Slide 107 text

Test for failures • NetFlix Simian Army + Chaos Monkey  • What happens if you lose a caching layer? • What happens if dependencies slow down?

Slide 108

Slide 108 text

Performance Ma*ers • Treat performance as a feature • Using the 14kb Rule for instant loading • Markup management • Elimina$ng excess AJAX calls • Working with and around applica$on cache • Developing a responsive design + image strategy • Implemen$ng a good touch-ﬁrst strategy • Code management for good produc$on and development experiences • Using task runners to build and deploy produc$on code

Slide 109

Slide 109 text

Best Prac$ces • Treat performance as a feature • Capacity plan and load test the server-side • Op-mize and performance test the client-side • Understand your star-ng point • Instrument everything • Monitor performance in development and produc-on • Measure the diﬀerence of every change • Automate performance tes-ng in your build and deployment process • Understand how failures impact performance

Slide 110

Slide 110 text

The protocols are evolving • The limita$ons of HTTP/1.X forced us to develop various applica$on workarounds (sharding, concatena$on, spri$ng, inlining, etc.) to op$mize performance. However, in the process we’ve also introduced numerous regressions: poor caching, unnecessary downloads, delayed execu$on, and more. • HTTP/2 eliminates the need for these hacks and allows us to both simplify our applica$ons and deliver improved performance. • You should unshard, unconcat, and unsprite your assets • You should switch from inlining to server push • Read Ilya Grigorik awesome book on browser performance - h*p://hpbn.co/ h*p2

Slide 111

Slide 111 text

Integrate automated performance tes-ng into con-nuous integra-on for server-side and client-side