Tweet
Tweet

Slide 1

Slide 1 text

Fast Everything: A Philosophy for App Performance Aaron Quint / @aq / Ruby Nation 2015

Slide 2

Slide 2 text

I’m @aq HELLO!

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Ruby, JS, Go Performance,etc. quirkey.com/hireme Available for hire.

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

CatskillsConf.com Oct 23-25, 2015

Slide 7

Slide 7 text

Been building Ruby and Rails apps for 10 years now

Slide 8

Slide 8 text

Lets start with some general assumptions

Slide 9

Slide 9 text

Fast is relative

Slide 10

Slide 10 text

Probably Rails You are working on a web application with ruby

Slide 11

Slide 11 text

If it doesn’t, then why did you pick it? All the software you choose for your stack starts fast

Slide 12

Slide 12 text

Which means you have the power to make it fast ∴ Slow is a result of your code or your users

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

And that means we should steal the tools, not “leave” Ruby There are better tools and ideas outside of ruby

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

The U.S.E. Method

Slide 18

Slide 18 text

For every resource Utilization Saturation Errors

Slide 19

Slide 19 text

And other anti-patterns A cure for Streetlighting

Slide 20

Slide 20 text

i.e. its not often applied to monitoring itself, but it does work in many cases Note: USE is usually applied to in-time data/discovery

Slide 21

Slide 21 text

Yes! Can we apply the same methodology to our applications?

Slide 22

Slide 22 text

nginx unicorn rails/application cpu mem redis postgresql resque Production controllers models Development ELK raindrops as:: notifications SLOWLOG pg_stat_statments pgbadger stackprof resque-metrics objspace ? cpu mem rblineprof ppprofiler stackprof benchmark/ips memory_profiler Ruby Performance Tools ab wrk memcached STATS

Slide 23

Slide 23 text

Not enough time, could do a talk about each of them Thats a lot of Tools!

Slide 24

Slide 24 text

Just the highlights

Slide 25

Slide 25 text

ElasticSearch + Logstash + Kibana … + Statsd + Graphite ELK

Slide 26

Slide 26 text

If you log it … A Simple Idea: Logs -> Data -> Search + Metrics

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

Part of Unicorn. *Required* raindrops

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

If you’re running on Postgres, there are great tools just for you pg_stat_statements/ pgbadger

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

paperless@[local]/paperless-production=> SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit / nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent FROM pg_stat_statements ORDER BY total_time DESC LIMIT 5; -[ RECORD 1 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "email_addresses" WHERE (LOWER(email_addresses.email_address) = LOWER(?)) LIMIT ? calls | 744384675 total_time | 189703962.235031 rows | 721856740 hit_percent | 82.6364412412666404 -[ RECORD 2 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT "discount_redemptions".id FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) LIMIT ? /*uuid:http-5458c26d50bf0042c4c5007fa02c2d80*/ calls | 26472909 total_time | 136787447.010999 rows | 124208 hit_percent | 99.9999962924170858 -[ RECORD 3 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT count(*) AS count_all FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) / *uuid:http-19854d4f87d507488eb92de7e3d18d12*/ calls | 25914050 total_time | 127617483.302989 rows | 25914050 hit_percent | 100.0000000000000000 -[ RECORD 4 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "events" WHERE ("events"."id" = ?) /*uuid:resque-8262bba6d8bfba15871083236792e93b*/ calls | 2006362390 total_time | 110051869.950854 rows | 2002901227 hit_percent | 96.8441002881508271 -[ RECORD 5 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "discount_redemptions" WHERE ("discount_redemptions".account_id = ?) AND ("discount_redemptions"."complete" = ?) /*uuid:http-f90e03d33cfd0fa541cc87d5a7038157*/ calls | 26336287 total_time | 108405954.313974 rows | 1189818 hit_percent | 100.0000000000000000

Slide 34

Slide 34 text

github.com/basecamp/marginalia Bonus: Tag your queries

Slide 35

Slide 35 text

the tool you can use in production stackprof

Slide 36

Slide 36 text

Ruby Process (Unicorn) AC::Dispatch MyController::Create Template::Render Ar::Find

Slide 37

Slide 37 text

Ruby Process (Unicorn) StackProf.start rb_profile_frames() rb_profile_frames() rb_profile_frames() rb_profile_frames() StackProf.stop StackProf.dump

Slide 38

Slide 38 text

$ stackprof ~/Downloads/stackprof-cpu-1402024056.dump ================================== Mode: cpu(1000) Samples: 562 (0.35% miss rate) GC: 71 (12.63%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 33 (5.9%) 33 (5.9%) ActiveRecord::Base.scoped_methods 30 (5.3%) 30 (5.3%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#… 26 (4.6%) 26 (4.6%) ActiveSupport::CoreExtensions::Hash::Keys#assert_valid_keys 24 (4.3%) 24 (4.3%) block in ActiveRecord::ConnectionAdapters::..#execute 14 (2.5%) 14 (2.5%) block in ActiveSupport::Inflector#singularize 13 (2.3%) 12 (2.1%) block in ActiveSupport::Notifications::Fanout#listeners_for 12 (2.1%) 12 (2.1%) ActiveRecord::Reflection::AssociationReflection#klass 19 (3.4%) 12 (2.1%) ActiveRecord::ConnectionAdapters::ConnectionHandler#… 11 (2.0%) 11 (2.0%) block in ActiveRecord::Base.with_scope 797 (141.8%) 11 (2.0%) ActiveRecord::Base.with_scope 18 (3.2%) 8 (1.4%) ActiveRecord::Base.quote_bound_value 7 (1.2%) 7 (1.2%) Haml::Helpers#preserve 79 (14.1%) 7 (1.2%) block in ActiveRecord::Base.with_scope 6 (1.1%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.connection_handler= 72 (12.8%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.with_scope

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

github.com/quirkey/stackprof-remote Bonus: Easy remote/ interactive sessions

Slide 41

Slide 41 text

???? USE for Ruby Memory

Slide 42

Slide 42 text

What do other people do?

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

Moving into the future

Slide 45

Slide 45 text

Memory seems like a good place to start Step 1: Improve Existing Tools and Fill in Gaps

Slide 46

Slide 46 text

Form like voltron Step 2: Combine into more powerful tools

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

\

Slide 49

Slide 49 text

Great to keep shipping …

Slide 50

Slide 50 text

But can it introspect?

Slide 51

Slide 51 text

Aaron Quint @aq quirkey.com/hireme github.com/quirkey beatsryetypes.com catskillsconf.com THANKS!