Fast Everything: Ruby Performance Tools and Understanding

A talk given at RubyNation 2015 about applying Brendan Gregg's USE philosophy to Ruby Performance Tools.

Transcript

1. Fast Everything: A Philosophy for App Performance Aaron Quint / @aq / Ruby Nation 2015
2. I’m @aq HELLO!
4. Ruby, JS, Go Performance,etc. quirkey.com/hireme Available for hire.
6. CatskillsConf.com Oct 23-25, 2015
7. Been building Ruby and Rails apps for 10 years now
8. Lets start with some general assumptions
9. Fast is relative
10. Probably Rails You are working on a web application with ruby
11. If it doesn’t, then why did you pick it? All the software you choose for your stack starts fast
12. Which means you have the power to make it fast ∴ Slow is a result of your code or your users
14. And that means we should steal the tools, not “leave” Ruby There are better tools and ideas outside of ruby
17. The U.S.E. Method
18. For every resource Utilization Saturation Errors
19. And other anti-patterns A cure for Streetlighting
20. i.e. its not often applied to monitoring itself, but it does work in many cases Note: USE is usually applied to in-time data/discovery
21. Yes! Can we apply the same methodology to our applications?
22. nginx unicorn rails/application cpu mem redis postgresql resque Production controllers models Development ELK raindrops as:: notifications SLOWLOG pg_stat_statments pgbadger stackprof resque-metrics objspace ? cpu mem rblineprof ppprofiler stackprof benchmark/ips memory_profiler Ruby Performance Tools ab wrk memcached STATS
23. Not enough time, could do a talk about each of them Thats a lot of Tools!
24. Just the highlights
25. ElasticSearch + Logstash + Kibana … + Statsd + Graphite ELK
26. If you log it … A Simple Idea: Logs -> Data -> Search + Metrics
29. Part of Unicorn. *Required* raindrops
31. If you’re running on Postgres, there are great tools just for you pg_stat_statements/ pgbadger
33. paperless@[local]/paperless-production=> SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit / nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent FROM pg_stat_statements ORDER BY total_time DESC LIMIT 5; -[ RECORD 1 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "email_addresses" WHERE (LOWER(email_addresses.email_address) = LOWER(?)) LIMIT ? calls | 744384675 total_time | 189703962.235031 rows | 721856740 hit_percent | 82.6364412412666404 -[ RECORD 2 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT "discount_redemptions".id FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) LIMIT ? /*uuid:http-5458c26d50bf0042c4c5007fa02c2d80*/ calls | 26472909 total_time | 136787447.010999 rows | 124208 hit_percent | 99.9999962924170858 -[ RECORD 3 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT count(*) AS count_all FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) / *uuid:http-19854d4f87d507488eb92de7e3d18d12*/ calls | 25914050 total_time | 127617483.302989 rows | 25914050 hit_percent | 100.0000000000000000 -[ RECORD 4 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "events" WHERE ("events"."id" = ?) /*uuid:resque-8262bba6d8bfba15871083236792e93b*/ calls | 2006362390 total_time | 110051869.950854 rows | 2002901227 hit_percent | 96.8441002881508271 -[ RECORD 5 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "discount_redemptions" WHERE ("discount_redemptions".account_id = ?) AND ("discount_redemptions"."complete" = ?) /*uuid:http-f90e03d33cfd0fa541cc87d5a7038157*/ calls | 26336287 total_time | 108405954.313974 rows | 1189818 hit_percent | 100.0000000000000000
34. github.com/basecamp/marginalia Bonus: Tag your queries
35. the tool you can use in production stackprof
36. Ruby Process (Unicorn) AC::Dispatch MyController::Create Template::Render Ar::Find
37. Ruby Process (Unicorn) StackProf.start rb_profile_frames() rb_profile_frames() rb_profile_frames() rb_profile_frames() StackProf.stop StackProf.dump
38. $ stackprof ~/Downloads/stackprof-cpu-1402024056.dump ================================== Mode: cpu(1000) Samples: 562 (0.35% miss rate) GC: 71 (12.63%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 33 (5.9%) 33 (5.9%) ActiveRecord::Base.scoped_methods 30 (5.3%) 30 (5.3%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#… 26 (4.6%) 26 (4.6%) ActiveSupport::CoreExtensions::Hash::Keys#assert_valid_keys 24 (4.3%) 24 (4.3%) block in ActiveRecord::ConnectionAdapters::..#execute 14 (2.5%) 14 (2.5%) block in ActiveSupport::Inflector#singularize 13 (2.3%) 12 (2.1%) block in ActiveSupport::Notifications::Fanout#listeners_for 12 (2.1%) 12 (2.1%) ActiveRecord::Reflection::AssociationReflection#klass 19 (3.4%) 12 (2.1%) ActiveRecord::ConnectionAdapters::ConnectionHandler#… 11 (2.0%) 11 (2.0%) block in ActiveRecord::Base.with_scope 797 (141.8%) 11 (2.0%) ActiveRecord::Base.with_scope 18 (3.2%) 8 (1.4%) ActiveRecord::Base.quote_bound_value 7 (1.2%) 7 (1.2%) Haml::Helpers#preserve 79 (14.1%) 7 (1.2%) block in ActiveRecord::Base.with_scope 6 (1.1%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.connection_handler= 72 (12.8%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.with_scope
40. github.com/quirkey/stackprof-remote Bonus: Easy remote/ interactive sessions
41. ???? USE for Ruby Memory
42. What do other people do?
44. Moving into the future
45. Memory seems like a good place to start Step 1: Improve Existing Tools and Fill in Gaps
46. Form like voltron Step 2: Combine into more powerful tools
48. \
49. Great to keep shipping …
50. But can it introspect?
51. Aaron Quint @aq quirkey.com/hireme github.com/quirkey beatsryetypes.com catskillsconf.com THANKS!