Fast Everything: Ruby Performance Tools and Understanding

Fast Everything: Ruby Performance Tools and Understanding

A talk given at RubyNation 2015 about applying Brendan Gregg's USE philosophy to Ruby Performance Tools.

F04bfa14141dca6713f0d9caa763e26b?s=128

Aaron Quint

June 13, 2015
Tweet

Transcript

  1. Fast Everything: A Philosophy for App Performance Aaron Quint /

    @aq / Ruby Nation 2015
  2. I’m @aq HELLO!

  3. None
  4. Ruby, JS, Go Performance,etc. quirkey.com/hireme Available for hire.

  5. None
  6. CatskillsConf.com Oct 23-25, 2015

  7. Been building Ruby and Rails apps for 10 years now

  8. Lets start with some general assumptions

  9. Fast is relative

  10. Probably Rails You are working on a web application with

    ruby
  11. If it doesn’t, then why did you pick it? All

    the software you choose for your stack starts fast
  12. Which means you have the power to make it fast

    ∴ Slow is a result of your code or your users
  13. None
  14. And that means we should steal the tools, not “leave”

    Ruby There are better tools and ideas outside of ruby
  15. None
  16. None
  17. The U.S.E. Method

  18. For every resource Utilization Saturation Errors

  19. And other anti-patterns A cure for Streetlighting

  20. i.e. its not often applied to monitoring itself, but it

    does work in many cases Note: USE is usually applied to in-time data/discovery
  21. Yes! Can we apply the same methodology to our applications?

  22. nginx unicorn rails/application cpu mem redis postgresql resque Production controllers

    models Development ELK raindrops as:: notifications SLOWLOG pg_stat_statments pgbadger stackprof resque-metrics objspace ? cpu mem rblineprof ppprofiler stackprof benchmark/ips memory_profiler Ruby Performance Tools ab wrk memcached STATS
  23. Not enough time, could do a talk about each of

    them Thats a lot of Tools!
  24. Just the highlights

  25. ElasticSearch + Logstash + Kibana … + Statsd + Graphite

    ELK
  26. If you log it … A Simple Idea: Logs ->

    Data -> Search + Metrics
  27. None
  28. None
  29. Part of Unicorn. *Required* raindrops

  30. None
  31. If you’re running on Postgres, there are great tools just

    for you pg_stat_statements/ pgbadger
  32. None
  33. paperless@[local]/paperless-production=> SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit /

    nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent FROM pg_stat_statements ORDER BY total_time DESC LIMIT 5; -[ RECORD 1 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "email_addresses" WHERE (LOWER(email_addresses.email_address) = LOWER(?)) LIMIT ? calls | 744384675 total_time | 189703962.235031 rows | 721856740 hit_percent | 82.6364412412666404 -[ RECORD 2 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT "discount_redemptions".id FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) LIMIT ? /*uuid:http-5458c26d50bf0042c4c5007fa02c2d80*/ calls | 26472909 total_time | 136787447.010999 rows | 124208 hit_percent | 99.9999962924170858 -[ RECORD 3 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT count(*) AS count_all FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) / *uuid:http-19854d4f87d507488eb92de7e3d18d12*/ calls | 25914050 total_time | 127617483.302989 rows | 25914050 hit_percent | 100.0000000000000000 -[ RECORD 4 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "events" WHERE ("events"."id" = ?) /*uuid:resque-8262bba6d8bfba15871083236792e93b*/ calls | 2006362390 total_time | 110051869.950854 rows | 2002901227 hit_percent | 96.8441002881508271 -[ RECORD 5 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "discount_redemptions" WHERE ("discount_redemptions".account_id = ?) AND ("discount_redemptions"."complete" = ?) /*uuid:http-f90e03d33cfd0fa541cc87d5a7038157*/ calls | 26336287 total_time | 108405954.313974 rows | 1189818 hit_percent | 100.0000000000000000
  34. github.com/basecamp/marginalia Bonus: Tag your queries

  35. the tool you can use in production stackprof

  36. Ruby Process (Unicorn) AC::Dispatch MyController::Create Template::Render Ar::Find

  37. Ruby Process (Unicorn) StackProf.start rb_profile_frames() rb_profile_frames() rb_profile_frames() rb_profile_frames() StackProf.stop StackProf.dump

  38. $ stackprof ~/Downloads/stackprof-cpu-1402024056.dump ================================== Mode: cpu(1000) Samples: 562 (0.35% miss

    rate) GC: 71 (12.63%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 33 (5.9%) 33 (5.9%) ActiveRecord::Base.scoped_methods 30 (5.3%) 30 (5.3%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#… 26 (4.6%) 26 (4.6%) ActiveSupport::CoreExtensions::Hash::Keys#assert_valid_keys 24 (4.3%) 24 (4.3%) block in ActiveRecord::ConnectionAdapters::..#execute 14 (2.5%) 14 (2.5%) block in ActiveSupport::Inflector#singularize 13 (2.3%) 12 (2.1%) block in ActiveSupport::Notifications::Fanout#listeners_for 12 (2.1%) 12 (2.1%) ActiveRecord::Reflection::AssociationReflection#klass 19 (3.4%) 12 (2.1%) ActiveRecord::ConnectionAdapters::ConnectionHandler#… 11 (2.0%) 11 (2.0%) block in ActiveRecord::Base.with_scope 797 (141.8%) 11 (2.0%) ActiveRecord::Base.with_scope 18 (3.2%) 8 (1.4%) ActiveRecord::Base.quote_bound_value 7 (1.2%) 7 (1.2%) Haml::Helpers#preserve 79 (14.1%) 7 (1.2%) block in ActiveRecord::Base.with_scope 6 (1.1%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.connection_handler= 72 (12.8%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.with_scope
  39. None
  40. github.com/quirkey/stackprof-remote Bonus: Easy remote/ interactive sessions

  41. ???? USE for Ruby Memory

  42. What do other people do?

  43. None
  44. Moving into the future

  45. Memory seems like a good place to start Step 1:

    Improve Existing Tools and Fill in Gaps
  46. Form like voltron Step 2: Combine into more powerful tools

  47. None
  48. \

  49. Great to keep shipping …

  50. But can it introspect?

  51. Aaron Quint @aq quirkey.com/hireme github.com/quirkey beatsryetypes.com catskillsconf.com THANKS!