Fast Everything: Ruby Performance Tools and Understanding

Fast Everything: Ruby Performance Tools and Understanding

A talk given at RubyNation 2015 about applying Brendan Gregg's USE philosophy to Ruby Performance Tools.

F04bfa14141dca6713f0d9caa763e26b?s=128

Aaron Quint

June 13, 2015
Tweet

Transcript

  1. 3.
  2. 5.
  3. 11.

    If it doesn’t, then why did you pick it? All

    the software you choose for your stack starts fast
  4. 12.

    Which means you have the power to make it fast

    ∴ Slow is a result of your code or your users
  5. 13.
  6. 14.

    And that means we should steal the tools, not “leave”

    Ruby There are better tools and ideas outside of ruby
  7. 15.
  8. 16.
  9. 20.

    i.e. its not often applied to monitoring itself, but it

    does work in many cases Note: USE is usually applied to in-time data/discovery
  10. 22.

    nginx unicorn rails/application cpu mem redis postgresql resque Production controllers

    models Development ELK raindrops as:: notifications SLOWLOG pg_stat_statments pgbadger stackprof resque-metrics objspace ? cpu mem rblineprof ppprofiler stackprof benchmark/ips memory_profiler Ruby Performance Tools ab wrk memcached STATS
  11. 23.
  12. 26.

    If you log it … A Simple Idea: Logs ->

    Data -> Search + Metrics
  13. 27.
  14. 28.
  15. 30.
  16. 31.
  17. 32.
  18. 33.

    paperless@[local]/paperless-production=> SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit /

    nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent FROM pg_stat_statements ORDER BY total_time DESC LIMIT 5; -[ RECORD 1 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "email_addresses" WHERE (LOWER(email_addresses.email_address) = LOWER(?)) LIMIT ? calls | 744384675 total_time | 189703962.235031 rows | 721856740 hit_percent | 82.6364412412666404 -[ RECORD 2 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT "discount_redemptions".id FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) LIMIT ? /*uuid:http-5458c26d50bf0042c4c5007fa02c2d80*/ calls | 26472909 total_time | 136787447.010999 rows | 124208 hit_percent | 99.9999962924170858 -[ RECORD 3 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT count(*) AS count_all FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) / *uuid:http-19854d4f87d507488eb92de7e3d18d12*/ calls | 25914050 total_time | 127617483.302989 rows | 25914050 hit_percent | 100.0000000000000000 -[ RECORD 4 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "events" WHERE ("events"."id" = ?) /*uuid:resque-8262bba6d8bfba15871083236792e93b*/ calls | 2006362390 total_time | 110051869.950854 rows | 2002901227 hit_percent | 96.8441002881508271 -[ RECORD 5 ]--------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- query | SELECT * FROM "discount_redemptions" WHERE ("discount_redemptions".account_id = ?) AND ("discount_redemptions"."complete" = ?) /*uuid:http-f90e03d33cfd0fa541cc87d5a7038157*/ calls | 26336287 total_time | 108405954.313974 rows | 1189818 hit_percent | 100.0000000000000000
  19. 38.

    $ stackprof ~/Downloads/stackprof-cpu-1402024056.dump ================================== Mode: cpu(1000) Samples: 562 (0.35% miss

    rate) GC: 71 (12.63%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 33 (5.9%) 33 (5.9%) ActiveRecord::Base.scoped_methods 30 (5.3%) 30 (5.3%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#… 26 (4.6%) 26 (4.6%) ActiveSupport::CoreExtensions::Hash::Keys#assert_valid_keys 24 (4.3%) 24 (4.3%) block in ActiveRecord::ConnectionAdapters::..#execute 14 (2.5%) 14 (2.5%) block in ActiveSupport::Inflector#singularize 13 (2.3%) 12 (2.1%) block in ActiveSupport::Notifications::Fanout#listeners_for 12 (2.1%) 12 (2.1%) ActiveRecord::Reflection::AssociationReflection#klass 19 (3.4%) 12 (2.1%) ActiveRecord::ConnectionAdapters::ConnectionHandler#… 11 (2.0%) 11 (2.0%) block in ActiveRecord::Base.with_scope 797 (141.8%) 11 (2.0%) ActiveRecord::Base.with_scope 18 (3.2%) 8 (1.4%) ActiveRecord::Base.quote_bound_value 7 (1.2%) 7 (1.2%) Haml::Helpers#preserve 79 (14.1%) 7 (1.2%) block in ActiveRecord::Base.with_scope 6 (1.1%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.connection_handler= 72 (12.8%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.with_scope
  20. 39.
  21. 43.
  22. 45.

    Memory seems like a good place to start Step 1:

    Improve Existing Tools and Fill in Gaps
  23. 47.
  24. 48.

    \