A talk given at RubyNation 2015 about applying Brendan Gregg's USE philosophy to Ruby Performance Tools.
Fast Everything:A Philosophy forApp PerformanceAaron Quint / @aq / Ruby Nation 2015
View Slide
I’m @aqHELLO!
Ruby, JS, Go Performance,etc.quirkey.com/hiremeAvailable for hire.
CatskillsConf.com Oct 23-25, 2015
Been building Ruby and Railsapps for 10 years now
Lets start with somegeneral assumptions
Fast is relative
Probably RailsYou are working on a webapplication with ruby
If it doesn’t, then why did you pick it?All the software you choosefor your stack starts fast
Which means you havethe power to make it fast∴ Slow is a result of yourcode or your users
And that means we should steal the tools,not “leave” RubyThere are better tools andideas outside of ruby
The U.S.E. Method
For every resourceUtilizationSaturationErrors
And other anti-patternsA cure for Streetlighting
i.e. its not often applied to monitoring itself,but it does work in many casesNote: USE is usually appliedto in-time data/discovery
Yes!Can we apply the samemethodology to our applications?
nginxunicornrails/applicationcpumemredispostgresqlresqueProductioncontrollersmodelsDevelopmentELKraindropsas::notificationsSLOWLOGpg_stat_statmentspgbadgerstackprofresque-metricsobjspace?cpumemrblineprofppprofilerstackprofbenchmark/ipsmemory_profilerRuby Performance ToolsabwrkmemcachedSTATS
Not enough time, could do a talk about each ofthemThats a lot of Tools!
Just the highlights
ElasticSearch + Logstash + Kibana… + Statsd + GraphiteELK
If you log it …A Simple Idea:Logs -> Data -> Search + Metrics
Part of Unicorn. *Required*raindrops
If you’re running on Postgres,there are great tools just for youpg_stat_statements/pgbadger
paperless@[local]/paperless-production=> SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit /nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percentFROM pg_stat_statements ORDER BY total_time DESC LIMIT 5;-[ RECORD1 ]-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------query | SELECT * FROM "email_addresses" WHERE (LOWER(email_addresses.email_address) = LOWER(?)) LIMIT ?calls | 744384675total_time | 189703962.235031rows | 721856740hit_percent | 82.6364412412666404-[ RECORD2 ]-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------query | SELECT "discount_redemptions".id FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?)LIMIT ? /*uuid:http-5458c26d50bf0042c4c5007fa02c2d80*/calls | 26472909total_time | 136787447.010999rows | 124208hit_percent | 99.9999962924170858-[ RECORD3 ]-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------query | SELECT count(*) AS count_all FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) /*uuid:http-19854d4f87d507488eb92de7e3d18d12*/calls | 25914050total_time | 127617483.302989rows | 25914050hit_percent | 100.0000000000000000-[ RECORD4 ]-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------query | SELECT * FROM "events" WHERE ("events"."id" = ?) /*uuid:resque-8262bba6d8bfba15871083236792e93b*/calls | 2006362390total_time | 110051869.950854rows | 2002901227hit_percent | 96.8441002881508271-[ RECORD5 ]-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------query | SELECT * FROM "discount_redemptions" WHERE ("discount_redemptions".account_id = ?) AND("discount_redemptions"."complete" = ?) /*uuid:http-f90e03d33cfd0fa541cc87d5a7038157*/calls | 26336287total_time | 108405954.313974rows | 1189818hit_percent | 100.0000000000000000
github.com/basecamp/marginaliaBonus: Tag your queries
the tool you can use in productionstackprof
Ruby Process (Unicorn)AC::DispatchMyController::CreateTemplate::RenderAr::Find
Ruby Process (Unicorn)StackProf.start rb_profile_frames() rb_profile_frames() rb_profile_frames() rb_profile_frames()StackProf.stopStackProf.dump
$ stackprof ~/Downloads/stackprof-cpu-1402024056.dump==================================Mode: cpu(1000)Samples: 562 (0.35% miss rate)GC: 71 (12.63%)==================================TOTAL (pct) SAMPLES (pct) FRAME33 (5.9%) 33 (5.9%) ActiveRecord::Base.scoped_methods30 (5.3%) 30 (5.3%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#…26 (4.6%) 26 (4.6%) ActiveSupport::CoreExtensions::Hash::Keys#assert_valid_keys24 (4.3%) 24 (4.3%) block in ActiveRecord::ConnectionAdapters::..#execute14 (2.5%) 14 (2.5%) block in ActiveSupport::Inflector#singularize13 (2.3%) 12 (2.1%) block in ActiveSupport::Notifications::Fanout#listeners_for12 (2.1%) 12 (2.1%) ActiveRecord::Reflection::AssociationReflection#klass19 (3.4%) 12 (2.1%) ActiveRecord::ConnectionAdapters::ConnectionHandler#…11 (2.0%) 11 (2.0%) block in ActiveRecord::Base.with_scope797 (141.8%) 11 (2.0%) ActiveRecord::Base.with_scope18 (3.2%) 8 (1.4%) ActiveRecord::Base.quote_bound_value7 (1.2%) 7 (1.2%) Haml::Helpers#preserve79 (14.1%) 7 (1.2%) block in ActiveRecord::Base.with_scope6 (1.1%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.connection_handler=72 (12.8%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.with_scope
github.com/quirkey/stackprof-remoteBonus: Easy remote/interactive sessions
????USE for Ruby Memory
What do other people do?
Moving into the future
Memory seems like a good place to startStep 1:Improve Existing Tools andFill in Gaps
Form like voltronStep 2:Combine into more powerful tools
\
Great to keep shipping …
But can it introspect?
Aaron Quint@aqquirkey.com/hiremegithub.com/quirkeybeatsryetypes.comcatskillsconf.comTHANKS!