$30 off During Our Annual Pro Sale. View Details »

Fast Everything: Ruby Performance Tools and Understanding

Fast Everything: Ruby Performance Tools and Understanding

A talk given at RubyNation 2015 about applying Brendan Gregg's USE philosophy to Ruby Performance Tools.

Aaron Quint

June 13, 2015
Tweet

More Decks by Aaron Quint

Other Decks in Programming

Transcript

  1. Fast Everything:
    A Philosophy for
    App Performance
    Aaron Quint / @aq / Ruby Nation 2015

    View Slide

  2. I’m @aq
    HELLO!

    View Slide

  3. View Slide

  4. Ruby, JS, Go Performance,etc.
    quirkey.com/hireme
    Available for hire.

    View Slide

  5. View Slide

  6. CatskillsConf.com Oct 23-25, 2015

    View Slide

  7. Been building Ruby and Rails
    apps for 10 years now

    View Slide

  8. Lets start with some
    general assumptions

    View Slide

  9. Fast is relative

    View Slide

  10. Probably Rails
    You are working on a web
    application with ruby

    View Slide

  11. If it doesn’t, then why did you pick it?
    All the software you choose
    for your stack starts fast

    View Slide

  12. Which means you have
    the power to make it fast
    ∴ Slow is a result of your
    code or your users

    View Slide

  13. View Slide

  14. And that means we should steal the tools,
    not “leave” Ruby
    There are better tools and
    ideas outside of ruby

    View Slide

  15. View Slide

  16. View Slide

  17. The U.S.E. Method

    View Slide

  18. For every resource
    Utilization
    Saturation
    Errors

    View Slide

  19. And other anti-patterns
    A cure for Streetlighting

    View Slide

  20. i.e. its not often applied to monitoring itself,
    but it does work in many cases
    Note: USE is usually applied
    to in-time data/discovery

    View Slide

  21. Yes!
    Can we apply the same
    methodology to our applications?

    View Slide

  22. nginx
    unicorn
    rails/application
    cpu
    mem
    redis
    postgresql
    resque
    Production
    controllers
    models
    Development
    ELK
    raindrops
    as::
    notifications
    SLOWLOG
    pg_stat_statments
    pgbadger
    stackprof
    resque-metrics
    objspace
    ?
    cpu
    mem
    rblineprof
    ppprofiler
    stackprof
    benchmark/ips
    memory_profiler
    Ruby Performance Tools
    ab
    wrk
    memcached
    STATS

    View Slide

  23. Not enough time, could do a talk about each of
    them
    Thats a lot of Tools!

    View Slide

  24. Just the highlights

    View Slide

  25. ElasticSearch + Logstash + Kibana
    … + Statsd + Graphite
    ELK

    View Slide

  26. If you log it …
    A Simple Idea:
    Logs -> Data -> Search + Metrics

    View Slide

  27. View Slide

  28. View Slide

  29. Part of Unicorn. *Required*
    raindrops

    View Slide

  30. View Slide

  31. If you’re running on Postgres,
    there are great tools just for you
    pg_stat_statements/
    pgbadger

    View Slide

  32. View Slide

  33. paperless@[local]/paperless-production=> SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit /
    nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent
    FROM pg_stat_statements ORDER BY total_time DESC LIMIT 5;
    -[ RECORD
    1 ]---------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------
    query | SELECT * FROM "email_addresses" WHERE (LOWER(email_addresses.email_address) = LOWER(?)) LIMIT ?
    calls | 744384675
    total_time | 189703962.235031
    rows | 721856740
    hit_percent | 82.6364412412666404
    -[ RECORD
    2 ]---------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------
    query | SELECT "discount_redemptions".id FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?)
    LIMIT ? /*uuid:http-5458c26d50bf0042c4c5007fa02c2d80*/
    calls | 26472909
    total_time | 136787447.010999
    rows | 124208
    hit_percent | 99.9999962924170858
    -[ RECORD
    3 ]---------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------
    query | SELECT count(*) AS count_all FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) /
    *uuid:http-19854d4f87d507488eb92de7e3d18d12*/
    calls | 25914050
    total_time | 127617483.302989
    rows | 25914050
    hit_percent | 100.0000000000000000
    -[ RECORD
    4 ]---------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------
    query | SELECT * FROM "events" WHERE ("events"."id" = ?) /*uuid:resque-8262bba6d8bfba15871083236792e93b*/
    calls | 2006362390
    total_time | 110051869.950854
    rows | 2002901227
    hit_percent | 96.8441002881508271
    -[ RECORD
    5 ]---------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------
    query | SELECT * FROM "discount_redemptions" WHERE ("discount_redemptions".account_id = ?) AND
    ("discount_redemptions"."complete" = ?) /*uuid:http-f90e03d33cfd0fa541cc87d5a7038157*/
    calls | 26336287
    total_time | 108405954.313974
    rows | 1189818
    hit_percent | 100.0000000000000000

    View Slide

  34. github.com/basecamp/marginalia
    Bonus: Tag your queries

    View Slide

  35. the tool you can use in production
    stackprof

    View Slide

  36. Ruby Process (Unicorn)
    AC::Dispatch
    MyController::Create
    Template::Render
    Ar::Find

    View Slide

  37. Ruby Process (Unicorn)
    StackProf.start rb_profile_frames() rb_profile_frames() rb_profile_frames() rb_profile_frames()
    StackProf.stop
    StackProf.dump

    View Slide

  38. $ stackprof ~/Downloads/stackprof-cpu-1402024056.dump
    ==================================
    Mode: cpu(1000)
    Samples: 562 (0.35% miss rate)
    GC: 71 (12.63%)
    ==================================
    TOTAL (pct) SAMPLES (pct) FRAME
    33 (5.9%) 33 (5.9%) ActiveRecord::Base.scoped_methods
    30 (5.3%) 30 (5.3%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#…
    26 (4.6%) 26 (4.6%) ActiveSupport::CoreExtensions::Hash::Keys#assert_valid_keys
    24 (4.3%) 24 (4.3%) block in ActiveRecord::ConnectionAdapters::..#execute
    14 (2.5%) 14 (2.5%) block in ActiveSupport::Inflector#singularize
    13 (2.3%) 12 (2.1%) block in ActiveSupport::Notifications::Fanout#listeners_for
    12 (2.1%) 12 (2.1%) ActiveRecord::Reflection::AssociationReflection#klass
    19 (3.4%) 12 (2.1%) ActiveRecord::ConnectionAdapters::ConnectionHandler#…
    11 (2.0%) 11 (2.0%) block in ActiveRecord::Base.with_scope
    797 (141.8%) 11 (2.0%) ActiveRecord::Base.with_scope
    18 (3.2%) 8 (1.4%) ActiveRecord::Base.quote_bound_value
    7 (1.2%) 7 (1.2%) Haml::Helpers#preserve
    79 (14.1%) 7 (1.2%) block in ActiveRecord::Base.with_scope
    6 (1.1%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.connection_handler=
    72 (12.8%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.with_scope

    View Slide

  39. View Slide

  40. github.com/quirkey/stackprof-remote
    Bonus: Easy remote/
    interactive sessions

    View Slide

  41. ????
    USE for Ruby Memory

    View Slide

  42. What do other people do?

    View Slide

  43. View Slide

  44. Moving into the future

    View Slide

  45. Memory seems like a good place to start
    Step 1:
    Improve Existing Tools and
    Fill in Gaps

    View Slide

  46. Form like voltron
    Step 2:
    Combine into more powerful tools

    View Slide

  47. View Slide

  48. \

    View Slide

  49. Great to keep shipping …

    View Slide

  50. But can it introspect?

    View Slide

  51. Aaron Quint
    @aq
    quirkey.com/hireme
    github.com/quirkey
    beatsryetypes.com
    catskillsconf.com
    THANKS!

    View Slide