Fast Everything: Ruby Performance Tools and Understanding

A talk given at RubyNation 2015 about applying Brendan Gregg's USE philosophy to Ruby Performance Tools.

Transcript

1. Fast Everything:
   A Philosophy for
   App Performance
   Aaron Quint / @aq / Ruby Nation 2015
   

   View Slide

2. I’m @aq
   HELLO!
   

   View Slide

3. View Slide

4. Ruby, JS, Go Performance,etc.
   quirkey.com/hireme
   Available for hire.
   

   View Slide

5. View Slide

6. CatskillsConf.com Oct 23-25, 2015
   

   View Slide

7. Been building Ruby and Rails
   apps for 10 years now
   

   View Slide

8. Lets start with some
   general assumptions
   

   View Slide

9. Fast is relative
   

   View Slide

10. Probably Rails
    You are working on a web
    application with ruby
    

    View Slide

11. If it doesn’t, then why did you pick it?
    All the software you choose
    for your stack starts fast
    

    View Slide

12. Which means you have
    the power to make it fast
    ∴ Slow is a result of your
    code or your users
    

    View Slide

13. View Slide

14. And that means we should steal the tools,
    not “leave” Ruby
    There are better tools and
    ideas outside of ruby
    

    View Slide

15. View Slide

16. View Slide

17. The U.S.E. Method
    

    View Slide

18. For every resource
    Utilization
    Saturation
    Errors
    

    View Slide

19. And other anti-patterns
    A cure for Streetlighting
    

    View Slide

20. i.e. its not often applied to monitoring itself,
    but it does work in many cases
    Note: USE is usually applied
    to in-time data/discovery
    

    View Slide

21. Yes!
    Can we apply the same
    methodology to our applications?
    

    View Slide

22. nginx
    unicorn
    rails/application
    cpu
    mem
    redis
    postgresql
    resque
    Production
    controllers
    models
    Development
    ELK
    raindrops
    as::
    notifications
    SLOWLOG
    pg_stat_statments
    pgbadger
    stackprof
    resque-metrics
    objspace
    ?
    cpu
    mem
    rblineprof
    ppprofiler
    stackprof
    benchmark/ips
    memory_profiler
    Ruby Performance Tools
    ab
    wrk
    memcached
    STATS
    

    View Slide

23. Not enough time, could do a talk about each of
    them
    Thats a lot of Tools!
    

    View Slide

24. Just the highlights
    

    View Slide

25. ElasticSearch + Logstash + Kibana
    … + Statsd + Graphite
    ELK
    

    View Slide

26. If you log it …
    A Simple Idea:
    Logs -> Data -> Search + Metrics
    

    View Slide

27. View Slide

28. View Slide

29. Part of Unicorn. *Required*
    raindrops
    

    View Slide

30. View Slide

31. If you’re running on Postgres,
    there are great tools just for you
    pg_stat_statements/
    pgbadger
    

    View Slide

32. View Slide

33. paperless@[local]/paperless-production=> SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit /
    nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent
    FROM pg_stat_statements ORDER BY total_time DESC LIMIT 5;
    -[ RECORD
    1 ]---------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------
    query | SELECT * FROM "email_addresses" WHERE (LOWER(email_addresses.email_address) = LOWER(?)) LIMIT ?
    calls | 744384675
    total_time | 189703962.235031
    rows | 721856740
    hit_percent | 82.6364412412666404
    -[ RECORD
    2 ]---------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------
    query | SELECT "discount_redemptions".id FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?)
    LIMIT ? /*uuid:http-5458c26d50bf0042c4c5007fa02c2d80*/
    calls | 26472909
    total_time | 136787447.010999
    rows | 124208
    hit_percent | 99.9999962924170858
    -[ RECORD
    3 ]---------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------
    query | SELECT count(*) AS count_all FROM "discount_redemptions" WHERE ("discount_redemptions".cart_id = ?) /
    *uuid:http-19854d4f87d507488eb92de7e3d18d12*/
    calls | 25914050
    total_time | 127617483.302989
    rows | 25914050
    hit_percent | 100.0000000000000000
    -[ RECORD
    4 ]---------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------
    query | SELECT * FROM "events" WHERE ("events"."id" = ?) /*uuid:resque-8262bba6d8bfba15871083236792e93b*/
    calls | 2006362390
    total_time | 110051869.950854
    rows | 2002901227
    hit_percent | 96.8441002881508271
    -[ RECORD
    5 ]---------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------
    query | SELECT * FROM "discount_redemptions" WHERE ("discount_redemptions".account_id = ?) AND
    ("discount_redemptions"."complete" = ?) /*uuid:http-f90e03d33cfd0fa541cc87d5a7038157*/
    calls | 26336287
    total_time | 108405954.313974
    rows | 1189818
    hit_percent | 100.0000000000000000
    

    View Slide

34. github.com/basecamp/marginalia
    Bonus: Tag your queries
    

    View Slide

35. the tool you can use in production
    stackprof
    

    View Slide

36. Ruby Process (Unicorn)
    AC::Dispatch
    MyController::Create
    Template::Render
    Ar::Find
    

    View Slide

37. Ruby Process (Unicorn)
    StackProf.start rb_profile_frames() rb_profile_frames() rb_profile_frames() rb_profile_frames()
    StackProf.stop
    StackProf.dump
    

    View Slide

38. $ stackprof ~/Downloads/stackprof-cpu-1402024056.dump
    ==================================
    Mode: cpu(1000)
    Samples: 562 (0.35% miss rate)
    GC: 71 (12.63%)
    ==================================
    TOTAL (pct) SAMPLES (pct) FRAME
    33 (5.9%) 33 (5.9%) ActiveRecord::Base.scoped_methods
    30 (5.3%) 30 (5.3%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#…
    26 (4.6%) 26 (4.6%) ActiveSupport::CoreExtensions::Hash::Keys#assert_valid_keys
    24 (4.3%) 24 (4.3%) block in ActiveRecord::ConnectionAdapters::..#execute
    14 (2.5%) 14 (2.5%) block in ActiveSupport::Inflector#singularize
    13 (2.3%) 12 (2.1%) block in ActiveSupport::Notifications::Fanout#listeners_for
    12 (2.1%) 12 (2.1%) ActiveRecord::Reflection::AssociationReflection#klass
    19 (3.4%) 12 (2.1%) ActiveRecord::ConnectionAdapters::ConnectionHandler#…
    11 (2.0%) 11 (2.0%) block in ActiveRecord::Base.with_scope
    797 (141.8%) 11 (2.0%) ActiveRecord::Base.with_scope
    18 (3.2%) 8 (1.4%) ActiveRecord::Base.quote_bound_value
    7 (1.2%) 7 (1.2%) Haml::Helpers#preserve
    79 (14.1%) 7 (1.2%) block in ActiveRecord::Base.with_scope
    6 (1.1%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.connection_handler=
    72 (12.8%) 6 (1.1%) block (2 levels) in ActiveRecord::Base.with_scope
    

    View Slide

39. View Slide

40. github.com/quirkey/stackprof-remote
    Bonus: Easy remote/
    interactive sessions
    

    View Slide

41. ????
    USE for Ruby Memory
    

    View Slide

42. What do other people do?
    

    View Slide

43. View Slide

44. Moving into the future
    

    View Slide

45. Memory seems like a good place to start
    Step 1:
    Improve Existing Tools and
    Fill in Gaps
    

    View Slide

46. Form like voltron
    Step 2:
    Combine into more powerful tools
    

    View Slide

47. View Slide

48. \
    

    View Slide

49. Great to keep shipping …
    

    View Slide

50. But can it introspect?
    

    View Slide

51. Aaron Quint
    @aq
    quirkey.com/hireme
    github.com/quirkey
    beatsryetypes.com
    catskillsconf.com
    THANKS!
    

    View Slide