Real World Ruby Performance

F04bfa14141dca6713f0d9caa763e26b?s=47 Aaron Quint
November 19, 2014

Real World Ruby Performance

My talk from RubyConf 2014 about Ruby Performance and the philosophy of performance.

F04bfa14141dca6713f0d9caa763e26b?s=128

Aaron Quint

November 19, 2014
Tweet

Transcript

  1. Real WORLD RUBY PERFORMANCE Aaron Quint / @aq / Ruby

    Conf 2014
  2. @tmm1 @SamSaffron @_ko1 SHOUTOUT

  3. We’ll come back to who I am later. It’s [relatively]

    unimportant. SKIPPING THE INTRO
  4. I’ve learned so much over the past 5 years, what

    could I share? This TALK was HARD TO WRITE
  5. It’s a ⌘+C ⌘+P culture. TIPS And tricks are the

    CLIFF NOTES of tech learning
  6. How to THINK about a problem is much more interesting

    than how to solve it. As a mentor I want to teach philosophy not snippets
  7. The tools and tricks will change over time. Today, Take

    away the process
  8. A multi-step process. Ruby Performance as therapy

  9. It’s a multi-step process Relax, Open up We’re going to

    go deep
  10. Step 1: Acceptance

  11. It’s your Fault.

  12. Really?

  13. Yes.

  14. None
  15. It’s not you, It’s me.

  16. It’s not you, It’s me.

  17. — George Costanza (Inventor of “It’s not you, it’s me”)

    It’s not you, It’s me.
  18. Performance is about context

  19. Doesn’t scale for what? To what degree? With what hardware?

    … “X Doesn’t SCALE” IS BS
  20. So when we talk about our ruby being slow

  21. None
  22. Rails

  23. Rails 10ms

  24. Rails Your application 10ms

  25. Rails Your application DB 10ms

  26. Rails Your application DB 10ms 20ms

  27. Rails Your application DB Cache 10ms 20ms

  28. Rails Your application DB Cache 10ms 20ms 10ms

  29. Rails Your application DB Cache 10ms 20ms 10ms 250ms

  30. IT’s MY FAULT.

  31. Step 2: Diagnosis

  32. Where did I go wrong?

  33. METRICS! Measurement! MMMNUMBERS! Milliseconds MATTER!

  34. Use the right one for the job. Tools abound!

  35. Step 3: Treatment

  36. what are the steps to fix this problem?

  37. How many strokes for the lowest #? Playing golf.

  38. Two angles of optimization

  39. Proxies/Balancers Application Datastores Filesystem/OS/Hardware Individual Request Path (Controller#action)

  40. aka, speeding up a single query, controller action, or code

    path Vertical: Fix individual Elements
  41. aka, Adding more workers per-node, buying better hardware Horizontal: Address

    hardware or software across a cluster
  42. Important Themes:

  43. Context is crucial to acceptance

  44. Visibility and Introspect- ability are crucial to diagnosis

  45. Knowing your tools is crucial to treatment

  46. I’m Aaron Quint. I’m the chief Scientist at Paperless Post.

  47. None
  48. Opposing forces. Features vs. speed

  49. We realized that being fast meant being stable

  50. CASE STUDIES in performance therapy

  51. None
  52. Case 1: JSON FOR DAYS

  53. None
  54. None
  55. None
  56. package:7292:1123434234234

  57. package:7290:11234342343424 partner:8:11234342343424 partner:8:11234342343424

  58. package:7290:11234342343424 partner:8:11234342343424 partner:8:11234342343424 package:7292:1123434234234

  59. package:7290:11234342343424 partner:8:11234342343424 partner:8:11234342343424 package:7292:1123434234234

  60. Uncached performance is still a problem

  61. ppprofiler to the rescue

  62. ppprofiler

  63. ppprofiler • Auto-cache toggling • Benchmark • Rblineprof • As::Notification

    Counts (SQL/Cache, etc) • MemoryProfiler (NEW!) • Gist-able (markdown) output
  64. None
  65. None
  66. None
  67. Rinse and Repeat Make the slowest lines faster

  68. None
  69. None
  70. None
  71. None
  72. None
  73. Case 2: FINGER IN THE SOCKET

  74. Before Vday we were looking for any wins

  75. IN BETWEEN THE LINES! stackprof + stackprof-remote

  76. None
  77. Ruby Process (Unicorn)

  78. Ruby Process (Unicorn)

  79. Ruby Process (Unicorn)

  80. Ruby Process (Unicorn) AC::Dispatch

  81. Ruby Process (Unicorn) AC::Dispatch MyController::Create

  82. Ruby Process (Unicorn) AC::Dispatch MyController::Create Template::Render

  83. Ruby Process (Unicorn) AC::Dispatch MyController::Create Template::Render Ar::Find

  84. Ruby Process (Unicorn)

  85. Ruby Process (Unicorn) StackProf.start rb_profile_frames() rb_profile_frames() rb_profile_frames() rb_profile_frames() StackProf.stop StackProf.dump

  86. ! [paperless@production-webapp10 current]$ stackprof tmp/stackprof-cpu-30715-1391204970.dump ================================== Mode: cpu(1000) Samples: 1761

    (3.61% miss rate) GC: 128 (7.27%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 344 (19.5%) 342 (19.4%) Statsd#send_to_socket 393 (22.3%) 44 (2.5%) Statsd#sampled 44 (2.5%) 44 (2.5%) block in ActiveRecord::ConnectionAdapters::PostgreSQLPoolAdapter#execute 56 (3.2%) 29 (1.6%) block in ActiveSupport::Notifications::Fanout#listeners_for 29 (1.6%) 29 (1.6%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#extract_pg_identifier_from_name 26 (1.5%) 26 (1.5%) ActiveSupport::Notifications::Fanout::Subscribers::Evented#subscribed_to? 25 (1.4%) 25 (1.4%) String#blank? 25 (1.4%) 25 (1.4%) block (2 levels) in ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#select 24 (1.4%) 24 (1.4%) ActiveRecord::Base.scoped_methods 22 (1.2%) 22 (1.2%) Dalli::Server::KSocket#kgio_wait_readable 21 (1.2%) 21 (1.2%) ActiveSupport::CoreExtensions::Hash::Keys#assert_valid_keys 42 (2.4%) 20 (1.1%) block in Dalli::Server::KSocket#readfull 28 (1.6%) 19 (1.1%) ActiveRecord::ConnectionAdapters::ConnectionHandler#retrieve_connection_pool 18 (1.0%) 18 (1.0%) #<Module:0x00000002004b08>.instrumenter 17 (1.0%) 16 (0.9%) Dalli::Server#deserialize 15 (0.9%) 15 (0.9%) block (2 levels) in ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#select_raw 14 (0.8%) 14 (0.8%) #<Module:0x000000033e25d0>.decode_www_form_component 13 (0.7%) 13 (0.7%) Dalli::Server#write 15 (0.9%) 11 (0.6%) ActiveSupport::CoreExtensions::Time::Calculations#minus_with_coercion 10 (0.6%) 10 (0.6%) block in ActiveRecord::Base.with_scope 10 (0.6%) 10 (0.6%) block in ActiveRecord::ConnectionAdapters::QueryCache#cache_sql 21 (1.2%) 10 (0.6%) Yajl::Encoder.encode 10 (0.6%) 10 (0.6%) Set#add 10 (0.6%) 10 (0.6%) block (2 levels) in ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#result_as_array 10 (0.6%) 10 (0.6%) ActiveSupport::CoreExtensions::Time::Calculations::ClassMethods#time_with_datetime_fallback 9 (0.5%) 9 (0.5%) ActiveRecord::DynamicFinderMatch#initialize 9 (0.5%) 9 (0.5%) ActiveSupport::LogSubscriber.logger 9 (0.5%) 9 (0.5%) block in ActionController::Base.action_methods 9 (0.5%) 9 (0.5%) block in ActionController::Base.action_methods 9 (0.5%) 9 (0.5%) block (2 levels) in ActiveRecord::Base.connection_handler=
  87. Hmm, why is statsd slow?

  88. Pull out good old benchmark

  89. $ ruby test/profile/statsd.rb user system total real udp with connect

    0.010000 0.000000 0.010000 ( 0.074522) udp without connect 0.120000 0.530000 0.650000 ( 13.096515) statsd with connect 0.000000 0.090000 0.090000 ( 0.103520) statsd without connect 0.100000 0.620000 0.720000 ( 13.483539)
  90. WIN!

  91. None
  92. Case 3: THE HOLIDAY SCALE

  93. None
  94. Some times you can throw money at the problem

  95. None
  96. Case 4: SHRINKING THE GAP

  97. Start at the top, work your way down. Starting with

    a HITLIST
  98. Number of Requests x 90th Percentile Response Time Total Time

  99. None
  100. None
  101. Using Stackprof flamegraphs on production.

  102. Using Stackprof flamegraphs on production. SET IT ON FIRE!

  103. None
  104. None
  105. None
  106. None
  107. None
  108. Big wins are not the point

  109. If you’re not failing you’re not being honest

  110. Don’t just make tools, learn to use them

  111. twitter: @aq github.com/quirkey github.com/paperlesspost Thanks!