Real World Ruby Performance

Transcript

1. 1.

   Real WORLD RUBY PERFORMANCE Aaron Quint / @aq / Ruby

   Conf 2014
2. 2.

   @tmm1 @SamSaffron @_ko1 SHOUTOUT

3. 3.

   We’ll come back to who I am later. It’s [relatively]

   unimportant. SKIPPING THE INTRO
4. 4.

   I’ve learned so much over the past 5 years, what

   could I share? This TALK was HARD TO WRITE
5. 5.

   It’s a ⌘+C ⌘+P culture. TIPS And tricks are the

   CLIFF NOTES of tech learning
6. 6.

   How to THINK about a problem is much more interesting

   than how to solve it. As a mentor I want to teach philosophy not snippets
7. 7.

   The tools and tricks will change over time. Today, Take

   away the process
8. 8.

   A multi-step process. Ruby Performance as therapy

9. 9.

   It’s a multi-step process Relax, Open up We’re going to

   go deep
10. 10.

    Step 1: Acceptance

11. 11.

    It’s your Fault.

12. 12.

    Really?

13. 13.

    Yes.

14. 14.

    None
15. 15.

    It’s not you, It’s me.

16. 16.

    It’s not you, It’s me.

17. 17.

    — George Costanza (Inventor of “It’s not you, it’s me”)

    It’s not you, It’s me.
18. 18.

    Performance is about context

19. 19.

    Doesn’t scale for what? To what degree? With what hardware?

    … “X Doesn’t SCALE” IS BS
20. 20.

    So when we talk about our ruby being slow

21. 21.

    None
22. 22.

    Rails

23. 23.

    Rails 10ms

24. 24.

    Rails Your application 10ms

25. 25.

    Rails Your application DB 10ms

26. 26.

    Rails Your application DB 10ms 20ms

27. 27.

    Rails Your application DB Cache 10ms 20ms

28. 28.

    Rails Your application DB Cache 10ms 20ms 10ms

29. 29.

    Rails Your application DB Cache 10ms 20ms 10ms 250ms

30. 30.

    IT’s MY FAULT.

31. 31.

    Step 2: Diagnosis

32. 32.

    Where did I go wrong?

33. 33.

    METRICS! Measurement! MMMNUMBERS! Milliseconds MATTER!

34. 34.

    Use the right one for the job. Tools abound!

35. 35.

    Step 3: Treatment

36. 36.

    what are the steps to fix this problem?

37. 37.

    How many strokes for the lowest #? Playing golf.

38. 38.

    Two angles of optimization

39. 39.

    Proxies/Balancers Application Datastores Filesystem/OS/Hardware Individual Request Path (Controller#action)

40. 40.

    aka, speeding up a single query, controller action, or code

    path Vertical: Fix individual Elements
41. 41.

    aka, Adding more workers per-node, buying better hardware Horizontal: Address

    hardware or software across a cluster
42. 42.

    Important Themes:

43. 43.

    Context is crucial to acceptance

44. 44.

    Visibility and Introspect- ability are crucial to diagnosis

45. 45.

    Knowing your tools is crucial to treatment

46. 46.

    I’m Aaron Quint. I’m the chief Scientist at Paperless Post.

47. 47.

    None
48. 48.

    Opposing forces. Features vs. speed

49. 49.

    We realized that being fast meant being stable

50. 50.

    CASE STUDIES in performance therapy

51. 51.

    None
52. 52.

    Case 1: JSON FOR DAYS

53. 53.

    None
54. 54.

    None
55. 55.

    None
56. 56.

    package:7292:1123434234234

57. 57.

    package:7290:11234342343424 partner:8:11234342343424 partner:8:11234342343424

58. 58.

    package:7290:11234342343424 partner:8:11234342343424 partner:8:11234342343424 package:7292:1123434234234

59. 59.

    package:7290:11234342343424 partner:8:11234342343424 partner:8:11234342343424 package:7292:1123434234234

60. 60.

    Uncached performance is still a problem

61. 61.

    ppprofiler to the rescue

62. 62.

    ppprofiler

63. 63.

    ppprofiler • Auto-cache toggling • Benchmark • Rblineprof • As::Notification

    Counts (SQL/Cache, etc) • MemoryProfiler (NEW!) • Gist-able (markdown) output
64. 64.

    None
65. 65.

    None
66. 66.

    None
67. 67.

    Rinse and Repeat Make the slowest lines faster

68. 68.

    None
69. 69.

    None
70. 70.

    None
71. 71.

    None
72. 72.

    None
73. 73.

    Case 2: FINGER IN THE SOCKET

74. 74.

    Before Vday we were looking for any wins

75. 75.

    IN BETWEEN THE LINES! stackprof + stackprof-remote

76. 76.

    None
77. 77.

    Ruby Process (Unicorn)

78. 78.

    Ruby Process (Unicorn)

79. 79.

    Ruby Process (Unicorn)

80. 80.

    Ruby Process (Unicorn) AC::Dispatch

81. 81.

    Ruby Process (Unicorn) AC::Dispatch MyController::Create

82. 82.

    Ruby Process (Unicorn) AC::Dispatch MyController::Create Template::Render

83. 83.

    Ruby Process (Unicorn) AC::Dispatch MyController::Create Template::Render Ar::Find

84. 84.

    Ruby Process (Unicorn)

85. 85.

    Ruby Process (Unicorn) StackProf.start rb_profile_frames() rb_profile_frames() rb_profile_frames() rb_profile_frames() StackProf.stop StackProf.dump

86. 86.

    ! [paperless@production-webapp10 current]$ stackprof tmp/stackprof-cpu-30715-1391204970.dump ================================== Mode: cpu(1000) Samples: 1761

    (3.61% miss rate) GC: 128 (7.27%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 344 (19.5%) 342 (19.4%) Statsd#send_to_socket 393 (22.3%) 44 (2.5%) Statsd#sampled 44 (2.5%) 44 (2.5%) block in ActiveRecord::ConnectionAdapters::PostgreSQLPoolAdapter#execute 56 (3.2%) 29 (1.6%) block in ActiveSupport::Notifications::Fanout#listeners_for 29 (1.6%) 29 (1.6%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#extract_pg_identifier_from_name 26 (1.5%) 26 (1.5%) ActiveSupport::Notifications::Fanout::Subscribers::Evented#subscribed_to? 25 (1.4%) 25 (1.4%) String#blank? 25 (1.4%) 25 (1.4%) block (2 levels) in ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#select 24 (1.4%) 24 (1.4%) ActiveRecord::Base.scoped_methods 22 (1.2%) 22 (1.2%) Dalli::Server::KSocket#kgio_wait_readable 21 (1.2%) 21 (1.2%) ActiveSupport::CoreExtensions::Hash::Keys#assert_valid_keys 42 (2.4%) 20 (1.1%) block in Dalli::Server::KSocket#readfull 28 (1.6%) 19 (1.1%) ActiveRecord::ConnectionAdapters::ConnectionHandler#retrieve_connection_pool 18 (1.0%) 18 (1.0%) #<Module:0x00000002004b08>.instrumenter 17 (1.0%) 16 (0.9%) Dalli::Server#deserialize 15 (0.9%) 15 (0.9%) block (2 levels) in ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#select_raw 14 (0.8%) 14 (0.8%) #<Module:0x000000033e25d0>.decode_www_form_component 13 (0.7%) 13 (0.7%) Dalli::Server#write 15 (0.9%) 11 (0.6%) ActiveSupport::CoreExtensions::Time::Calculations#minus_with_coercion 10 (0.6%) 10 (0.6%) block in ActiveRecord::Base.with_scope 10 (0.6%) 10 (0.6%) block in ActiveRecord::ConnectionAdapters::QueryCache#cache_sql 21 (1.2%) 10 (0.6%) Yajl::Encoder.encode 10 (0.6%) 10 (0.6%) Set#add 10 (0.6%) 10 (0.6%) block (2 levels) in ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#result_as_array 10 (0.6%) 10 (0.6%) ActiveSupport::CoreExtensions::Time::Calculations::ClassMethods#time_with_datetime_fallback 9 (0.5%) 9 (0.5%) ActiveRecord::DynamicFinderMatch#initialize 9 (0.5%) 9 (0.5%) ActiveSupport::LogSubscriber.logger 9 (0.5%) 9 (0.5%) block in ActionController::Base.action_methods 9 (0.5%) 9 (0.5%) block in ActionController::Base.action_methods 9 (0.5%) 9 (0.5%) block (2 levels) in ActiveRecord::Base.connection_handler=
87. 87.

    Hmm, why is statsd slow?

88. 88.

    Pull out good old benchmark

89. 89.

    $ ruby test/profile/statsd.rb user system total real udp with connect

    0.010000 0.000000 0.010000 ( 0.074522) udp without connect 0.120000 0.530000 0.650000 ( 13.096515) statsd with connect 0.000000 0.090000 0.090000 ( 0.103520) statsd without connect 0.100000 0.620000 0.720000 ( 13.483539)
90. 90.

    WIN!

91. 91.

    None
92. 92.

    Case 3: THE HOLIDAY SCALE

93. 93.

    None
94. 94.

    Some times you can throw money at the problem

95. 95.

    None
96. 96.

    Case 4: SHRINKING THE GAP

97. 97.

    Start at the top, work your way down. Starting with

    a HITLIST
98. 98.

    Number of Requests x 90th Percentile Response Time Total Time

99. 99.

    None
100. 100.

     None
101. 101.

     Using Stackprof flamegraphs on production.

102. 102.

     Using Stackprof flamegraphs on production. SET IT ON FIRE!

103. 103.

     None
104. 104.

     None
105. 105.

     None
106. 106.

     None
107. 107.

     None
108. 108.

     Big wins are not the point

109. 109.

     If you’re not failing you’re not being honest

110. 110.

     Don’t just make tools, learn to use them

111. 111.

     twitter: @aq github.com/quirkey github.com/paperlesspost Thanks!